BUILT INTO EVERY GOODSPEED APP
A/B Experiments (Stable Bucketing)
useExperiment assigns users to variants by hashing their user ID, persists assignments in Supabase for stable cohorts, and exposes variant strings for any path.
- Tier: Common
- Status: Template pattern
- Config: features.experiments.enabled
WHY IT MATTERS
Most A/B testing setups suffer from the same structural problem: assignment is not truly stable. A user refreshes the app, the experiment re-evaluates, and they land in a different variant. The cohort is contaminated. Analysis yields a statistically significant result that points in the wrong direction because the same user showed up in both groups. The gas-template avoids this entirely with a write-wins assignment model backed by Supabase. On the first mount of useExperiment, the hook computes a candidate variant by hashing the user ID and experiment name together using FNV-1a, a fast non-cryptographic hash that produces the same output for the same input on every device, every session, for the lifetime of the experiment. It then attempts to upsert a row into the experiments table with ignoreDuplicates: true. Whoever inserts first wins. Every subsequent call for the same user and experiment name finds the existing row and returns it, regardless of which device, session, or app version the user is on. The assignment is stable because it is written to a database, not computed in memory on every render.
The design also addresses cold-path performance. A round-trip to Supabase on every screen mount would add visible latency on every navigation. The hook uses AsyncStorage as a session-local cache: once the assignment is resolved and written to the experiments table, it is also written under a prefixed key in AsyncStorage. On subsequent mounts the cache hit short-circuits the Supabase call entirely. The cache key includes the user ID and experiment name, so a user signing into a second device fetches from Supabase once and then hits the cache from that point forward. For anonymous users who have not signed in, the hook calls assignBucket with null, which hashes the literal string 'anon' concatenated with the experiment name. This gives anonymous sessions a stable in-memory assignment within a single session without polluting the Supabase table with rows that cannot be attributed to a real user. When the user signs in, the authenticated bucket takes over. Assignments fire a single captureEvent call to PostHog on first write, tagged with the experiment name and variant, which is all you need to define a funnel and run significance calculations in PostHog's experimentation dashboard.
HOW IT IS WIRED
Real code from the GAS template
The code below is drawn from hooks/useExperiment.ts in the gas-template repository. This is the code your generated app gets, not pseudocode, not a description of intent.
// hooks/useExperiment.ts (gas-template)
// Stable variant assignment: FNV-1a hash -> upsert ignoreDuplicates -> read back.
export function useExperiment(name: string, variants: string[]): string {
const [variant, setVariant] = useState<string>(variants[0]);
const variantsHash = useMemo(() => fnv1a32(variants.join('|')), [variants]);
useEffect(() => {
let cancelled = false;
(async () => {
const userId = await getCurrentUserId();
if (!userId) {
// Anonymous: deterministic within-session bucket, not persisted.
const bucket = assignBucket(null, name, variants.length);
if (!cancelled) setVariant(variants[bucket]);
return;
}
// AsyncStorage cache short-circuits the round-trip on subsequent mounts.
const cacheKey = `${CACHE_PREFIX}${name}:${userId}`;
const cached = await AsyncStorage.getItem(cacheKey).catch(() => null);
if (cached && variants.includes(cached)) {
if (!cancelled) setVariant(cached);
return;
}
// ignoreDuplicates: whoever inserts first wins; concurrent mounts are safe.
const candidate = variants[assignBucket(userId, name, variants.length)];
await supabase.from('experiments').upsert(
{ user_id: userId, experiment_name: name, variant: candidate },
{ onConflict: 'user_id,experiment_name', ignoreDuplicates: true },
);
const { data: row } = await supabase
.from('experiments').select('variant')
.eq('user_id', userId).eq('experiment_name', name).maybeSingle();
const resolved = row?.variant && variants.includes(row.variant)
? row.variant : candidate;
await AsyncStorage.setItem(cacheKey, resolved).catch(() => undefined);
if (resolved === candidate) {
captureEvent(EVENTS.experiment_assigned, { name, variant: resolved });
}
if (!cancelled) setVariant(resolved);
})();
return () => { cancelled = true; };
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [name, variantsHash]);
return variant;
}Source: goodspeed-apps/gas-template → hooks/useExperiment.ts
HONEST LIMITS
When A/B Experiments (Stable Bucketing) is the wrong choice
Experiments on trust surfaces deserve special scrutiny before launch. The payment flow, the auth screens, and any UI where a user is handing over credentials or agreeing to terms are the wrong places to run underpowered tests. The stable-bucketing model means once a user is assigned to a variant they stay there for the duration of the experiment, which is exactly what you want for analysis. But if the experiment is testing a change to the checkout experience with a sample size of 200 users over three days and a target metric of purchase conversion, the statistical power is almost certainly too low to trust the result. You will either miss a real effect or conclude a harmful variant is safe. Running A/B tests on high-stakes surfaces requires pre-registration: define the hypothesis, the primary metric, the minimum detectable effect, and the required sample size before touching any code. If that work has not been done, use a feature flag controlled rollout instead of an experiment. A second scenario where this pattern adds overhead without benefit is very small user bases. The experiments table requires a Supabase round-trip on the first mount for each experiment a user encounters. For an app with 50 active users and no meaningful statistical power to evaluate results anyway, the infrastructure cost exceeds the value. The feature-flags pattern with a deterministic rollout bucket is a lighter-weight alternative: it achieves consistent per-user behavior with a single table read at startup and no per-experiment write. Also note that the AsyncStorage cache is keyed by user ID and experiment name but not by variant set. If you change the list of variants for a running experiment after users have already been assigned, the cached assignment may reference a variant string that is no longer in the new variants array. The hook guards against this by falling back to the candidate on a cache miss, but the safest practice is to never mutate the variant list for a running experiment. Create a new experiment name instead.
Tier: Common · Template pattern
Evaluate your use case
Check whether a/b experiments (stable bucketing) aligns with your target audience, platform constraints, and regulatory environment before enabling it.
Audit the config
The `features.experiments.enabled` flag controls this feature. Set it to false in gas.config.ts to disable the feature entirely with no residual code paths.
Seek alternatives
If the built-in implementation does not fit, the generated codebase is standard React Native + Expo code. Any library in the Expo ecosystem can replace the default.
APPS USING THIS FEATURE
Every generated Goodspeed app includes a/b experiments (stable bucketing). Browse the ideas catalog to see apps across all categories that ship with this feature wired in.
CAPABILITIES
A/B Experiments (Stable Bucketing) capability breakdown
Concrete dimensions of what the built-in a/b experiments (stable bucketing) implementation covers. These reflect the actual template code, not a marketing summary.
| Item | Description | Strength |
|---|---|---|
| Assignment model | FNV-1a hash of (userId + experiment name) modulo variant count. Deterministic on every device, every session -- same user always lands in the same bucket for the same experiment. | Deterministic hash |
| Persistence layer | Assignments are written to the experiments Supabase Postgres table with ignoreDuplicates: true. Whoever inserts first wins; subsequent upserts from other devices are no-ops. | Supabase Postgres |
| Client-side cache | Resolved variants are written to AsyncStorage under experiment:<name>:<userId>. Cache hits short-circuit the Supabase round-trip on subsequent mounts within the same install. | AsyncStorage (local) |
| Anonymous user handling | Users who have not signed in receive a deterministic in-session bucket based on hashing 'anon' + experiment name. Assignment is not persisted to Supabase until a real user ID is available. | Anon bucket (in-session) |
| Analytics event | A single captureEvent(EVENTS.experiment_assigned, { name, variant }) fires to PostHog on the first successful write. Repeat visits do not fire duplicate events. Use this as the exposure event in PostHog's experimentation dashboard. | PostHog (on first assign) |
COMMON QUESTIONS
How is the variant assignment stable across devices and sessions?
The variant is stored in the experiments table in Supabase with a unique constraint on (user_id, experiment_name). The upsert call sets ignoreDuplicates: true, which means the database only accepts the first write for that pair. Every subsequent call from any device or session attempts the same upsert and gets a no-op, then reads the existing row. The read always returns the value that won the first insert. Because Postgres serializes concurrent writes at the constraint level, two simultaneous first-mounts from different devices (say, a user signing in on both a phone and a tablet at the same time) also resolve correctly: one insert wins, both devices read the same row. The assignment is stable for the lifetime of the experiment row in the database, not just for a single session or install.
What happens if I add a new variant to a running experiment?
The hook guards against stale cache entries by checking whether the cached variant string is still present in the current variants array before using it. If the cached value is a variant name that no longer exists in the new array, the cache entry is treated as a miss and the hook falls through to Supabase. From Supabase it reads the existing row, and if the stored variant is also not in the new array (because the row was written under the old variant set), it falls back to the freshly computed candidate. This means users who were assigned under the old variant set may silently migrate to a new bucket. The safest practice is to never mutate the variant list or rename variants for an experiment that is already running. Instead, stop the existing experiment by removing its useExperiment calls, and start a new experiment with a new name and the updated variant set.
Does this work for experiments outside React components, such as in a background task?
The useExperiment hook is React-specific and relies on useState and useEffect. For imperative code outside components, you can call the underlying functions directly: use assignBucket(userId, experimentName, variants.length) from lib/hash.ts for a deterministic candidate, then run the upsert-and-read pattern against the experiments table directly with the Supabase client. The hook is a thin wrapper over exactly those two primitives. For server-side assignment in a Supabase Edge Function or an API route that needs to make a branching decision before returning a response, the same FNV-1a hash and modulo approach applies. The hash function is pure and has no React or React Native dependencies, so it runs identically in a Node environment or Deno-based Edge Function.
How do I analyze experiment results in PostHog?
The hook fires captureEvent(EVENTS.experiment_assigned, { name, variant }) to PostHog on the first successful assignment write. This event is your exposure event: it marks the moment a user entered the experiment. In PostHog's Experiments product, create an experiment with the same name string you pass to useExperiment, set the exposure event to experiment_assigned filtered on name = your_experiment_name, and define your goal metric (a separate PostHog event you fire when the target behavior occurs, such as a subscription started or a key feature used). PostHog computes statistical significance and confidence intervals from the exposure events and goal events automatically. The key constraint is that you need enough unique users exposed and enough goal events for the results to be meaningful. PostHog's sample size calculator is the right tool for deciding the minimum cohort size before you start the experiment.
GET IT BUILT INTO YOUR APP