Module 01

The Hypothesis Framework

Why "Let's see what happens" is a recipe for failure. Structuring rigorous experiments using the scientific method.

In most companies, A/B testing is treated like a slot machine. Marketing teams throw random ideas at the website—"Make the button green!", "Change the headline!", "Add a carousel!"—and hope one of them hits the jackpot.

This is not experimentation; this is gambling. And just like gambling, the house (random variance) usually wins.

To run a mature experimentation program, you must stop testing ideas and start testing hypotheses. An idea is a suggestion. A hypothesis is a falsifiable statement rooted in data.

1. The "If-Then-Because" Model

Before any test is approved for the roadmap, it must pass the structure test. If you cannot fill in these three blanks, you are not ready to launch.

IF We remove the "Company News" section from the checkout page...

THEN The Checkout Completion Rate will increase by at least 2%...

BECAUSE Qualitative session recordings show users are clicking news links and abandoning the purchase funnel (Distraction Hypothesis).

Why this works:

IF: Defines the specific change (Treatment).
THEN: Defines the Primary Metric and the Minimum Detectable Effect (MDE). You are predicting the magnitude of the win.
BECAUSE: Connects the test to prior research. This prevents random guessing.

2. Null vs. Alternative Hypothesis

Statistically, we never prove that the new version (B) is better. We simply try to disprove that it is the same.

The Null Hypothesis ($H_0$)

This is the default state of the universe. It assumes your brilliant new design has zero effect.

H_0: \mu_{control} = \mu_{test}

The Alternative Hypothesis ($H_1$)

This is what we hope to find. It states that there is a statistically significant difference between the two variations.

H_1: \mu_{control} \neq \mu_{test}

The Burden of Proof

In A/B testing, the "Innocent until proven Guilty" principle applies. We assume the Null Hypothesis is true until the data screams otherwise. We require a p-value < 0.05 to reject the Null, meaning there is less than a 5% chance the result was just luck.

3. One-Sided vs. Two-Sided Tests

When configuring your test engine, you will often be asked: "Is this a one-tailed or two-tailed test?"

One-Sided (One-Tailed): You only care if B is better than A. You ignore the possibility that B is worse. This requires smaller sample sizes but is dangerous.
Two-Sided (Two-Tailed): You are testing for any difference. Did B improve conversion? Or did B crash conversion?

My Recommendation: Always use Two-Sided tests. In business, knowing you broke something (negative lift) is just as valuable as knowing you improved it.

4. Next Steps

Now that we have a hypothesis ("Conversion will rise by 2%"), we face a measurement problem. Which conversion rate? Clicks? Purchases? Revenue per user? This leads us to Metric Selection.

Previous — Start —

Next Module Metric Selection: The OEC & Guardrails →