In Module 01, we set a hypothesis: "Checkout rate will increase." But how do we measure that? Do we look at the raw number of checkouts? The percentage of users? The revenue per user?
Picking the wrong metric is the easiest way to run a "successful" test that bankrupts the company. For example, if you optimize purely for Conversion Rate, the easiest way to win is to lower your prices by 90%. Your conversion will skyrocket, but your business will die.
This is the "North Star" of your experiment. It is the single metric that decides whether the test wins or loses. It must balance short-term behavior (clicks) with long-term value (retention/revenue).
Good OEC: Revenue Per Visitor (RPV).
Bad OEC: Click-Through Rate (CTR).
Why is CTR bad? Because you can increase clicks by using "clickbait" or misleading buttons. Users will click, realize they were tricked, and then leave. Your experiment "won" on clicks but lost on retention.
A mature experiment doesn't just track one number. It tracks three categories of metrics simultaneously.
The decision maker. If this moves significantly, the test wins.
Ex: Checkout Conversion Rate, Revenue Per User.
These help debug the user behavior. They don't decide the winner, but they explain the story.
Ex: Add-to-Cart Rate, Time on Page.
Metrics that must not get worse. If these drop, the test is aborted, even if the Primary Metric wins.
Ex: Latency, Unsubscribes.
Guardrails are critical because experiments often have unintended side effects. You need to protect the business from these "invisible" costs.
We have our hypothesis. We have our Primary Metric (e.g., Conversion Rate). Now we need to ask: "How big of a change do we care about?"
Is a 0.1% increase worth it? Or do we need 5%? This decision determines your sample size. This is the Minimum Detectable Effect (MDE).