Measure and improve · Core
Run A/B tests and experiments
An A/B test shows two versions to similar groups of users and compares results. It is the most reliable way to know whether a change caused an improvement.
About 5-20 daysEvergreenLast reviewed 5 November 2026
Who this applies to
- Skills it builds
- Data and measurementDiscovery and research
- PM types
- All
- Product types
- All
- Starting points
- All
- Ways of working
- All
- Experience
- Intermediate (1 to 4 years)Experienced (5+ years)
- Size of organisation
- All
Steps, by experience
Beginner (under 1 year)
- Write a hypothesis: what you will change, for whom, and what you expect to happen.
- Pick one main metric and one or two guardrails before you start.
- Ask an analyst how many users and how long you need for a reliable answer.
- Split users at random between the current version and the new one.
- Do not stop early because the result looks good. Wait until the planned end.
- Record the result, including when nothing changed, and decide what to do next.
Intermediate (1 to 4 years)
- Write a hypothesis with a primary metric and guardrails.
- Calculate sample size and duration before starting.
- Run the test without peeking or changing it midway.
- Analyse, record and share the result.
Experienced (5+ years)
- Run an A/A test or sample checks to make sure the experiment system is trustworthy.
- Use an overall evaluation measure that reflects long-term value.
- Watch for novelty effects and differences between user groups.
- Build an experiment log so the organisation learns from every test.
Who can do this
Owner: Product manager
Can be delegated to
- Data Scientist / ML Engineer or Data Analyst: Design the test, calculate sample size and analyse results
- Frontend Engineer or Full-stack Engineer: Build the variants behind a feature flag
Reviewed by: Data Scientist / ML Engineer, Policy / Legal / Compliance
Tool examples
- Amplitude
- Use the experiment features to assign users and analyse results with guardrails.
- Mixpanel
- Use the experiments report to compare variants on your main metric.
- Google Analytics 4
- Compare variant audiences on key events if you run tests through another tool.
- Notion
- Keep an experiment log: hypothesis, design, result and decision.
Definition of done
- Hypothesis, metric and guardrails set before starting
- Sample size and duration calculated
- Result recorded and decision made
Common pitfalls
- Stopping early when the result looks good
- Testing many changes at once and not knowing which worked
- Only recording wins
If you lead the team
What good looks like: Experiments are well designed, run long enough and change decisions.
Ask your PM
- What did our last experiment show, and what did we do as a result?
- How do we make sure results are trustworthy?
Warning signs
- Tests are stopped early when they look good.
- Results are ignored when they disagree with opinions.