Guides
A/B Testing Guide for Growth Teams
A step-by-step guide to running high-quality A/B tests that drive real growth. From hypothesis formation to statistical significance.
Define Your Growth Metric
Before running any experiment, identify the one metric that matters most. This should be tied directly to business value — not vanity metrics. Common choices: activation rate, conversion rate, revenue per user, or retention at day 7/30.
Tips
- — Use the ICE framework (Impact, Confidence, Ease) to prioritize which metric to optimize first
- — Ensure you have reliable tracking before running experiments
Form a Hypothesis
Write a clear hypothesis: 'If we [change], then [metric] will [improve] because [reason].' The 'because' is critical — it forces you to think about causation, not just correlation. Without it, you're just guessing.
Tips
- — Review heatmaps, session recordings, and user feedback for hypothesis inspiration
- — One hypothesis per experiment — never test multiple hypotheses simultaneously
Calculate Sample Size
Before launching, calculate the minimum sample size needed for statistical significance. Use a power analysis with: baseline conversion rate, minimum detectable effect (MDE), significance level (usually 0.05), and power (usually 0.8). Underpowered tests waste time and resources.
Warning
- — Running a test too short is the #1 mistake growth teams make
- — Never peek at results early and stop the test when it 'looks significant'
Build and Launch the Experiment
Implement the variation with minimal code changes. Use feature flags to control rollout. Ensure random, even traffic split between control and variant. QA both versions across devices and browsers before going live.
Tips
- — Start with a 50/50 split for maximum statistical power
- — Run tests for full week cycles to account for day-of-week effects
Analyze Results and Document
Wait for the pre-calculated sample size before analyzing. Check for statistical significance (p < 0.05). Look at secondary metrics to ensure no negative downstream effects. Document everything: hypothesis, results, screenshots, and learnings.
Tips
- — Build a shared experiment log that the whole team can reference
- — Even 'failed' experiments generate valuable learnings — document them equally
Warning
- — Beware of novelty effects — new designs often show temporary lifts that fade
Scale What Works
If the variant wins, roll it out to 100% of users. Then ask: can this learning be applied elsewhere? A winning headline framework might work across multiple pages. A successful pricing change might inform other product tiers. Systematize your wins.
Tips
- — Build a 'growth playbook' of proven patterns your team can reuse
- — Share wins across the company to build experimentation culture
Pro Tips
- 01Most experiments fail. If your win rate is above 30%, you're not testing bold enough hypotheses — you're optimizing, not learning
- 02Never trust an A/B test that ran for less than 2 full business weeks. Weekend vs weekday behavior differences will fool you
- 03The biggest ROI experiments are usually on pricing and packaging, not button colors. Test the things that make your palms sweat
- 04Build a 'growth playbook' of every experiment — wins and losses. New hires should be able to read it and understand your growth thesis
- 05If your sample size calculator says you need 50,000 visitors and you get 5,000/month — don't run the test. Redesign for a bigger effect size or pick a different lever
Growth Hacking Tools
Strategic Playbooks
Playbook
How to Build a Strategic Partnership Program From Scratch
An operator playbook for designing, launching, and scaling a strategic partnership program — from first hire to a measurable revenue contribution.
Playbook
The Enterprise Tech Partnership Playbook
How tech companies should structure strategic partnerships with enterprise customers and platforms — moving beyond logo deals to real co-engineering, co-selling, and joint roadmaps.
Playbook
The VC Portfolio BD Playbook: Building Real Partnership Value at Scale
How venture firms should structure portfolio business development to actually move partner-sourced revenue across their companies — not just facilitate intros.
Roles That Use This Knowledge
Role
Chief Revenue Officer (CRO)
C-suite executive owning all revenue-generating functions — sales, partnerships, customer success, and often marketing — at scaling B2B companies.
Role
Director of Channel Partnerships
Senior partnerships leader running the channel program — resellers, distributors, MSPs, and SI partners — including recruiting, enabling, and managing partner-sourced revenue.
Role
Head of Strategic Partnerships
Senior leader who designs and runs the company's strategic partnership program, owning partner relationships, deal structures, and partner-sourced revenue contribution.
Decisions to Compare
Compare
Business Development Rep (BDR) vs. Account Executive (AE): Roles and Career Path
Compare Business Development Rep (BDR) and Account Executive (AE) roles — what each does, comp differences, and how to progress between them.
Compare
Business Development vs. Sales: How They're Different and When to Use Each
Side-by-side comparison of business development and sales — the motions, the metrics, the org structures, and how to know which function you actually need.
Compare
Channel Partner vs. Reseller: Are They the Same? Differences Explained
Compare channel partners and resellers — overlapping terms, real differences, and how to structure a program that works.
Explore Further
Hub
Tools
Free calculators and interactive utilities
Hub
Playbooks
Strategic playbooks for partnerships and BD
Hub
Case Studies
Strategic breakdowns of leading companies and projects
Hub
Lists
Curated rankings of the best companies, tools, and programs
Hub
Profiles
Founders, investors, and operators shaping tech
Hub
Strategies
Pattern analysis of how top companies built their advantage
Hub
Postmortems
Why FTX, WeWork, Theranos and other major failures collapsed
Hub
Roles
Business development and partnership roles defined
Hub
Salaries
Compensation data by role and city
Hub
Compare
Side-by-side comparisons of roles and strategies
About the Author
David Shadrake
David Shadrake works on strategic business development and tech partnerships, with focus areas across AI, fintech, venture capital, growth, sales, SEO, blockchain, and broader tech innovation. Read more of his perspective on partnerships, market dynamics, and emerging technology at davidshadrake.com.