Table of Contents

Chapter 1 How A/B Testing Helped Win the White House—Twice 1

PART I LESSONS LEARNED FROM 200,000 A/B TESTS (AND COUNTING) 13

Chapter 2 What to Test 17

Chapter 3 Seek the Global Maximum 33

Chapter 4 Less Is More: Reduce Choices 47

Chapter 5 Words Matter: Focus on Your Call to Action 59

Chapter 6 Fail Fast and Learn 71

PART II IMPLEMENTING A/B TESTING: PLAY-BY-PLAY GUIDE 83

Chapter 7 Choose the Solution That's Right for Your Organization 87

Chapter 8 The Cure for the HiPPO Syndrome 99

Chapter 9 The A/B Testing Dream Team 109

Chapter 10 Iterate, Iterate, Iterate 119

PART III ADVANCED TOPICS IN A/B TESTING 131

Chapter 11 How A/B Tests Can Go Awry 135

Chapter 12 Beyond the Page: Non-Website A/B Testing 143

Chapter 13 Personalize, Personalize, Personalize 155

Conclusion 165

Appendix 1: 60 Things to A/B Test 167

Appendix 2: Metrics and the Statistics behind A/B Testing 179

Acknowledgments 195

Index 197

About the Author

DAN SIROKER is the cofounder and CEO of Optimizely. The inspiration for Optimizely came from Dan’s experience as the Director of Analytics for the 2008 Obama Presidential campaign. Optimizely is the product Dan wishes he had back then to make it easy for anybody to do A/B testing. In 2012, Forbes named Dan one of the Top 30 under 30 in Technology. Dan was formerly a product manager for Google Chrome and AdWords.

PETE KOOMEN is the co-founder and President of Optimizely. After earning his MS in computer science from the University of Illinois at Urbana-Champaign, Pete joined Google as a product manager where he helped launch and grow Google App Engine to more than 150,000 developers. In 2009, Pete teamed up with Dan to start their first company, CarrotSticks, an online math game for kids. Less than a year later, Pete and Dan created Optimizely during the Y Combinator winter 2010 class. In between mountain climbing adventures, Pete runs product, engineering, and design at Optimizely.