Ron Chernow
Review
This book serves a healthy reminder that most Product Teams spend too much time building and shipping features. Exploratory testing gives us a chance to validate our assumptions before we commit to building. This book only scratches the surface of exploratory testing, but it’s a good introduction, and I recommend it to people getting into product management. I can recommend ‘Testing Business Ideas’ for a more in depth look at some of the techniques mentioned in this book.
You Might Also Like…
Key Takeaways
The 20% that gave me 80% of the value.
- There are two types of experimentation in Product Management
- Optimisation → making small changes to a live product. The results are measured and the better performing variant is preferred.
- Exploratory → validating key assumptions before building a solution.
- You can validate assumptions faster through experimentation than you can building software. Experiments help you move fast, increase the odds of success and make the most of limited resources. Talking to potential customers is also a good approach, see Talking with Humans for more about that.
- The high-level experiment process:
- Identify your key risks and assumptions - in your business model, product or feature
- There’s a number of frameworks you can use to tease out assumptions:
- Business Model Canvas by Alex Osterwalder
- Assumptions exercise from Talking to Humans
- Lean canvas by Ash Maurya
- Assumption Mapping by David Bland
- Prioritise the riskiest assumptions - (the hypothesis to test)
- Use a 2x2 of Impact and uncertainty
- Identify and sequence experiments
- You can ideate on how to test an assumption, just like you can ideate around features. Involve the team, generate as many ideas as you can - then pick the best
- Incorporate the results into your decision making process
- Don’t forget this one. It’s amazing how many teams experiment and measure - but the results don’t inform the future direction.
- Visualise your learnings from each experiment against each assumption - so you can see everything on a page.
- There’s a correlation between how much effort an experiment is - and how believable the results are. From a paper prototype to a live product or business. You can plot them on a ‘truth curve’
- The Five Traits of Good Experiments
- They are structured and planned. Use a template.
- They are focused. Test a core hypothesis, don’t try to do too much at once.
- They are believable. You can trust what you’re learning.
- They are flexible. Remain open to making small improvements as you go.
- They are compact. You can run them in an efficient amount of time.
- The Experiment Template
- What hypotheses do we want to prove / disprove?
- For each hypothesis, what quantifiable result indicates success? (Pass/Fail metrics)
- Who are the target participants of this experiment
- How many participants do we need?
- How are we going to get them?
- How do we run the experiment?
- How long does the experiment run for?
- Are there other qualitative things to learn during this experiment?
- Always be asking: How can we learn just as much with half the time and effort
- You can build a culture of experimentation in either a Top-Down or Bottom-Up
- Target execs that realise that success rate of initiatives is too low requires
- OR start a grass roots movement by experimenting where you can, and publicising the results
Deep Summary
Longer form notes, typically condensed, reworded and de-duplicated.
There are two types of experimentation in Product Management (this book focuses on the latter).
- Optimisation is about making small changes to a live product. The results are measured and the better performing variant is preferred.
- Exploratory is about validating key assumptions before building a solution.
Why Experiment?
Experimentation is valuable because it’s often a faster way to gain confidence in your idea than building and shipping software. Experiments help you move fast, increase the odds of success and make the most of limited resources. Talking to potential customers is also a good approach, see Talking with Humans for more about that.
You can de-risk a new feature, product or service. If you’ve made some big assumptions, test them before building your product. If you have the right mindset and are open to learning, insights from your experiments will help you refine your idea and improve your chance of success.
We run experiments in order to make better decisions. They gather crucial information that helps us create better strategies and take smarter actions in less time and with less cost. Experiments save time and cost. “Measure twice, cut once”. They are rarely a waste of time, there’s always something to learn about your potential customers.
Experiment Process
The Five Traits of Good Experiments
- They are structured and planned. Use a template.
- They are focused. Test a core hypothesis, don’t try to do too much at once.
- They are believable. You can trust what you’re learning.
- They are flexible. Remain open to making small improvements as you go.
- They are compact. You can run them in an efficient amount of time.
The Anatomy of an Experiment
A well-run experiment requires discipline. Casual chaotic experiments lead to chaotic data which is hard to use in decision making. Use a template or checklist to maintain standards.
Experiment Template
Always be asking: How can we learn just as much with half the time and effort
In a rush? Use a single sentence template. For [customer segment], we believe that [outcome] will happen when we run [experiment description]
10 Experiment Tips
- Test the most important risks and assumptions first.
- Balance testing the product with testing the business model.
- Get creative. Theres always another way to test.
- Sloppy experiments lead to sloppy results. Plan.
- Set success criteria ahead of time.
- Always be looking learn more with half the effort / time
- For larger experiments do a trial run.
- Combine with speaking to customers at the same time (customer research)
- Combine evidence and judgement to make smart decisions
- Choose experiment types that give believable results in the shortest time.
An Experiment Is Not Throwing Things Against the Wall. The best experiments have structure rather than chaos.
Experiment Archetypes
Testing Demand
Testing Products/ Features
How to build a culture of experimentation
Things to do to help your case
- Do experiments only when the need is great. Keep them small and tight.
- Communicate early and smartly to other groups
- Start with smaller tactical projects, aggregate the success into a larger narrative.
- Pull management into key meetings, make your learning process their learning process
- Control your narrative. Share what you learnt, and how. Draw connection back to the big goals.
Memorable Quotes
How can I learn about this starting today?
Requirements are actually hypotheses... realising this should be liberating. David Bland
How can we learn just as much with half the time and effort
Don’t build too much — that is always failure mode. Teams get so invested in what they are making for the experiment that they lose sight that their creation is just to learn
The experimentation game is validating key assumptions before building a solution