Product #67

super:Link

https://open.substack.com/pub/productandrew/p/product-67?r=12u3a4&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

Testing with Humans · Ron Chernow · 2018

This book serves a healthy reminder that most Product Teams spend too much time building and shipping features. Exploratory testing gives us a chance to validate our assumptions before we commit to building. It only scratches the surface of exploratory testing, but it’s a good introduction, and I recommend it to people getting into product management.

Key Points

There are two types of experimentation in Product Management:

Optimisation: Making small changes to a live product. Results are measured and the better performing variant is preferred.
Exploratory: Validating key assumptions before building a solution.

You can validate assumptions faster through experimentation than you can by building software. Experiments help you move fast, increase the odds of success and make the most of limited resources. Talking to potential customers is also a good approach, see ‘Talking with Humans’ for more on that.
The high-level experiment process:

Identify your key risks and assumptions - in your business model, product or feature

There’s a number of frameworks you can use to tease out assumptions:

Business Model Canvas by Alex Osterwalder
Assumptions exercise from Talking to Humans
Lean canvas by Ash Maurya
Assumption Mapping by David Bland

Prioritise the riskiest assumptions - (the hypothesis to test)

Use a 2x2 of Impact and uncertainty

Identify and sequence experiments

You can ideate on how to test an assumption, just like you can ideate around features. Involve the team, generate as many ideas as you can - then pick the best

Incorporate the results into your decision making process

Don’t forget this one → It’s amazing how many teams experiment and measure - but the results don’t inform the future direction.
Visualise your learnings from each experiment against each assumption - so you can see everything on a page.

There’s a correlation between how much effort an experiment is - and how believable the results are. From a paper prototype to a live product or business. You can plot them on a ‘truth curve’
The Five Traits of Good Experiments

They are structured and planned. Use a template.
They are focused. Test a core hypothesis, don’t try to do too much at once.
They are believable. You can trust what you’re learning.
They are flexible. Remain open to making small improvements as you go.
They are compact. You can get results quickly.

The Experiment Template:

What hypotheses do we want to prove / disprove?
For each hypothesis, what quantifiable result indicates success? (Pass/Fail metrics)
Who are the target participants of this experiment?
How many participants do we need?
How are we going to get them?
How do we run the experiment?
How long does the experiment run for?
Are there other qualitative things to learn during this experiment?

Always be asking: How can we learn just as much with half the time and effort?
You can build a culture of experimentation in either a Top-Down or Bottom-Up way:

Target execs that realise that success rate of initiatives is too low
OR start a grass roots movement by experimenting where you can, and publicising the results

Full Book Summary · Amazon

Subscribe Button

In the News

Nike invested billions of dollars into advertising that was less effective but easier to measure—a lesson for product managers everywhere· Article

A judge has ruled that Google maintained a monopoly in search. This isn’t surprising, but the implications could be profound for Alphabet. The last case of this magnitude was Microsoft in 2000, where the court ruled that Microsoft should be broken up. Microsoft skilfully avoided a forced breakup by making major concessions and agreeing to aggressive DOJ oversight. Will Alphabet be able to do the same? Article

Quick Links

4 common product discovery mistakes and how to avoid them · Article

A thread of different GTM frameworks · Tweet

How to figure out metrics ownership · Article

The ‘Documentation Tradeoff’ and agile alternatives · Article

Marty Cagan on the Product Model and Org Design · Article

Assumption Prioritisation Canvas - Identifying and testing key assumptions · Article

Marc Andreessen on the onion theory of risk — Can this be applied to product? · Video

A plea for the lost practice of information architecture · Article

Understanding the Sources of Information Systems Project Failure John McManus and A. Trevor Wood-Harper · 2007

Despite such failures, huge sums continue to be invested in information systems project and written off, for example the cost of project failure across the European Union was 142 billion Euros in 2004. 'Whilst our understanding of the importance of project failure has increased, many of the underlying reasons for failure still remain an issue and a point of contention for practitioners and academics alike. This paper examines through case research some of issues and casual factors of information systems project failure.

Engaging stakeholders well can be the difference between success and failure. Provide frequent updates and solicit constant feedback from yours.

Management issues cause 65% of IT project failures.
Technical shortcomings account for 35% of project failures.
Effective communication is critical for project success.
Lack of stakeholder and risk management contributes to failures.
Insufficient management support can derail projects.
Flawed estimation methods are a common pitfall.
Poor software requirements lead to technical challenges.
Unsuitable development tools can hinder project success.
Proper planning is essential, including contingency measures.
Technical support and user documentation are crucial post-deployment.

View the Paper

Book Highlights

If you deploy OKRs for your product organization, the key is to focus your OKRs at the product team level Marty Cagan · Inspired

It’s important to remember that the log data you collect is evidence, and you need to foster an objective view of what users do on the site. Translating it into ratings and opinions is a subjective process, which is something that has to be tweaked not only for each domain but also for each recommender algorithm Kim Falk · Practical Recommender Systems

How much flexibility is there in the Design Sprint recipe? Can we adjust the recipe? JK: Yes…but don’t adjust the recipe until you’ve tried the original. Jeff Gothelf and Josh Seiden (quoting Jake Knapp) · Lean UX

Trying to guess at what needs-based segments exist and which needs are unmet introduces risk and variability into the innovation process. This is why statistically valid quantitative research is an essential part of the ODI process. Anthony W. Ulwick · Jobs to be Done

An affordance is a relationship between the properties of an object and the capabilities of the agent that determine just how the object could possibly be used Don Norman · The Design of Everyday Things

Quotes & Tweets

We have this playbook for building things, and it goes like this: Phase one is creating something that sparks joy and is appreciated by people. Then, you aim for something retentive, where people have a good experience and want to come back. These two aspects aren't always the same; some things people find awesome might not bring them back regularly. For example, with things like ChatGPT, the technology is miraculous, but that doesn't mean everyone will use it every week.
The first step is creating that initial spark; the second is ensuring retention. Once you have retention, then you can start encouraging more people to join. If people aren't retained, why ask others to sign up?
So, step 1: spark. Step 2: retention. Step 3: growth and scaling the community. And only then comes step 4: monetization.
It takes time to go through all these phases. We're really only beginning to monetize experiences like WhatsApp with business messaging. But with two billion people using the product daily, we've scaled it pretty far. I think this model works well for us Zuck

Communication is about what is received, not what is intended. If there is a gap between what you are saying and what they are hearing, you have to find a new way to say it. James Clear