The Lean Product Playbook

The Lean Product Playbook

Author

Dan Olsen

Year
2015
image

Review

The Lean Product Playbook offers the clearest description of a process to achieve product-market fit. The book's primary premise is that there is a logical sequence to follow, which includes:

  1. Determining your target customers
  2. Identifying underserved customer needs
  3. Defining your value proposition
  4. Specifying your minimum viable product (MVP) feature set
  5. Creating your MVP prototype
  6. Testing your MVP with customers

The book is filled with practical advice. One example that stood out was the categorisation of feedback into three areas: functionality, UX, and messaging. This categorisation provides an incredibly practical framework for thinking about feedback and determining appropriate remedies, which is something I hadn't considered before.

The Lean Product Playbook is a great resource. The sections on design and delivery could have been clipped to maintain a tighter focus on the core topic of achieving product-market fit. If you weren’t approaching product development in the right order, then this book will pay for itself 1000 times over.

You Might Also Like:

image

Key Takeaways

The 20% that gave me 80% of the value.

  • Products fail when they don't meet customer needs in a way that’s better than alternatives.
  • Product-Market fit is when you’ve built a product that creates significant customer value.
  • There’s a logical sequential process in which to approach achieving product market fit (both for new products and new features)
image
  • The Lean Product Process consists of six steps:
    1. Determine your target customers
    2. Identify underserved customer needs
    3. Define your value proposition
    4. Specify your minimum viable product (MVP) feature set
    5. Create your MVP prototype
    6. Test your MVP with customers
  • Think of each layer as an opportunity to test and refine a hypothesis before moving on. Approach them in the right order, and reduce waste. The key is to avoid building more that you need to test the most important hypothesis.
  • Separate the problem space (customer needs) from the solution space (any representation of the product).Form a robust problem-space definition before starting product design. Articulate the hypotheses you’ve made and solicit customer feedback on early ideas to test those hypotheses.
  • Form a hypotheses about customer needs and the relative importance of them. Contextual inquiry or customer discovery can help you. Observe pain points, ask them what they like and don't like.
    • Critics of user-centered design quote Henry Ford: “If I had asked people what they wanted, they would have said a faster horse.” Yes customers won’t identify the next breakthrough solution, but we’re not expecting them to. Those who invoke Henry Ford do so as an excuse not to talk to customers, because they think they have all the answers.
  • That said: feedback from the solution space can inform your problem space hypotheses.
  • “Customers don't care about your solution. They care about their problems.”
  • Different customers will have different needs—and even those who have the same needs can have distinct views on their relative importance.
  • Develop a hypotheses about your target market.
  • Segment your target market by defining your target customer precisely (demographic, psychographic, behavioural, or needs). Personas can help. One-on-one interviews with customers are vital for understanding their needs and collecting data for personas.
  • Early adopters might have different needs and preferences to the early majority.
  • Focus on customers that use your product frequently and recommend it to others.
  • Customer needs, benefits, desires, wants, user goals, user stories, and pain points all essentially refer to what customers value. Pain points are needs that current solutions do not adequately meet, resulting in customer dissatisfaction.
  • Needs are layered like onions, with multiple levels that can be uncovered by asking probing questions. Benefits should be articulated precisely, starting with a verb, clearly conveying value, and often framed as increasing something desired or decreasing something undesired.
  • Keep asking "Why is that important to you?" until no new answers emerge.
  • The importance vs. satisfaction framework prioritises needs based on their potential to create customer value. Look for high importance + low satisfaction as they provide a significant opportunity to add value. Importance is a problem-space concept, while satisfaction is a solution-space concept.
  • The amount of customer value delivered by a product or feature can be calculated as importance * satisfaction (the area of the rectangle formed with the origin on an importance vs. satisfaction graph). The opportunity to create additional value is (importance * (1 - satisfaction)) or (maximum possible value - delivered value). Improvements generate value equal to the area of the incremental rectangle.
image
  • The Kano model categorises customer needs into three types forming hierarchy that must be addressed in order, with must-haves as the foundation:
    • must-haves (their absence causes dissatisfaction)
    • performance needs (more is better)
    • delighters (unexpected benefits that exceed expectations)
  • Once you've identified the important customer needs you could address, you must deliberately decide which ones your product will focus on. A good product addresses a set of needs that are both important and cohesive.
  • Identify a candidate for the minimum viable product (MVP) and test it to reveal incorrect assumptions. Each test helps you learn new information that refines your understanding with each iteration.
  • Strategy means deciding what your product won't do. Your product value proposition identifies the specific needs you'll address and how your product is better and different than alternatives.
  • Use the Kano model to classify needs as must-haves, performance benefits, or delighters in the context of your competitors. Articulate the benefits you plan to provide and how you aim to be better than competitors. Project forward in time, anticipating market trends and likely competitor moves, especially in rapidly changing markets.
  • For each benefit in your value proposition, brainstorm feature ideas. Transition from problem space to solution space. Capture the ideas, organise them by benefit, and prioritise the list for each benefit based on expected customer value.
  • Write features as user stories to keep the corresponding benefit clear.
  • Break features down into smaller chunks of functionality to reduce scope and build only the most valuable pieces. Smaller batch sizes increase velocity by enabling faster feedback, reducing risk and waste.
  • Estimate the effort for each user story using story points. Stories above a maximum threshold of points should be broken down into smaller stories below the threshold.
  • Prioritise feature chunks using Return on Investment return on investment.
  • Decide on the minimum set of functionality for your MVP candidate. Include all must-haves, enough of the main performance benefit to beat the competition, and the top delighter for differentiation.
  • Limit planning to 1-2 versions ahead, as much will change after first customer feedback.
  • Test your MVP with customers by creating a UX prototype. The goal is to build a prototype for testing, not the actual product to enable faster learning with fewer resources. The process applies to entire products, new features, or redesigns.
  • The term "MVP tests" is more precise than "MVP" and avoids debate over what constitutes an MVP. Limit scope but aim to deliver customer value.
  • Marketing tests gauge interest in the product description. Product tests solicit feedback on actual functionality.
  • Quantitative tests involve many customers and yield aggregate results on what actions were taken. Qualitative tests involve fewer customers but provide insights into why they took certain actions.
  • Marketing MVP Tests:
    • Qualitative Tests: These involve soliciting feedback on marketing materials such as landing pages, videos, ads, and emails.
    • Quantitative Tests: These tests, such as landing page/smoke tests, explainer videos, Ad campaigns, and A/B testing, validate demand and optimize customer acquisition and conversion.
  • Product MVP Tests:
    • Qualitative Tests: These tests assess and improve product-market fit by testing the product's design with customers before and after building it. This includes testing design deliverables before coding, testing live product after coding, and using Wizard of Oz and concierge MVPs, and fake door/404 page tests.
    • Quantitative Tests: These tests, such as product analytics and A/B Testing, provide numerical data for analysis.
  • The UX Design Iceberg framework has four layers: conceptual design (underlying concept), information architecture (structure), interaction design (user flows and feedback), and visual design (look and feel).
  • Lean emphasises quick learning and iteration to modify hypotheses and MVPs based on customer feedback, improving product-market fit with each round.
  • The Hypothesize-Design-Test-Learn loop starts with problem space hypotheses, transitions to solution space with design artefacts or products, tests with customers, learns from observations, and revises hypotheses.
  • Refer to the Product-Market Fit Pyramid when validating and invalidating hypotheses. Address issues at the lowest level first before moving up.
  • Capture key observations from each user testing wave in a table, prioritising changes based on the most common feedback.
  • No hard rule determines when an MVP is validated "enough." Testing with live products is better than artefacts, as actual behaviour trumps opinions.
  • If iterations don't yield progress, map problems to the Product-Market Fit Pyramid. Make sure you’re not Iterating at the wrong level.
  • Consider pivoting if target customers are lukewarm on the MVP or no customer archetype is excited about it. Tests can clarify the best pivot direction
  • Build your product using agile development (iterative and incremental methodologies to deliver working software early and continuously, focusing on customer value). You’ll be able to: react to changes quickly, get earlier customer feedback and reduce estimation error through smaller batch sizes.
  • Before launch, new products rely on qualitative research with prospective customers. After launch, quantitative learning methods (analytics and A/B testing) become available.
  • Dave McClure's "Startup Metrics for Pirates" framework (AARRR) includes acquisition, activation, retention, revenue, and referral. Tracking 2-3 key metrics for each element is recommended.
  • The "metric that matters most" (MTMM) changes over time due to diminishing returns. For new products, the typical MTMM order is retention (product-market fit), conversion (optimising the funnel), and acquisition (attracting prospects).
  • Retention rate is the best metric to measure product-market fit. Retention curves visualise customer retention over time, with key parameters being initial drop-off rate, rate of descent, and terminal value. Improving these parameters indicates stronger product-market fit.
  • Cohort analysis examines metrics for different user groups (cohorts) over time. Improving product-market fit is reflected in newer cohorts having higher retention curves.
  • The equation of your business (revenue = number of paying users × average revenue per user) can be analysed on a per-user basis to optimise the business model.
  • Customer lifetime value (LTV) is the profit a customer generates without considering acquisition costs. When LTV exceeds customer acquisition cost (CAC), each new customer is profitable. Increasing LTV involves increasing average revenue per user (ARPU) and decreasing churn rate.
  • Companies that learn from customers and iterate quickly have a competitive advantage. Speed becomes a powerful weapon in today's fast-paced world.
  • The Lean Product Analytics Process:
    1. Define key metrics for the business
    2. Measure metrics to establish baseline values
    3. Evaluate each metric's upside potential and ease of improvement
    4. Select the "metric that matters most" (MTMM) based on the most promising opportunities
    5. Brainstorm improvement ideas for the top metric and estimate their impact (form hypotheses)
    6. Choose the highest ROI idea to pursue
    7. Design and implement the top improvement idea, ideally using A/B testing
    8. Iterate and improve the metric until diminishing returns, then identify the next MTMM
  • Be cautious of getting stuck at a local maximum when improving a metric. Consider completely different alternatives to break through and achieve further improvements.
  • A/B testing is a powerful evidence-based decision-making tool, but it should be complemented with qualitative learning to understand the "whys" behind user behavior.
  • The Product-Market Fit Pyramid's layers are interconnected. Changing foundational hypotheses (target customers, underserved benefits, value proposition) after building the product is like an earthquake, requiring rebuilding from scratch.
  • The Lean Product Process sequence validates key hypotheses in an order that reduces risk and increases the odds of achieving product-market fit.
  • The problem space requires more qualitative research to create, test, and improve hypotheses, while the solution space is more amenable to quantitative A/B testing.
  • There is a natural progression from qualitative learning (defining the product) to quantitative learning (optimising the product) after launch. Both are necessary for creating a successful product.
image

Deep Summary

Longer form notes, typically condensed, reworded and de-duplicated.

Introduction

  • Most new products fail, usually because they don't meet customer needs in a way that’s better than alternatives. They never get to product-market fit.
  • The Product-Market Fit Pyramid builds on The Lean Startup principles. It has five key components: your target customer, your customer's underserved needs, your value proposition, your feature set, and your user experience (UX).Each of these is actually a testable hypothesis and can be logically sequences as below.

Part I Core Concepts

Chapter 1: Achieving Product-Market Fit

Product-market fit means being in a good market with a product that can satisfy that market. Marc Andreessen (who coined ‘product-market fit)
  • Product Market-Fit: you have built a product that creates significant customer value. Your product meets real customer needs and does so in a way that is better than the alternatives.
  • Product Market-Fit is not about being profitable. There’s a distinction between creating value and capturing it. Clearly you must capture it first.

The Product-Market Fit Pyramid

image
  • A market consists of all the existing and potential customers that share a particular customer need to set of related needs.
  • A product is a specific offering intended to meet a set of customer needs. PMF applies to services as well as products.
  • Within the product and market sections, each layer depends on the layer immediately beneath it.
  • UX is how your users experience your functionality. Each feature aims to serve a customer need. The value proposition is the set of needs you aspire to meet with your product. You should aim to serve the underserved needs of your target customers.
  • You want to address customer needs that aren't adequately met. Customers judge your product relative to available alternatives in the competitive landscape.
  • Product-market fit then is the measure of how well your product (the top three layers of the pyramid) satisfies the market (the bottom two layers of the pyramid).
  • The Lean Product Process consists of six steps:
    1. Determine your target customers
    2. Identify underserved customer needs
    3. Define your value proposition
    4. Specify your minimum viable product (MVP) feature set
    5. Create your MVP prototype
    6. Test your MVP with customers
  • The process guides you through each layer of the pyramid from the bottom up, helping you articulate and test your key hypotheses in an efficient way (you want to avoid building more than you need to test your hypotheses).
  • The process works for making product improvements as well as creating new products.
  • You can learn faster with fewer resources by testing your hypotheses before you build your product.
  • You don’t need to repeat every step each time. Once you've successfully identified target customers and needs, you may not need to revisit those areas for a while.
  • The process increases your chances of achieving product-market fit by encouraging a certain amount of rigour in product thinking. It helps you think about the key assumptions and decisions to be made when creating a product.
  • If you’re not making these assumptions or decisions explicitly, then you are making them implicitly.
  • The Lean Product Process will reduce rework, but it won’t eliminate it as the process is still iterative and will require you to revise your hypotheses, designs, and product as you go.
  • The goal of the process is to help you achieve product-market fit as quickly as possible.

Chapter 2 Problem Space versus Solution Space

  • Separate the problem space from solution space.
    • The solution space includes any representation of a product that is used by or intended for use by a customer. It is the opposite of a blank slate. You’ve chosen a specific implementation. You've determined how the product looks, what it does, and how it works.
    • In the problem space there is no product or design. It’s where all the customer needs that you'd like your product to deliver live. A ‘need’ can be a customer pain point, a desire, a job to be done, or a user story.
  • Problems define markets.
  • Sometimes your biggest competitor isn’t a direct competitor, but an alternative. E.g. a note taking app vs pen and paper.
  • A market is a set of related customer needs, and isn’t tied to any specific solutions that meet those needs. Market disruptions occur when a new type of product (solution space) better meets the market needs (problem space).
  • Learn to write detailed product requirements that stay in the problem space. Focus on “what” the product needs to accomplish for customers.
  • “What” is problem space and “how” is the solution space.
  • Outside-in product teams start by understanding the customer's problem space. Talk with them to understand their needs and how they feel about existing solutions. Form a robust problem-space definition before starting product design. Articulate the hypotheses you’ve made and solicit customer feedback on early ideas to test those hypotheses.
  • Critics of user-centered design quote Henry Ford: “If I had asked people what they wanted, they would have said a faster horse.” Yes customers won’t identify the next breakthrough solution, but we’re not expecting them to. Those who invoke Henry Ford do so as an excuse not to talk to customers, because they think they have all the answers.
  • A good understanding of customer needs helps us explore new solutions and estimate how valuable they’ll find them.
  • You've got to start with the customer experience and work backwards to the technology. You can't start with the technology and try to figure out…
  • Apple has a reputation for not soliciting customer feedback but they have an in-depth understanding of customer needs (passcode → Touch ID → Face ID)

You can use the Solution Space to Discover the Problem Space

  • Customers struggle to articulate their needs. It’s the product team's job to unearth them.
  • Form a hypotheses about customer needs and the relative importance of them. Contextual inquiry or customer discovery (interviewing and observing customers) can help you. Observe pain points, ask them what they like and don't like.
  • Feedback from the solution space can inform your problem space hypotheses. Having solution space discussions with customers is more fruitful than trying to discuss the problem space with them. They’re better at articulating what they like (and don’t like) about your product and making comparisons to other solutions.
  • “Customers don't care about your solution. They care about their problems.”

Part II The Lean Product Process

Chapter 3 Determine Your Target Customer (Step 1)

  • Different customers will have different needs—and even those who have the same needs can have distinct views on their relative importance.
  • Develop a hypotheses about your target market, once you have a prototype show customers to gain clarity about the target market you're attracting.
  • If the needs of an adjacent market are similar, you may only need to make minor changes.
  • Market segmentation: dividing a market into subsets based on attributes. Segment your target market by defining your target customer precisely (demographic, psychographic, behavioural, or needs).
  • Psychographic attributes are often more useful than demographics. Relevant behavioural attributes (whether or not someone takes a particular action or how frequently they do) or needs-based segmentation can be powerful too.
  • For B2B products distinguish users from buyers, they’ll have different needs.
  • The technology adoption life cycle divides a market into distinct customer segments: Innovators, early adopters, early majority, late majority, laggards.
    • There’s a chasm between early adopters and the early majority. As you try to gain adoption by additional segments over time, you will discover that they have different needs and preferences — that require you to change your product before they will adopt it.
  • Personas describe your target customer precisely, and what they want to accomplish. They are hypothetical archetypes of actual users. Use them early in your product process as they’re a good way to capture your hypotheses. They can help align a company to focus on the same customer and empower them to make decisions based on user needs.
    • Include: name, job title, key demographics, primary needs/goals, relevant behaviors, frustrations with current solutions, level of domain expertise, usage context, and technology adoption stage.
  • Conducting one-on-one interviews with customers is vital for understanding their needs and collecting data for personas. Avoid creating personas based on averages or speculation, and instead ensure they represent real individuals. Regularly communicate with customers to stay in touch with their needs, especially with the ones who appreciate your product most. This continuous interaction aids in refining target market hypotheses and uncovering underserved needs.
  • Focus on customers that use your product frequently and recommend it to others.

Chapter 4 Identify Underserved Customer Needs (Step 2)

  • Customer needs, benefits, desires, wants, user goals, user stories, and pain points all essentially refer to what customers value. Pain points are needs that current solutions do not adequately meet, resulting in customer dissatisfaction.
  • Needs are layered like onions, with multiple levels that can be uncovered by asking probing questions. Benefits should be articulated precisely, starting with a verb, clearly conveying value, and often framed as increasing something desired or decreasing something undesired.
  • Customer discovery interviews test benefit hypotheses by asking customers what the benefit means to them, how it might help, how valuable it would be, and the reasons why it is or isn't valuable. Different customers may describe the same idea using different words.
  • Laddering interviews climb from detailed, granular benefits to higher-level, more abstract ones by repeatedly asking "Why is that important to you?" until no new answers emerge. Needs can have hierarchies where lower-level needs must be met before higher-level ones matter, similar to Maslow's hierarchy of human needs.
image
  • The importance vs. satisfaction framework prioritises needs based on their potential to create customer value. Look for high importance + low satisfaction as they provide a significant opportunity to add value. Importance is a problem-space concept, while satisfaction is a solution-space concept.
  • Disruptive innovation provides a 10x improvement in customer value compared to existing alternatives, while incremental innovation adds small amounts of value incrementally with each new product version. Measuring importance and satisfaction via surveys enables quantitative analysis, even with small sample sizes ("quant on qual").
  • Gap analysis calculates the difference between importance and satisfaction ratings. The bigger the gap, the more underserved the need, but this method treats all equal-sized gaps the same. Jobs-to-be-done methodology refines this with an opportunity score that adds importance as a tiebreaker for gaps of equal size.
  • The jobs-to-be-done approach focuses on identifying the desired outcomes customers want to achieve when "hiring" a product or service to complete a task or job. Precisely defining 50-150 detailed outcomes or benefits, not just a handful, creates a rich problem space definition that is essential for driving meaningful innovation.
  • The amount of customer value delivered by a product or feature can be calculated as importance * satisfaction (the area of the rectangle formed with the origin on an importance vs. satisfaction graph). The opportunity to create additional value is (importance * (1 - satisfaction)) or (maximum possible value - delivered value). Improvements generate value equal to the area of the incremental rectangle.
image
  • The Kano model categorises customer needs into three types forming hierarchy that must be addressed in order, with must-haves as the foundation:
    • must-haves (their absence causes dissatisfaction)
    • performance needs (more is better)
    • delighters (unexpected benefits that exceed expectations)
  • Applying these frameworks helps product teams thoroughly explain customer benefits and systematically identify opportunities to create the most customer value by focusing on important needs that are currently underserved by existing solutions in the market.

Chapter 5 Define Your Value Proposition (Step 3)

  • Once you've identified the important customer needs you could address, you must deliberately decide which ones your product will focus on. A good product addresses a set of needs that are both important and cohesive.
  • Adding too many features leads to a bloated, less usable, and less valuable product. Focus is critical, especially when defining a new product, to avoid wasting resources on an overly large initial scope.
  • There is inherent uncertainty in your hypotheses and knowledge, which is why you start by identifying the minimum viable product (MVP). Testing the MVP may reveal incorrect assumptions, requiring revisiting your problem space hypotheses. Even if you're heading in the right direction, you'll learn new information that refines your understanding with each iteration.
  • Strategy means deciding what your product won't do. Your product value proposition identifies the specific needs you'll address and how your product is better and different than alternatives.
  • Use the Kano model to classify needs as must-haves, performance benefits, or delighters in the context of your competitors. Must-haves are required but not core to your value proposition since all products in the category have them. Your core value lies in the performance benefits you choose to compete on and the unique delighters you provide.
  • To beat an incumbent market leader, your value proposition must at least match them on the important performance benefits they deliver.
  • Create a value proposition template listing the relevant must-haves, performance benefits, and delighters, with columns for each competitor and your product. Score each row for the competitors and your planned product using "Yes" for must-haves, a qualitative or quantitative scale for performance benefits, and "Yes" for applicable unique delighters.
  • image
  • The completed grid articulates the benefits you plan to provide and how you aim to be better than competitors. To be strategic, project forward in time, anticipating market trends and likely competitor moves, especially in rapidly changing markets. Use separate "now" and "later" columns for each competitor and your product.
  • Analysing your product strategy in this way ensures you're not just solving for current conditions and reduces the risk of heading down a path that will be suboptimal in the future.

Chapter 6 Specify Your Minimum Viable Product (MVP) Feature Set (Step 4)

  • For your MVP, identify the minimum functionality required to validate that you're heading in the right direction. This is an MVP candidate, as it's based on hypotheses not yet validated with customers.
  • For each benefit in your value proposition, brainstorm as many feature ideas as possible for how your product could deliver that benefit. This is the transition from problem space to solution space. Practice divergent thinking, generating ideas without judgment.
  • Capture all the ideas, organise them by benefit, and prioritise the list for each benefit based on expected customer value. Identify the top 3-5 features per benefit, as things will change after showing the prototype to customers.
  • Write features as user stories to keep the corresponding benefit clear, using the template: "As a [user], I want to [action], so that I can [benefit]." Good user stories are Independent, Negotiable, Valuable, Estimable, Small, and Testable (INVEST).
  • Break features down into smaller chunks of functionality to reduce scope and build only the most valuable pieces. Smaller batch sizes increase velocity by enabling faster feedback, reducing risk and waste.
  • Estimate the effort for each user story using story points. Stories above a maximum threshold of points should be broken down into smaller stories below the threshold.
  • Prioritise feature chunks using Return on Investment (ROI): the ratio of return (a measure of customer value) to investment (development time). Focus on highest ROI features first and avoid lower ROI ones. Approximating ROI by scoring value and effort as high/medium/low works if numerical estimates are difficult.
image
  • Create a grid listing benefits and the top prioritised feature chunks for each. Decide on the minimum set of functionality for your MVP candidate, including all must-haves, enough of the main performance benefit to beat competition, and the top delighter for differentiation.
  • Limit planning to 1-2 versions ahead, as much will change after first customer feedback. Reflect on the rigorous thinking done to this point: hypotheses about target customers, their underserved needs, your planned value proposition, top feature ideas broken into chunks, prioritisation based on ROI, and MVP candidate selection.
  • The MVP is still a bundle of interrelated hypotheses. Before testing with customers, create a solution-space representation of the MVP candidate to show them, which is the next step.

Chapter 7 Create Your MVP…

  • Once you've specified the MVP candidate's feature set, test it with customers by creating a user experience (UX) prototype. The goal is to build a prototype for testing, not the actual product.
  • Testing prototypes before building the live MVP enables faster learning with fewer resources. The process applies to entire products, new features, or redesigns.
  • The term "MVP tests" is more precise than "MVP" and avoids debate over what constitutes an MVP. MVPs should be limited in scope but still deliver customer value above a minimum threshold.
  • MVP tests can validate either product or marketing. Marketing tests gauge interest in the product description, while product tests solicit feedback on actual functionality.
  • MVP tests can also be quantitative or qualitative. Quantitative tests involve many customers and yield aggregate results on what actions were taken. Qualitative tests involve fewer customers but provide insights into why they took certain actions. Both types are valuable and often alternated.
  • Marketing MVP tests include showing customers marketing materials (landing pages, videos, ads, emails) and soliciting feedback on how compelling they find the value proposition. Five-second tests assess how well messaging conveys the product's purpose and appeal.
  • Quantitative marketing MVP tests validate demand and optimise customer acquisition and conversion. They include:
    • Landing page/smoke tests: Measure conversion rate from a web page describing the planned product
    • Explainer videos: Judge effectiveness by the conversion rate driven by the video
    • Ad campaigns: Test different search terms, ad copy, messaging, and imagery to increase clickthrough rates
    • A/B testing: Compare the performance of two alternative designs on a key metric simultaneously
    • Crowdfunding: Validate willingness to pay and quantify demand by accepting pre-orders before building the product
  • Qualitative product MVP tests assess and improve product-market fit by testing the product's design with customers before and after building it. They include:
    • Testing design deliverables before coding: Hand sketches (low fidelity, for internal use), wireframes (medium fidelity, can be static or clickable), mockups (high fidelity, can be static or clickable), and interactive prototypes (highest fidelity without being live)
    • Testing live product after coding: Moderated (with the customer) or unmoderated (recorded), in-person or remote; may uncover issues not found in lower-fidelity tests
    • Wizard of Oz and concierge MVPs: Test a live product or service using manual workarounds to validate what it should do before building it; best for services with high customer interaction
    • Fake door/404 page tests: Validate demand for a potential new feature by measuring the percentage of customers who click a link to it and showing a "coming soon" message
  • Quantitative product MVP tests include:
    • Product analytics: Provide insights into actual customer usage patterns
    • A/B testing: Compare the performance of two alternative user experiences within the product on a key metric
Qualitative Tests
Quantitative Tests
Marketing Tests
Marketing materials
Landing page/Smoke test, Explainer video, Ad campaign, Marketing A/B tests, Crowdfunding
Product Tests
Wireframes, Mockups, Interactive prototype, Wizard of Oz & Concierge, Live product
Fake door/404 page, Product analytics & A/B tests
  • The matrix of MVP tests categorises options along the product vs. marketing and qualitative vs. quantitative dimensions, helping teams choose appropriate tests for their learning goals.

Chapter 8 Apply the Principles of Great UX Design

  • User experience (UX) design is critical for achieving product-market fit. A great UX feels easy to use, enables users to focus on their tasks, and may evoke positive emotions. It helps convey the desired customer benefits and achieves high usability and delight.
  • Usability indicates how easy the product is to use, focusing on users' goals and tasks. It includes the percentage of users who can successfully complete tasks, the effort required (clicks, taps, keystrokes, time), and perceived ease of use. Usability expectations vary based on the target users' knowledge and skills.
  • Delight goes beyond usability, evoking positive emotions through aesthetics, simplicity, anticipating user needs, personality, dynamic responses, and surprise.
  • The UX Design Iceberg framework has four layers: conceptual design (underlying concept), information architecture (structure), interaction design (user flows and feedback), and visual design (look and feel).
image
  • Conceptual design should resonate with target customers' thinking. User research techniques like interviews, tests, and surveys help gain a deep understanding of users. Personas, archetypes of actual users, inform design decisions.
  • Information architecture (IA) defines the structure of information and functionality. Card sorting helps identify users' preferred organisation. Good IA results in intuitive organisation and find-ability. Sitemaps define product pages and their relationships.
  • Interaction design specifies user flows, actions, and product responses. It includes form design, operating modes, states, error messages, and response time. Flowcharts are the primary artefact for defining user flows.
  • Wireframes represent page/screen layouts without visual design details. They are solution space artefacts that can be tested with users. Designers identify and reuse common page/screen patterns.
  • Visual design includes colour, typography, and graphics. Colour palettes, typefaces, font sizes, images, illustrations, icons, and other elements contribute to aesthetics, hierarchy, and usability. Style guides ensure consistency, and layout grids enable precise alignment.
  • Mockups are high-fidelity design deliverables that build on wireframes by adding visual design.
  • Design principles include Gestalt principles (proximity and similarity), visual hierarchy (size, colour, location), and composition (unity, contrast, balance, use of space).
  • Responsive design adapts the UI to different screen sizes using breakpoints and dynamic styling changes.
  • Copy (text) in marketing pages and the product itself impacts conversion rates and usability. Clear, simple language is important.
  • UX design requires multiple skills, and many teams have a "design gap." Designers often specialise in either visual or interaction design.

Chapter 9 Test Your MVP with Customers (Step 6)

  • Quantitative tests (A/B tests, landing page tests) are straightforward, tracking conversion rates or other metrics against target values. Qualitative user testing identifies unknown issues and blind spots, validating or invalidating hypotheses.
  • One-on-one user tests yield the best results. Group dynamics can lead to biased data. Testing in waves of 5-8 users balances uncovering issues with diminishing returns.
  • In-person testing allows for richer data collection (facial expressions, eye tracking) and rapport building. Remote moderated testing reaches distant users. Unmoderated remote testing is quicker but lacks the ability to ask follow-up questions.
  • Screeners ensure the right target customers are tested. Demographics, behaviors, and psychographics should align with personas. Remote testing and research companies help with recruiting.
  • Scheduling tests in advance, regardless of readiness, prevents delays. Guerrilla testing (e.g., at cafes or malls) is low-cost and immediate but less controlled. Compensation ranges from $75-$125/hour, via gift cards or product credit.
  • Observing users in their natural context ("follow me homes") provides valuable insights. Ramen user testing eliminates nonessentials, using office space and team members.
  • Encourage the team to observe tests together to align on problems and solutions. Record in-person tests only if necessary.
  • Prepare a test script with artefacts, tasks, and questions. Conduct a pilot test first. Structure: warm-up/discovery (10-15 min), product feedback (40-45 min), wrap-up (5-10 min).
  • Set expectations for honest, critical feedback. Encourage think-aloud protocol. Avoid leading questions and excessive intervention. Answer questions with questions to probe deeper.
  • Use open-ended questions (why, how, what) instead of closed-ended (yes/no). Practice mentally before asking. Let users struggle to uncover issues.
  • In the wrap-up, ask for overall impressions, ratings, and likelihood to use. Offer a form for less biased responses. Answer user questions and provide compensation. Gauge interest in future research or product availability.
  • Map feedback to functionality, UX, and messaging. Progress is indicated by more positive feedback, absence of prior complaints, and rising ratings.
  • Differentiate between usability (ease of use) and product-market fit (value) feedback. Poor usability or misaligned messaging can mask product-market fit.
  • Ensure the right target customers are tested to avoid misleading data. Revisit value proposition hypotheses if product-market fit is lacking after addressing usability issues.

Chapter 10 Iterate and Pivot to Improve Product-Market Fit

  • Lean emphasises quick learning and iteration to modify hypotheses and MVPs based on customer feedback, improving product-market fit with each round.
  • The Build-Measure-Learn loop has nuances:
    • "Build" is better described as "design something to test" to consume the least resources.
    • "Measure" includes qualitative observations, not just numerical data. "Test" is a broader label.
    • "Learn" involves both gaining new insights and modifying hypotheses. Splitting it into "learn" and "hypothesise" clarifies the process.
  • The Hypothesize-Design-Test-Learn loop starts with problem space hypotheses, transitions to solution space with design artefacts or products, tests with customers, learns from observations, and revises hypotheses.
  • Testing problem space thinking with solution space artefacts and feedback enables quicker learning and customer value delivery.
  • Refer to the Product-Market Fit Pyramid when validating and invalidating hypotheses. Address issues at the lowest level first before moving up.
  • image
  • Capture key observations from each user testing wave in a table, prioritising changes based on feedback percentages.
  • No hard rule determines when an MVP is validated "enough." Testing with live products is better than artefacts, as actual behaviour trumps opinions.
  • If iterations don't yield progress, map problems to the Product-Market Fit Pyramid. Iterating at the wrong level may occur.
  • Pivots are significant changes in direction when main hypotheses are changed. Deciding to persevere, pivot, or stop is challenging, often constrained by resources.
  • Consider pivoting if target customers are lukewarm on the MVP or no customer archetype is excited about it. Tests can clarify the best pivot direction.
  • Mountain climbing analogy for product-market fit and pivoting:
    • Each mountain represents a market opportunity based on target market and value proposition hypotheses. Higher climb means stronger fit.
    • Iterations improve fit, but gains may diminish. Adjacent opportunities (taller mountains) may be discovered through testing.
    • Revising hypotheses and pivoting can lead to higher product-market fit in the new opportunity.

Part III Building and Optimising Your Product

Chapter 12 Build Your Product Using Agile Development

  • Agile development uses iterative and incremental methodologies to deliver working software early and continuously, focusing on customer value. Benefits include faster reaction to changes, earlier customer feedback, and reduced estimation error through smaller batch sizes.
  • The cone of uncertainty shows the range of expected estimation error over a project's life. Unknown unknowns cause asymmetric estimation errors, with tasks taking longer than estimated. Breaking large tasks into smaller ones reduces unknown unknowns.
  • Agile is better for most software projects, while waterfall is suitable for high-risk projects with less room for change (e.g., spacecraft). Agile promotes cross-functional collaboration, flexibility, small batch sizes, short iterations, and continuous improvement.
  • Scrum is a popular Agile methodology that works in time-boxed sprints (usually 2-4 weeks). Work comes from a prioritized product backlog of user stories. The ideal team size is 5-9 people.
  • Sprint planning involves backlog grooming, story point estimation (e.g., Fibonacci, powers of two, T-shirt sizes), and determining the team's velocity (points completed per sprint). Planning Poker is a group estimation technique.
  • Daily Scrum meetings (standups) and burndown charts track progress. QA testing is done during the sprint, aiming for a shippable product increment. Sprint reviews (demos) and retrospectives focus on improvement.
  • Kanban visualizes work using boards with columns for different states (e.g., backlog, ready, in development, testing, deployed). Work in progress (WIP) limits constrain the number of items per column. Swimlanes can represent epics, user stories, team members, or projects.
  • Kanban measures throughput (items completed per time), cycle time (start to delivery), and lead time (request to delivery). Cumulative flow diagrams visualize work flow. Continuous improvement aims to reduce lead and cycle times.
  • Succeeding with Agile requires cross-functional collaboration, ruthless prioritization, staying ahead of developers, breaking stories down, and quality assurance (manual and automated testing).
  • Test-driven development (TDD) involves writing automated tests before coding, leading to higher test coverage and fewer regression bugs.
  • Continuous integration automatically builds and tests the product based on the latest code commits, identifying issues sooner. Continuous deployment automatically deploys code that passes all tests, with the ability to revert if problems arise (automated rollback).

Chapter 13 Measure Your Key Metrics

  • Before launch, new products rely on qualitative research with prospective customers. After launch, quantitative learning methods (analytics and A/B testing) become available.
  • Attitudinal information is what customers say about their attitudes and opinions, while behavioural information is what they actually do. Qualitative research helps understand why customers behave a certain way, while quantitative research reveals how many customers exhibit that behaviour.
  • User interviews (qualitative, attitudinal) aim to understand users' needs, preferences, and problem-solving approaches. Usability testing (qualitative, behavioural) focuses on observing users interacting with a product or prototype.
  • Surveys (quantitative, attitudinal) gather data from many users but are limited by respondents' knowledge and the wording of questions. Behavioural data is often more reliable than attitudinal data.
  • Net Promoter Score (NPS) is a widely used survey-based metric that measures customer satisfaction and product-market fit. Sean Ellis' product-market fit question assesses fit by asking how disappointed users would be if they could no longer use the product.
  • Analytics and A/B testing (quantitative, behavioural) measure actual customer behaviour at scale, enabling statistically significant conclusions and confident assessment of changes.
  • The author's analytics framework covers acquisition, conversion, retention, and revenue. Quantitative data identifies problems, while qualitative data provides insights to address them.
  • Dave McClure's "Startup Metrics for Pirates" framework (AARRR) includes acquisition, activation, retention, revenue, and referral. Tracking 2-3 key metrics for each element is recommended.
  • The "metric that matters most" (MTMM) changes over time due to diminishing returns. For new products, the typical MTMM order is retention (product-market fit), conversion (optimising the funnel), and acquisition (attracting prospects).
  • Retention rate is the best metric to measure product-market fit. Retention curves visualise customer retention over time, with key parameters being initial drop-off rate, rate of descent, and terminal value. Improving these parameters indicates stronger product-market fit.
  • Cohort analysis examines metrics for different user groups (cohorts) over time. Improving product-market fit is reflected in newer cohorts having higher retention curves.
  • The equation of your business (revenue = number of paying users × average revenue per user) can be analysed on a per-user basis to optimise the business model.
  • Customer lifetime value (LTV) is the profit a customer generates without considering acquisition costs. When LTV exceeds customer acquisition cost (CAC), each new customer is profitable. Increasing LTV involves increasing average revenue per user (ARPU) and decreasing churn rate.

Chapter 14 Use Analytics to Optimise Your Product and Business

  • Companies that learn from customers and iterate quickly have a competitive advantage. Speed becomes a powerful weapon in today's fast-paced world.
  • The Lean Product Analytics Process:
    1. Define key metrics for the business
    2. Measure metrics to establish baseline values
    3. Evaluate each metric's upside potential and ease of improvement
    4. Select the "metric that matters most" (MTMM) based on the most promising opportunities
    5. Brainstorm improvement ideas for the top metric and estimate their impact (form hypotheses)
    6. Choose the highest ROI idea to pursue
    7. Design and implement the top improvement idea, ideally using A/B testing
    8. Iterate and improve the metric until diminishing returns, then identify the next MTMM
  • Be cautious of getting stuck at a local maximum when improving a metric. Consider completely different alternatives to break through and achieve further improvements.
  • A/B testing is a powerful evidence-based decision-making tool, but it should be complemented with qualitative learning to understand the "whys" behind user behavior.
  • Launching an MVP and relying solely on A/B testing to achieve product-market fit is likely to waste resources and fail. Qualitative testing and learning are crucial in the pre-MVP stage.
  • The Product-Market Fit Pyramid's layers are interconnected. Changing foundational hypotheses (target customers, underserved benefits, value proposition) after building the product is like an earthquake, requiring rebuilding from scratch.
  • The Lean Product Process sequence validates key hypotheses in an order that reduces risk and increases the odds of achieving product-market fit.
  • The problem space requires more qualitative research to create, test, and improve hypotheses, while the solution space is more amenable to quantitative A/B testing.
  • There is a natural progression from qualitative learning (defining the product) to quantitative learning (optimising the product) after launch. Both are necessary for creating a successful product.

Chapter 15 Conclusion

  • The Lean Product Process guides you through the formulation and testing of your hypotheses with these six steps:
    1. Determine your target customers
    2. Identify underserved customer needs
    3. Define your value proposition
    4. Specify your minimum viable product (MVP) feature set
    5. Create your MVP prototype
    6. Test your MVP with customers.
  • 10 best practices for creating successful products
    1. Have a point of view but stay open-minded.
    2. Articulate your hypotheses.
    3. Prioritise ruthlessly.
    4. Keep your scope small but focused.
    5. Talk to customers.
    6. Test before you build.
    7. Avoid a local maximum.
    8. Try out promising tools and techniques.
    9. Ensure your team has the right skills.
    10. Cultivate your team's collaboration.