Review

"How to Measure Anything" by Douglas W. Hubbard is a must-read for product managers, despite not being directly related to product development. This book provides invaluable insights into the art and science of measurement, which is crucial for successful product discovery. As product managers, we often face uncertainty when exploring new ideas and validating assumptions. Hubbard emphasizes that measurement is about reducing uncertainty, and even small amounts of early observations can be incredibly informative in the face of uncertainty. This concept is particularly relevant to product discovery, where we continuously seek to learn and make informed decisions based on data. Throughout the book, Hubbard presents a framework for quantifying seemingly immeasurable things, such as user preferences, market demand, and the potential impact of new features. By applying these techniques, product managers can make more accurate predictions, prioritize effectively, and allocate resources wisely. One of the key takeaways from this book is that measurement doesn't always require precise data or extensive analysis. Often, rough estimates and simple observations can provide valuable insights and help guide decision-making. This understanding can empower product managers to take action even when faced with limited information. While "How to Measure Anything" is not a quick read, it is well worth the investment.

You Might Also Like:

Book Review and Summary: Measure What Matters by John Doerr

Get a comprehensive review and summary of the best-selling book "Measure What Matters" by John Doerr, the business leader and venture capitalist. Learn about the key principles and practices for setting and achieving measurable goals using the OKR framework.

andrewclark.co.uk

Understanding Variation

Wheeler highlights the value of Process Behaviour Charts, which present data as a time series with control limits. This approach accounts for natural process variation and provides a more accurate understanding than single-point comparisons. The book, clear and accessible, discusses recording and analysing data and differentiating between signal and noise.These insights can greatly improve the way we understand data and manage processes, particularly for those in product management roles.

andrewclark.co.uk

Forget the Funnel

I can’t wait to try the approach outlined in this book. Having a customer-led growth framework makes a lot of sense. A company is a value exchange mechanism, it must provide customer value and capture some of it. The authors did a great job of showing how this framework is complementary and intersects nicely with customer research and JTBD theory. The way it promotes a radical focus is helpful, in my experience it’s often a lack of focus that stops product teams making rapid progress.

andrewclark.co.uk

Key Takeaways

The 20% that gave me 80% of the value.

• Anything can be measured. If it matters at all, it can be observed and measured.
• The concept of measurement as “uncertainty reduction” and not necessarily the elimination of uncertainty is a central theme of the book.
• Measure what matters to make better decisions. Focus on high-value measurements.
1. Define the decision.
2. Determine what you know now.
3. Compute the value of additional information. (If none, go to step 5.)
4. Measure where information value is high. (Return to steps 2 and 3 until further measurement is not needed.)
5. Make a decision and act on it (Return to step 1 and repeat as each action creates new decisions).
• Definition of Measurement (an information theory version): A quantitatively expressed reduction of uncertainty based on one or more observations.
• This “uncertainty reduction” is critical. Major decisions made under a state of uncertainty and when that uncertainty is about big, risky decisions, then uncertainty reduction has a lot of value—and that is why we will use this definition of measurement.
• The Baysian Interpretation of Measurement → when probability refers to the personal state of uncertainty of an observer or what some have called a “degree of belief.” Bayes’ theorem describes how new information can update prior probabilities. The prior probability often needs to be subjective
• Break down complex problems into estimable parts. Decomposition alone can significantly reduce uncertainty. It gives the estimator a basis for seeing where uncertainty about the quantity came from.
• You often have more data than you think. Look for creative ways to gather relevant information.
• Statistics courses should teach that even small samples can improve decision-making odds when uncertainty is high.
• Most measurements people regard as difficult involve indirect deductions and inferences. When you need to infer something “unseen” from something “seen.” Eratosthenes couldn’t directly see the curvature of Earth, but he could deduce it from shadows and the knowledge that Earth was roughly spherical.
• Inference examples:
• Measuring with very small random samples of a very large population
• Measuring the size of a mostly unseen population
• Measuring when many other, even unknown, variables are involved
• Measuring the risk of rare events through observation and reason.
• Measuring subjective preferences and values: by assessing how much people pay for these things with their money and time
• The Rule of Five: There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population (regardless of the population size). The range will be wide but it’s narrower than having no idea.
• The Single Sample Majority Rule (i.e. The Urn of Mystery Rule) Given maximum uncertainty about a population proportion, such that you believe the proportion could be anything between 0% and 100% with all values being equally likely, there is a 75% chance that a single randomly selected sample is from the majority of the population.
• Usually only a few things matter—most of the variables have an “information value” at or near zero. BUT some variables have an information value that is so high that some deliberate measurement effort is easily justified.
• There are variables that don’t justify measurement, but it’s a misconception that unless measurement meets an arbitrary standard (statistical significance) it has no value.
• A decision has to be defined well enough to be modelled quantitatively.
• What makes a measurement of high value is a lot of uncertainty combined with a high cost of being wrong.
• Don't assume only direct answers to questions are useful. Indirect observations can be valuable.
• Calibration training can improve intuition for assigning probabilities and ranges to uncertain quantities. Better estimates are attainable when estimators have been trained to remove their personal estimating biases. Putting odds on uncertain things is a learnable skill. For range questions you know of some bounds beyond which the answer would seem absurd.
• Tips to help improve your confidence interval intuition:
• To calibrate better pretend to have an equivalent bet with a 90% payoff, move your confidence interval bounds until you’re indifferent between the two.
• Repetition and feedback: It takes several tests in succession, assessing how well you did after each one and attempting to improve your performance in the next one.
• Consider potential problems: Look to identify potential problems for each of your estimates. Assume your answer is wrong and try to explain to yourself why.
• Avoid anchoring: Look at each bound on the range as a separate “binary” question. For a 90% CI you must be 95% sure that the true value is less than the upper bound. Increase the upper bound until you’re that certain.
• Reverse the anchoring effect: Instead of starting with a point estimate, start with an absurdly wide range and then start eliminating the values you know to be extremely unlikely. What values do I know to be ridiculous?”
• Risk is a state of uncertainty with potential negative outcomes. Quantify risks with probabilities and impacts.
• Using all optimistic values for the optimistic case and all pessimistic values for the pessimistic case is a common error and no doubt has resulted in a large number of misinformed decisions. The more variables you include, the greater the exaggeration of the range becomes.
• Monte Carlo simulations can model risks by generating scenarios based on input probabilities.
• Expected Opportunity Loss (EOL) is the chance of being wrong times the cost of being wrong.
• The Expected Value of Information (EVI) is the reduction in EOL from a measurement. It guides measurement efforts. The point of measurement is to reduce the uncertainty, you therefore reduce expected opportunity loss.
• A common misconception is that massive data is needed to gain useful insight when uncertainty is high. In reality, is you have high uncertainty, a small amount of data can significantly reduce uncertainty.
• If you’ve never measured it before, you may lack even a fundamental sense of scale for the quantity. So, the things you measured the most in the past have less uncertainty, and therefore less information value, when you need to estimate them for future decisions.
• Measurements have a time-value constraint therefore prefer small, iterative observations.
• The Economic Value of Information is often inversely proportional to the amount of measurement attention a variable gets (Measurement Inversion).
• You almost always have to look at something other than what you have been looking at in the past.
• The Risk Paradox: If an organisation uses quantitative risk analysis at all, it is usually for routine operational decisions. The largest, most risky decisions get the least amount of proper risk analysis.
• Instruments of measurement, while imperfect, offer advantages over unaided human judgment alone.
• Decomposition
• Decomposition involves computing an uncertain variable from less uncertain or more easily measurable components.
• Decomposing a variable into observable parts can make measurement easier.
• Decomposition alone often sufficiently reduces uncertainty, requiring no further observation (decomposition effect).
• Nearly 1/3 of decomposed variables need no additional measurement.
• The decomposition process itself reveals that seemingly immeasurable things can be measured.
• Following the trail: Observation:
• Follow its trail like a clever detective. Do forensic analysis of data you already have.
• Use direct observation. Start looking, counting, and/or sampling if possible.
• If it hasn’t left any trail so far, add a “tracer” to it so it starts leaving a trail.
• If you can’t follow a trail at all, create the conditions to observe it (an experiment).
• The information value curve is usually steepest at the beginning. The first 10 samples tell you a lot more than the next 10. The initial state of uncertainty tells you a lot about how to measure it.
• Consider potential errors and biases in measurements, but don't be paralysed by them. Some measurement is better than none.
• Controlled experiments compare test and control groups to establish causation, not just correlation.
• Absence of evidence is evidence of absence, contrary to the common saying.
• Surveys should avoid response bias through careful question design and behavioral observation.
• The value of additional measurement drops off quickly. Focus on initial high-value measurements and reassess.
• A measurement framework: Define decisions, model uncertainty, compute information value, measure, decide, repeat.
• People tend to measure what's easy, not what's important. Focus on high-information value variables.
• Managers often prefer measuring factors that produce good news. Avoid this bias.
• If you've never measured something before, you likely lack a fundamental sense of scale for it.
• Not knowing the business value of measurements leads to over-measuring low-value factors.
• Be resourceful and clever in collecting relevant data. Squeeze more out of limited information.
• Iteratively measure in small steps. Initial measurements often change the value of further measurement.
• Challenge assumptions about immeasurable factors. Measuring the seemingly immeasurable is often quite feasible.

Deep Summary

Longer form notes, typically condensed, reworded and de-duplicated.

Part 1: The Measurement Solution Exists

When you can measure what you are speaking about, and express it in numbers, you know something about it; but when you cannot express it in numbers, your knowledge is of a meager and unsatisfactory kind; it may be the beginning of knowledge, but you have scarcely in your thoughts advanced to the state of science. Lord Kelvin (1824–1907)

• Anything can be measured. If something can be observed, it can be measured.
• Fuzzy measurements are still measurements as they tell you more than you knew before.
• Things most likely to be seen as immeasurable are, virtually always, solved by relatively simple measurement methods.
• Intangibles do not exist, or, at the very least, could have no bearing on practical decisions.
• Many decision makers default to labelling something as intangible when the measurement method isn’t immediately apparent.
• If you care about this alleged intangible at all, it must be because it has observable consequences, and usually you care about it because you think knowing more about it would inform some decision.
• Why do we care about measurements at all? Well they help us make decisions. We often need a method to analyse options and reduce uncertainty about decisions.
• Measurement is a strategy for reducing uncertainty.
• If there’s uncertainty and consequence to a decision, then measurements have high value.
• Don’t confuse the proposition that anything can be measured with everything should be measured. Measurements should help inform a significant bet.
• Measure what matters, make better decisions.
1. Define the decision.
2. Determine what you know now.
3. Compute the value of additional information. (If none, go to step 5.)
4. Measure where information value is high. (Return to steps 2 and 3 until further measurement is not needed.)
5. Make a decision and act on it. (Return to step 1 and repeat as each action creates new decisions.)

Success is a function of persistence and doggedness and the willingness to work hard for twenty-two minutes to make sense of something that most people would give up on after thirty seconds. Malcolm Gladwell

• Eratosthenes (c.200 B.C) calculated Earth's circumference using the difference in sun angles at Syene (into a deep well) and Alexandria (a tall tower), achieving an estimate within 3% of the actual value.
• Enrico Fermi developed a knack for intuitive measurements. He estimated the strength of the first atomic bomb by dropping confetti. Quickly confirming the yield must be greater than 10 kilotons.
• Measurements are a multistep chain of thought. Inferences can be made from highly indirect observations.
• To answer a Fermi question you have to figure out what you know about the quantity in question. This is called Fermi decomposition.
• Fermi Decomposition Example: How many Piano turners are there in Chicago?
• Population of Chicago
• Average number of people per household
• % of households with regularly tuned pianos
• Frequency of tuning (e.g. once per year)
• How many pianos a tuner could tune in a day
• How many working days a year
• The method helps to estimate the uncertain quantity but also gives the estimator a basis for seeing where uncertainty about the quantity came from. The biggest source of uncertainty would point toward a measurement that would reduce the uncertainty the most.
• Assessing what you currently know about a quantity is a very important step for measurement of those things that do not seem as if you can measure them at all.
• Even touchy-feely-sounding things like “employee empowerment,” “creativity,” or “strategic alignment” must have observable consequences if they matter at all.
• Simple methods like a controlled experiment, sampling , randomisation, and ‘blinding’ to avoid bias from the test subject or researcher.
• Don’t dwell on the overwhelming uncertainties, instead make an attempt at measurement. Ask what do you know? Useful observations can tell you something you didn’t know before—even on a budget—if you approach the topic with just a little more creativity and less defeatism.
• The concept of measurement as “uncertainty reduction” and not necessarily the elimination of uncertainty is a central theme of this book.
• Usually things that seem immeasurable in business reveal themselves to much simpler methods of observation. Useful observations are not necessarily complex or expensive.
• Paul Meehl showed that simple statistical models outperformed human judgment in a wide range of tasks.
• Amos Tversky and Daniel Kahneman showed how we can measure and improve our skill at assigning subjective probabilities.

• Arguments against measurement being possible…
• They don’t understand the definition of “measurement”, they incorrectly think it means meeting a nearly unachievable standard of certainty, then few things will be measurable.
• They can’t define what they want to measure precisely
• They aren’t aware of the basic methods of empirical observation
• Arguments against attempting measurement:
• It would be too expensive
• It wouldn’t be useful (’you can prove anything with statistics’)
• We shouldn’t measure it because it would be immoral to
• Definition of Measurement (an information theory version):
• Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations.
• Real scientific methods report numbers in ranges: “the average yield of corn farms using this new seed increased between 10% and 18% (95% confidence interval).”
• This “uncertainty reduction” is critical. Major decisions made under a state of uncertainty can be made better, even if just slightly, by reducing uncertainty.
• A measurement doesn’t have to eliminate uncertainty. A reduction in uncertainty counts as a measurement and can potentially be worth much more than the cost of the measurement.
• Different scales of measurement by Stanley Smith Stevens:
• Nominal scales: no implicit order or sense of relative size. A thing is simply in one of the possible sets.
• Ordinal scales: allow us to say one value is “more” than another, but not by how much. Example: Mohs hardness scale
• Ratio scales: homogeneous units such as dollars or kilometers tell us how much something is bigger than another. They can be added, subtracted, multiplied, and divided.
• Interval scales have defined units like ratio scales, but the zero is an arbitrary point like the Celsius scale for temperature. Temperature measured on the Kelvin scale, however, would be a ratio scale and all the ratio scale operations can apply.
• In business, decision makers make decisions under uncertainty. When that uncertainty is about big, risky decisions, then uncertainty reduction has a lot of value—and that is why we will use this definition of measurement.

Bayesian Measurement: A Pragmatic Concept for Decisions

• Measurement is “uncertainty reduction”, uncertainty is a feature of the observer, not necessarily the thing being observed.
• The Baysian Interpretation of Measurement → when probability refers to the personal state of uncertainty of an observer or what some have called a “degree of belief.”
• Bayes’ theorem describes how new information can update prior probabilities. The prior probability often needs to be subjective.
• Some scientists and statisticians interpret probability differently. In the "frequentist" view, probability is not an observer's state, but a system's objective feature.
• Total elimination of uncertainty is not necessary for a measurement but there must be some expected uncertainty reduction.If error in decision making isn’t reduced then they are not conducting a measurement according to our definition.

The Object of Measurement

A problem well stated is a problem half solved. Charles Kettering (1876–1958)

There is no greater impediment to the advancement of knowledge than the ambiguity of words. Thomas Reid (1710–1769)

• Some concepts initially seem immeasurable because they are ambiguously defined. The Clarification Chain states… if it matters at all, it is detectable/observable. If it is detectable, it can be detected as an amount (or range of possible amounts). If it can be detected as a range of possible amounts, it can be measured.
• Clarifying ambiguous terms makes them measurable. Ask…
• "What do you mean by <fill in the blank>?"
• "Why do you care?
• All measurements of any interest must support at least one specific decision.

If a thing exists, it exists in some amount, if it exists in some amount, it can be measured Edward Lee Thorndike:

The Methods of Measurement

• Most measurements people regard as difficult involve indirect deductions and inferences. When you need to infer something “unseen” from something “seen.” Eratosthenes couldn’t directly see the curvature of Earth, but he could deduce it from shadows and the knowledge that Earth was roughly spherical.
• Statistics courses should teach that even small samples can improve decision-making odds.
• Inference examples:
• Measuring with very small random samples of a very large population
• Measuring the size of a mostly unseen population
• Measuring when many other, even unknown, variables are involved
• Measuring the risk of rare events through observation and reason.
• Measuring subjective preferences and values: by assessing how much people pay for these things with their money and time
• The Power of Small Samples (even with infinite population sizes)
• The Rule of Five: There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population (regardless of the population size). The range will be wide but it’s narrower than having no idea
• The Single Sample Majority Rule (i.e., The Urn of Mystery Rule) Given maximum uncertainty about a population proportion—such that you believe the proportion could be anything between 0% and 100% with all values being equally likely—there is a 75% chance that a single randomly selected sample is from the majority of the population.
• We have strong intuitions that are often wrong. Presumptions about how a sample might affect our uncertainty are likely to be wrong.
• “Experiment” comes from the Latin ex-, meaning “of/from,” and periri, meaning “try/attempt.” It means, in other words, to get something by trying.

“If you don’t know what to measure, measure anyway. You’ll learn what to measure.” David Moore · Statistician

• The only valid reason to say that a measurement shouldn’t be made is that the cost of the measurement exceeds its benefits.
• Usually only a Few Things Matter—most of the variables have an “information value” at or near zero. BUT some variables have an information value that is so high that some deliberate measurement effort is easily justified.
• There are variables that don’t justify measurement, but it’s a misconception that unless measurement meets an arbitrary standard (statistical significance) it has no value.
• What makes a measurement of high value is a lot of uncertainty combined with a high cost of being wrong.
• It is impossible to find any domain in which humans clearly outperformed crude extrapolation algorithms, less still sophisticated statistical ones.
• Four Useful Measurement Assumptions:
• It’s been measured before (if not you might get a nobel prize)
• You have far more data than you think (the info you need is accessible, just take the time to find it)
• You need far less data than you think.
• Useful, new observations are more accessible than you think.
• People often say: “Unlike other industries, in our industry every problem is unique and unpredictable”
• One way to underestimate the amount of available data is to assume that only direct answers to our questions are useful.
• When you know almost nothing, almost anything will tell you something. Kahneman and Tversky showed that the error of a sample is often overestimated, which results in an underestimation of the value of a sample.
• The scientific method isn’t just about having data. It’s also about getting data.
• The way you think of to gather data is the likely the “hard way”, assume there’s an easier way.
• Don’t jump to an impractically sophisticated method. You need less data than you think.

Part 2: Before You Measure

• Toward a universal approach to measurement
1. Define a decision problem and the relevant uncertainties. What is your dilemma?
2. Determine what you know now. Quantify your uncertainty about unknown quantities in the identified decision. Describe your uncertainty in terms of ranges and probabilities.
3. Compute the value of additional information, it helps us identify what to measure and informs us about how to measure it
4. Apply the relevant measurement instrument(s) to high-value measurements (e.g. random sampling, controlled experiments, and some more obscure variations on these). Can you squeeze more out of limited data, how to isolate the effects of one variable, how to quantify “soft” preferences, how new technologies can be exploited for measurement, and how to make better use of human experts.
5. Make a decision and act on it. When the economically justifiable amount of uncertainty has been removed, decision makers face a risk versus return decision.
• Data on dashboards is not usually selected and presented with specific decisions in mind.
• Requirements for a Decision:
• A decision has two or more realistic alternatives.
• A decision has uncertainty. If there is no uncertainty about a decision, then it’s not really much of a dilemma. There have to be two or more choices and the best choice is not certain.
• A decision has potentially negative consequences if it turns out you took the wrong position.
• A decision has a decision maker.
• A decision has to be defined well enough to be modeled quantitatively.
• A Ridiculously Simple Decision Model:
• Estimated Costs of Action X.
• Estimated Benefits of Action X.
• If Benefits of Action X exceed Costs of Action X, execute Action X.
• Now just decompose costs and benefits into more detail as needed.
• Armstrong and MacGregor found that decomposition didn’t help much if estimates already had relatively little error. BUT if error was high then decomposition was a huge benefit. The most uncertain variables with a simple decomposition reduced error by a factor as much as 10 or even 100.
• Definitions for Uncertainty, Risk, and Their Measurements
• Uncertainty: The lack of complete certainty. The existence of more than one possibility. The “true” outcome/state/result/value is not known.
• Measure of uncertainty:: A set of probabilities assigned to a set of possibilities.
• Risk: A state of uncertainty where some of the possibilities involve a loss or undesirable outcome.
• Measure of risk: A set of possibilities each with quantified probabilities and quantified losses (e.g. 10% chance of losing $10M)
• Calibrated probability assessments are the key to measuring your current state of uncertainty about anything. Learning how to quantify your current uncertainty about any unknown quantity is an important step in determining how to measure something in a way that is relevant to your needs.

• Express uncertainty as a range of probable values (a confidence interval). A 90% confidence interval is a range that has a 90% chance of containing the correct answer.
• We get better at subjective probability assessment by comparing our expected outcomes to actual outcomes.
• Very few people are naturally calibrated estimators. Almost everyone is biased toward “overconfidence” or “underconfidence” (most are overconfident).
• Overconfidence: far fewer than 90% of the true answers fall within the estimated ranges.
• Underconfidence: routinely understating knowledge and is correct much more often than he or she expects.
• Better estimates are attainable when estimators have been trained to remove their personal estimating biases. Putting odds on uncertain things is a learnable skill.
• Calibration training takes about half a day. Put a 90% CI range on the answers to a number of questions and score how many you get right. Adjust and repeat.
• For range questions you know of some bounds beyond which the answer would seem absurd
• Tips to help improve your confidence interval intuition:
• To calibrate better pretend to have an equivalent bet with a 90% payoff, move your confidence interval bounds until you’re indifferent between the two.
• Repetition and feedback: It takes several tests in succession, assessing how well you did after each one and attempting to improve your performance in the next one.
• Consider potential problems: Look to identify potential problems for each of your estimates. Assume your answer is wrong and try to explain to yourself why.
• Avoid anchoring: Look at each bound on the range as a separate “binary” question. For a 90% CI you must be 95% sure that the true value is less than the upper bound. Increase the upper bound until you’re that certain.
• Reverse the anchoring effect: Instead of starting with a point estimate, start with an absurdly wide range and then start eliminating the values you know to be extremely unlikely. What values do I know to be ridiculous?”
• An assumption is a statement we treat as true for the sake of argument, regardless of whether it is true.
• The lack of having an exact number is not the same as knowing nothing. No matter how little experts think they know about a quantity, it always turns out that there are still values they know are absurd.
• The concern that ranges are too wide to be useful is not uncommon. Modelling your initial uncertainty is an interim step, not the end goal.
• An honest range isn’t embarrassing, we need to know the uncertainty we have.

In the conception we follow and sustain here, only subjective probabilities exist—i.e., the degree of belief in the occurrence of an event attributed by a given person at a given instant and with a given set of information. Bruno de Finetti

• In addition to improving one’s ability to subjectively assess odds, calibration seems to eliminate many objections to probabilistic analysis in decision making.

• Risk is a state of uncertainty where some possible outcomes involve a loss of some kind.
• If a measurement matters, it's because it informs an uncertain decision with potential negative outcomes.
• Many organisations rate risk as high, medium or low, but ambiguous labels don’t help the decision maker at all. They add imprecision through a rounding error that gives the same score to hugely different risks.

Real Risk Analysis: The Monte Carlo

• A Monte Carlo simulation uses a computer to generate a large number of scenarios based on probabilities for inputs.
• All risk in any project can be expressed by: the ranges of uncertainty on the costs and benefits and probabilities on events that might affect them.
• If you knew every cost and benefit and when it will occur, you literally have no risk.
• You can easily run a Monte Carlo simulation on excel. To do so you will need:
• The 90% confidence interval (CI) for each of the variables.
• The shape of the distribution (e.g. normal, uniform, binary {aka Bernouli)
• A random number generator to create inputs for the simulation.
• A formula that combines these inputs to produce an output.
• The ability to run the simulation multiple times to generate a range of possible outcomes.
• Using all optimistic values for the optimistic case and all pessimistic values for the pessimistic case is a common error and no doubt has resulted in a large number of misinformed decisions. The more variables you include, the greater the exaggeration of the range becomes.
• A normal distribution is a very bad approximation for a variety of phenomena including fluctuations of the stock market, the cost of software projects, or the size of an earthquake, plague, or storm.
• Some variables in a model might not be independent of each other. We can address that by generating correlated random numbers for them or, preferably, by modelling what they have in common explicitly.
• Markov Simulations are simulations where a single scenario is itself separated into a large number of time intervals, each of which affects the following time interval. This can apply to complex manufacturing systems, stock prices, the weather, computer networks, and construction projects.
• Agent-Based Models split up the problem into time intervals, we can also have separate simulations for a large number of individuals acting independently or somewhat in concert. The term agent often implies that each actor follows a set of decision rules. Traffic simulations are an example.
• Document your risk tolerance so you can measure all risks by the same standard.
• Building a Monte Carlo simulation is barely much more complicated than constructing any spreadsheet-based business case.
• The Risk Paradox: If an organisation uses quantitative risk analysis at all, it is usually for routine operational decisions. The largest, most risky decisions get the least amount of proper risk analysis.
• Studies on the use of Monte Carlos find that they improve forecasts and decisions and enhance the financial performance of firms.
• Cost and schedule estimates from Monte Carlo simulations on average, have less than half the error of the traditional accounting estimates.

• The McNamara Fallacy:
1. Measure whatever can be easily measured.
2. Disregard that which can’t easily be measured or to give it an arbitrary quantitative value.
3. Presume that what can’t be measured easily isn’t important.
4. What can’t easily be measured really doesn’t exist.

The Chance of Being Wrong and the Cost of Being Wrong: Expected Opportunity Loss

• Decision risk is why we need measurement.
• Decision risk: When you’re uncertain about the right a decision and there’s a chance you’ll pick the wrong option.
• Opportunity loss (OL): The difference between the choice you take and the best alternative.
• The Expected Opportunity Loss (EOL) for a particular strategy is the chance of being wrong times the cost of being wrong (e.g. the probability-weighted average).
• The point of measurement is to reduce the uncertainty, you therefore reduce expected opportunity loss.
• The value of information is equal to the expected reduction in EOL. The difference between the EOL before a measurement and the EOL after a measurement is called the “Expected Value of Information” (EVI). The value of information is equal to the value of the reduction in risk.
• So the EVPI (expected value of perfect information) is simply the EOL of your chosen alternative.
• A formula that computes how much we lose depending on the outcome is called a “loss function.”
• Once we have the distribution for some range variable and the loss function for it, we can compute its EVPI by slicing up the distribution into lots of tiny parts, then work out the expected loss for each part, and then add them all
• Computing an “Incremental Probability”
• The “Normdist()” formula is used to compute probabilities in a distribution.
• The norminv() function will generate a normally distributed random.
• The normdist() allows you to work out the probability that a value is less than X in a normal distribution with a given mean and standard deviation.
• We would write it as: =normdist(X,mean,sd, 1)

• A common misconception is that massive data is needed to gain useful insight when uncertainty is high. In reality, is you have high uncertainty, a small amount of data can significantly reduce uncertainty.

• Be wary of time-sensitive decisions. The opportunity could evaporate if you take too long to decide. Reducing uncertainty doesn’t just take money, it takes time. Be careful not to solve the right problem too late. Measurements have a time-value constraint therefore prefer small, iterative observations.

How the Value of Information Changes Everything

• The vast majority of variables in almost all models have an information value of zero.
• The variables that have high information values are routinely not measured
• The variables that people spend the most time measuring were usually those with a very low (even zero) information value (i.e., it was highly unlikely that additional measurements of the variable would have any effect on decisions).
• The highest-value measurements are almost always are a bit of a surprise.

The Measurement Inversion: In a decision model with a large number of uncertain variables, the economic value of measuring a variable is usually inversely proportional to how much measurement attention it typically gets.

• People measure what they know how to measure or what they believe is easy to measure.
• If you’ve never measured it before, you may lack even a fundamental sense of scale for the quantity. So, the things you measured the most in the past have less uncertainty, and therefore less information value, when you need to estimate them for future decisions.
• Managers tend to measure things that are more likely to produce good news. Don’t let the champions of an investment be the only ones responsible for measuring their own performance.
• Finally, not knowing the business value of the information from a measurement means people can’t put the difficulty of a measurement in context.
• While most variables in large uncertain decisions had no information value, usually there are at least some with a significant information value which easily justified further measurements.
• You almost always have to look at something other than what you have been looking at in the past.
• If you don’t compute the value of measurements, you are probably measuring the wrong things, the wrong way. Be Iterative. The highest-value measurement is the beginning of the measurement, so do it in bits and take stock after each iteration.

Measurement Methods

Questions that help determine the appropriate category of measurement methods:

• What are the parts of the thing we're uncertain about? Further decompose the uncertain variable with high information value in your decision model, so it’s computed from other uncertain things.
• How has this (or its decomposed parts) been measured by others? Review existing research ("secondary research") on the measurement problem.
• How do the "observables" identified lend themselves to measurement? Determine how to observe the decomposed parts, possibly guided by secondary research.
• How much do we really need to measure it? Consider current uncertainty, threshold, and value of information. EVPI guides the appropriate level of effort for measurement.
• What are the sources of error? Identify potential biases, inconsistencies, and sampling errors in observations. Address these issues without being paralyzed by "exception anxiety."
• What instrument do we select? Based on the answers above and secondary research, design an appropriate measurement instrument.

• Of course instruments of measure have errors. The question is “Compared to what?” Compared to the unaided human? Compared to no attempt at measurement at all? Keep the purpose of measurement in mind: uncertainty reduction, not necessarily uncertainty elimination.
• Advantages of instruments:
• Instruments detect what you can't directly observe, acting as an intermediary between the phenomenon of interest and the observer.
• Instruments are generally more consistent than human judgment alone.
• Instruments can be calibrated to account for known errors and assess odds of inaccurate readings.
• Instruments often include controls to offset particular errors and allow comparisons.
• Instruments deliberately ignore some factors, as removing bias from every test is impossible.
• Instruments record data, serving as an objective log of activity. ‘Big Data’ extrapolates from instrument recordings that exceed human capabilities.
• Instruments can measure faster and cheaper than direct human observation, enabling efficient data gathering at scale.
• The Four Useful Measurement Assumptions:
• If it’s been done before, don't reinvent the wheel.
• Resourcefulness can unlock more data.
• Clever analysis reduces data needs.
• New data is often more accessible than anticipated.
• Decomposition
• Decomposition involves computing an uncertain variable from less uncertain or more easily measurable components.
• Decomposing a variable into observable parts can make measurement easier.
• Decomposition alone often sufficiently reduces uncertainty, requiring no further observation (decomposition effect).
• Nearly 1/3 of decomposed variables need no additional measurement.
• The decomposition process itself reveals that seemingly immeasurable things can be measured.
• Following the trail: Observation:
• Follow its trail like a clever detective. Do forensic analysis of data you already have.
• Use direct observation. Start looking, counting, and/or sampling if possible.
• If it hasn’t left any trail so far, add a “tracer” to it so it starts leaving a trail.
• If you can’t follow a trail at all, create the conditions to observe it (an experiment).
• Measure Just Enough
• Keep the information value in mind along with the threshold, the decision, and current uncertainty provides the purpose and context of the measurement.
• Initial measurements often change the value of continued measurement. The first few observations may be surprising, the information may be very informative, and the value of continuing the measurement may drop to zero.
• The information value curve is usually steepest at the beginning. The first 10 samples tell you a lot more than the next 10. The initial state of uncertainty tells you a lot about how to measure it.
• Consider the Error
• Error Terms:
• Systemic error/bias: Inherent tendency of a measurement process to favor a particular outcome; consistent bias.
• Random error: Unpredictable error for individual observations; follows probability rules in large groups.
• Accuracy: Low systemic error; not consistently over- or underestimating a value. Also called "validity" in some fields.
• Precision: Low random error; highly consistent results regardless of distance from true value. Also called "reliability" or "consistency" in some fields.
• Precision refers to reproducibility and conformity of measurements, while accuracy refers to closeness to the "true" value.
• All measurements have error. Recognizing this allows development of compensating strategies.
• Random error can be reduced by averaging several measurements.
• Random sampling, used properly, is a type of control. Random effects follow predictable aggregate patterns.
• Types of Observation Biases:
• Expectancy bias: Observers and subjects sometimes see what they want, consciously or not.
• Selection bias: Inadvertent nonrandomness in samples, even when attempting randomness.
• Observer bias (Heisenberg and Hawthorne bias): The act of observing changes the behavior of both subatomic particles and humans.
• Choose and Design the Instrument
• After decomposing the problem, considering secondary research, placing elements in an observation hierarchy, aiming for "just enough" uncertainty reduction, and accounting for specific errors:
1. Decompose the measurement into estimable parts.
2. Incorporate findings from secondary research.
3. Categorize decomposed elements into observation methods: trails, direct observation, tagging, or experiments.
4. Focus on "just enough" uncertainty reduction.
5. Address problem-specific errors.
• Iteratively consider multiple simple approaches that could render further measurement unnecessary. Just start making organised observations without being paralysed by anxiety over potential measurement issues.

• In many cases, it is impractical to measure every item in a population. A few samples can greatly reduce uncertainty, especially with homogeneous populations. Calibrated estimators can sometimes reduce uncertainty with only one sample.
• The idea of "statistical significance" is often misunderstood or misapplied. Small samples can produce statistically significant results and be informative for decision-making. The big payoff in tends to be early in the information gathering process.
• Nonparametric methods like the "Rule of Five" can estimate medians from small samples.
• Calibrated estimators intuitively include prior knowledge that "objective" methods ignore.
• Catch-recatch and other sampling methods compare overlap between samples to estimate population size. Measure the size of population by catching some and tagging them. Put them back. Then catch a second sample and see what % of them have your tags.
• Spot sampling takes random snapshots instead of constant tracking.
• Serial sampling tracks the same samples over time.
• Measuring relative to decision thresholds is often more relevant than reducing general uncertainty.
• Clustered sampling takes random samples of groups, then conducts a census within groups.
• Stratified sampling uses different methods for different subgroups.
• Controlled experiments compare test and control groups to determine causation.
• Correlation does not prove causation but is evidence of it; lack of correlation makes causation unlikely.
• Simple linear regressions can identify relationships; multiple regression models multiple factors simultaneously.
• Absence of evidence is evidence of absence, contrary to the common saying.
• Even single factor can provide some information, despite not telling the whole story.
• Surveys commonly use Likert scales, multiple choice, rank ordering, and open-ended questions. Strategies for avoiding response bias: keep questions short, avoid loaded/leading questions, avoid compound questions, reverse scales.
• Observing behaviour is often more reliable than asking opinions directly.
• People's partitioning of options and understanding of probabilities can bias their stated preferences.
• A general measurement framework: define decision, model uncertainty, compute value of information, measure high-value uncertainties, decide, repeat.