You look like a thing and I love you · Janelle Shane · 2020
Key Highlights:
Promising headlines about AI are everywhere, but real progress is often slow. AI is already embedded in everyday life, yet it isn’t flawless. Its quirks can cause serious consequences, such as YouTube recommending increasingly polarising content.
Understanding AI's limitations requires grasping five fundamental principles of AI weirdness.
- The real danger of AI isn't that it's too smart, it's that it's not smart enough.
- AI has roughly the brain power of a worm.
- AI doesn't truly understand the problem you're trying to solve.
- AI will do exactly what you tell it to do, or at least try its best to follow your instructions literally.
- AI will always take the path of least resistance.
Training AI is closer to teaching a child than programming a computer. We show it examples and say, “Here’s the data, you figure out how to copy it.” AI works particularly well when we don’t fully know the rules, when there are many subtle rules, or when new rules might still be discovered.
This can lead to surprising failures. Healthcare researchers, for example, discovered that a cancer detection algorithm had learned to detect rulers instead of tumours, because many of the tumour photos in its training data included rulers for scale. AI can also inherit human bias if the underlying dataset is biased, so we need to anticipate problems before they occur. Worrying about an AI takeover is a bit like worrying about overcrowding on Mars… it’s not today’s problem.
The difference between successful AI and failure often comes down to whether the task is suitable for AI learning. AI tends to go wrong in four ways: the problem is too hard; the problem is not what we thought it was; it finds sneaky shortcuts; or it learns from flawed data and becomes flawed itself.
AI might still be useful even when a human could do the task better, think of a robot vacuum that cleans while you’re busy. However, we must be very careful when mistakes have big consequences. The narrower and more clearly defined the task, the smarter the AI seems; as tasks become broader and more open-ended, AI tends to struggle, so it often makes sense to specialise.
AIs are slow learners, usually needing thousands of images to learn a new object. Because they learn so slowly, we often use simulations to train them faster than real time. They are also bad at remembering things and weak at forward planning, because they cannot see very far into the future. In text generation, for example, performance often degrades as the text gets longer, since the system has to remember what came before and plan ahead. Whenever you can solve a problem with common sense or simple rules, it is worth asking whether AI is really the simplest approach.
When a neural network trains itself, it becomes hard to understand exactly what it is reacting to or why. There are two main approaches to interpretability: examining which cells (neurons) activate for particular kinds of inputs, and tweaking an input image to see which changes make a given cell fire more strongly. OpenAI once trained an artificial neural network on Amazon review data to predict the next letter in a sequence and found that one neuron had effectively learned to track the sentiment of the review. Google discovered that one of its ImageNet models distinguished dogs from cats partly by focusing on whether the ears were floppy or pointy.
Class imbalance is a big problem: if the thing you are looking for is rare, such as fraud, an algorithm can achieve impressively high “accuracy” simply by always predicting the other class (not fraud). Different methods have different strengths and weaknesses. Markov chains tackle jobs similar to recurrent neural networks, such as predicting the next word in a sentence. They are lightweight and fast to train, but they cannot look very far into the future. Random forests are made up of many decision trees—flow charts that lead to outcomes based on the information available. Machine learning can build a whole forest of trees via trial and error. Each tree tends to learn different attributes and gets a “vote,” and the combined votes form the final prediction. Each tree only sees a small slice of the data, but together they form something more powerful.
Evolutionary algorithms treat each potential solution like an organism. In each generation, the most successful solutions “survive” and reproduce, mutating or combining to form new candidate solutions. Hyperparameters are the rules we set to govern these learning processes. Combining multiple ML algorithms often makes sense because each one works best in a narrow domain; deciding how to break a problem into subtasks for different sub-algorithms is a key part of success.
AI works poorly when the problem is too broad, when there is not enough data, when the data is confusing, when the training task is much simpler than the real problem, or when the training situation does not resemble the real world. More data is usually helpful for training AI. You can obtain more data through crowdsourcing, platforms like Mechanical Turk, or data augmentation techniques. Cleaning up messy input data is often one of the most effective ways to improve performance.
Training data quirks can produce odd behaviours. For example, some image generators seem to include giraffes in too many scenes, because giraffes are overrepresented in their training data. There is also a risk of unintentional memorisation, where a model memorises something in the original dataset—often sensitive personal information—and later exposes it to users.
AI can succeed at what you ask while still failing to do what you actually wanted. It is often useful to imagine that the system is deliberately misinterpreting your reward function, so you can spot where the incentives might go wrong.
Simulations used to train AI are only maps, not the territory. Unless you explicitly encode the relevant constraints, an AI has no obligation to obey laws of physics or other rules you forgot to include.
Whenever data comes from humans, it is likely to contain bias. If you use movie review data, for instance, you may inherit “review bombs,” where people leave negative reviews simply because a film has Black leads or women stars. The algorithm will learn those associations unless you intervene. You can, for example, adjust word vectors to weaken these biased connections. In doing so, you are effectively “playing god”—the fix is not perfect, but it is better than letting the worst of the internet define your model’s behaviour. Mathwashing or bias laundering refers to explaining away biased outcomes just because a computer, rather than a human, produced the decision.
Some researchers believe human dreams are a kind of low-stakes simulation training—a lower-fidelity, energy-efficient way to learn about important things and experiment safely. In AI, when class imbalance combines with biased datasets, it often amplifies bias even further. There are even adversarial tricks aimed at hiring algorithms, such as adding “Oxford” or “Cambridge” in invisible white text to a CV to pass automated filters.
When evaluating AI claims, it helps to ask a few simple questions. How broad is the problem? Where did the training data come from? Does the task require a lot of memory or long-term planning? And is the system simply copying human biases rather than overcoming them?
As AI becomes ever more capable, it still won’t know what we want. It will still try to do what we want. But there will always be a potential disconnect between what we want AI to do and what we tell it to do.
Going forward, working well with AI means learning how it functions, choosing the right kinds of problems for it to solve, and anticipating the ways it will misunderstand us. We also need to prevent it from copying the worst of what it finds in data, so that our systems reflect our values rather than our biases.
Full Book Summary · Amazon
Subscribe Button Quick Links
Automated Evals with Teresa Torres · Video
Vivek on the MIT 95% AI deployment failure rate · Video
Evals need not be an AI only thing · Article
You’re done with discovery when the path becomes clear · Article
Executive demanding features? Create a hypothesis · Article
The origins and development of the diffusion of innovations paradigm as an example of scientific growth
Thomas W Valente. Everett M Rogers. 1995. (View Paper → )
Diffusion is the process by which an innovation is communicated through certain channels over time among members of a social system. The diffusion of innovations is a communication theory which has laid the groundwork for behaviour change models across the social sciences, representing a widely applicable perspective…This paper describes some of the history of rural sociological research on the diffusion of agricultural innovations with the goal of understanding how the research tradition emerged and to determine how it influenced the larger body of diffusion research conducted later by scholars in other disciplinary specialties. The authors describe how diffusion of innovations research followed and deviated from the Kuhnian concept of paradigm development.
The diffusion of innovation paradigm began with Gabriel Tarde's "laws of imitation" in 1903 and early European diffusion studies in the early 20th century. It entered U.S. rural sociology through pioneering studies like Ryan and Gross's hybrid corn research in the 1940s, which examined how agricultural innovations spread through farming communities.
Everett Rogers formalised the paradigm in his landmark 1962 book Diffusion of Innovations, synthesising decades of earlier research. The paradigm rapidly expanded across disciplines—including medical sociology, education, marketing, geography, and anthropology—with diffusion-related publications doubling every decade from the 1940s through the 1980s.
Core Principles of Diffusion Theory
Innovation adoption follows five stages: knowledge, persuasion, decision, implementation, and confirmation. The theory features an S-shaped curve that shows how innovations are gradually adopted over time.
Adopters fall into five categories based on their speed of adoption: innovators, early adopters, early majority, late majority, and laggards.
Five key attributes determine how quickly innovations spread: relative advantage, compatibility, complexity, trialability, and observability.
Interpersonal network and opinion leaders play a crucial role in accelerating or slowing diffusion.
The diffusion of innovation paradigm exemplifies how scientific research traditions evolve into widely adopted frameworks, spreading across disciplines and enriched by multidisciplinary contributions.
/w=3840,quality=90,fit=scale-down)
Book Highlights
...one thing that we often forget is that there are other people involved who have influence over our project. It’s not enough to simply create an incredible app; we also have to get the support of our team. Without their support, our project can’t move forward. Tom Greever · Articulating Design Decisions
What do you want to know? • Why do you want to know? • What decision will you make based on the answers? • What number do you need to make the decision? Caroline Jarrett and Steve Krug · Surveys That Work
“Weed the garden” of stale activities and business units. Richard Rumelt · The Crux
/w=3840,quality=90,fit=scale-down)
Quotes & Tweets
Your capacity for excellence is inversely proportional to the number of your commitments. Shane Parrish
Design is not just what it looks like and feels like. Design is how it works. Steve Jobs