Lesson 3: Producing the value and the probability
A BOTEC needs a value and a probability. The value is usually a quantity that already exists in the world: how many chickens, how many people, how many tonnes. You're not predicting anything, you're working out a number that is in principle knowable. The probability is a genuine forecast — a claim about an uncertain event that hasn't happened yet. Nearly all the difficulty, and nearly all of this lesson, is on the probability side.
The value: decompose and look up
The move for the value is Fermi decomposition. It's named after the physicist Enrico Fermi, who was known for pulling surprisingly good estimates out of almost no information by breaking a question into pieces he could each have a go at. The textbook example: how many piano tuners work in Chicago? Nobody knows offhand. But you can decompose it —
Population of Chicago ≈ 3 million
People per household ≈ 3 → ~1 million households
Fraction with a piano ≈ 1/20 → ~50,000 pianos
Times tuned per year ≈ 1
Pianos one tuner handles per year ≈ 1,000
→ ~50 piano tuners
Every line is a guess. The point isn't the arithmetic; it's that each guess is now out in the open, so anyone who disagrees with your answer has to disagree with a specific number rather than with a feeling. The same move gives you the value at stake for a campaign:
Chickens processed per large European chain per year ≈ 50 million
Fraction of supply the policy would cover ≈ 80%
Average welfare improvement per chicken, on a 0–1 scale ≈ 0.1
→ ~4 million chicken-welfare-units per year
Most of these are lookups or defensible ballparks — the headcount you can source, the coverage you can estimate. There's no forecasting here, just decomposition and multiplication.
That's the value. Now the part that actually needs technique.
The probability: the actual forecast
In the 1980s, a young psychologist named Philip Tetlock started collecting predictions. Not vague ones — specific, checkable claims about what would happen next in world politics, made by exactly the people you'd expect to know: political scientists, area-studies experts, intelligence analysts, commentators paid to opine on the future. He gathered tens of thousands of these forecasts, wrote down what each expert had said, and then did the thing almost nobody bothers to do. He waited to see whether they came true.
It took about twenty years. The result, when it landed, was brutal: on average, the experts' predictions were no better than chance. A dart-throwing chimpanzee, as the comparison went, would have done about as well. Worse, the most famous experts — the ones with the media profiles and the confident television manner — tended to do worse than their less celebrated colleagues. Fame and confidence were, if anything, negatively correlated with getting it right.
But buried in the same data was a more hopeful finding, and it's the one this week is built on. A subset of forecasters genuinely beat chance, consistently, year after year. Tetlock called them superforecasters. When he looked at what set them apart, it wasn't intelligence, or credentials, or access to secret information. It was how they thought.
They broke big questions into smaller ones. They started not from the drama of the case in front of them but from how often things like it had gone one way or the other before. They sought out the people who disagreed with them. And they put numbers on their uncertainty — actual numbers, not "likely" or "a fair chance" — and nudged those numbers up or down as new evidence came in.
None of that is a gift you're born with. Tetlock spent the next decade proving the point, running tournaments in which trained volunteers beat professional intelligence analysts who had access to classified material. The moves are learnable. The rest of this lesson teaches the ones that carry the most weight — applied to the number we actually need: the probability that our campaign succeeds.
Start from the outside
Your instinct, faced with "will Feather Forward's campaign work?", is to think about Feather Forward: their strategy, their team, how promising this particular effort looks. That's the inside view, and on its own it runs optimistic — you end up dwelling on why this case is special and forgetting how hard the general task is.
So start somewhere else. Ask the outside-view question first: how often do things of this sort succeed, in situations of this sort? Not "campaigns exactly like this one" — that's a sample of one — but a reference class wide enough to hold real cases: well-funded corporate welfare campaigns targeting a single large company in a rich democracy. If campaigns like that succeed 20–30% of the time within five years, that is your starting number. The reason it goes first is precisely that it's the unflattering number — it's the anchor that stops your optimism about the specific case from running away with the estimate.
You don't need precision here, just the right order of magnitude. Fifteen minutes with a sector report, an academic paper, or a sceptically-prompted LLM is usually enough for a BOTEC. Write down both the number and the reference class you got it from — if you only write the number, you'll quietly re-anchor to your gut within the hour.
Adjust from the inside — carefully
Now bring in what you know about this specific campaign. Some of it argues the probability up (an unusually strong team, a target already under pressure), some of it argues it down (a hostile political climate, a company known for stonewalling). Treat these as adjustments off the anchor, in small and argued steps — not as licence to abandon the base rate for a number you like better. The pull is always upward; that's the thing to resist.
One particularly useful way to run the inside view is to break the outcome into a chain. Most things you care about need several steps to go right in sequence, and each link is easier to judge than the whole:
P(meetings with decision-makers) ≈ 70%
P(commitment | meetings) ≈ 40%
P(implemented | commitment) ≈ 50%
Combined: 0.70 × 0.40 × 0.50 = 14%
The chain pays off twice. Guessing each link is easier than guessing "will it succeed?" cold. And the chain is observable — if the campaign fails, you can usually see which link broke, and learn something for next time. That final link, whether a commitment actually gets honoured, is the one Lesson 5 comes back to: in the real world only 40–60% of commitments are fully implemented, so it's a real drag on the number, not a technicality.
Triangulate
You now have the probability by two routes: the outside view said 20–30%, the inside-view chain said 14%. Same order of magnitude — and that agreement is itself the reassuring part. Settle somewhere in that region.
The interesting case is when the two routes disagree sharply — say the base rate points to 40% and your chain lands at 2%. That gap isn't an annoyance to be split down the middle; it's a signal that one of the two routes contains a bad assumption, and it's told you exactly where to go looking. Using more than one route, and treating disagreement between them as information rather than noise, is most of what separates a robust estimate from a lucky one.
Then argue against yourself
Before you commit the number, take the other side seriously. If you've landed on 15%, ask what someone who confidently said 40% can see that you can't. The sharpest version of this is a pre-mortem: imagine the campaign has already failed, and write down the most likely reason. It tends to surface the failure you were glossing over — and more often than not it's a link in your own chain that you'd quietly rated too high.
Most of the standard ways people go wrong are just these moves skipped. Anchoring on the first number that popped into your head — the outside view is the cure, start from the rate. Letting one recent, vivid case stand in for the base rate — that's what a proper reference class is for. False precision — if you can't defend the third digit, widen your range instead of inventing it. And the big one: "this campaign is unusually good." It may well be. But most campaigns that looked unusually good still failed, which is exactly what the base rate already knows.
The number has to be honest, not just considered
Everything above helps you produce a probability. None of it helps if your probabilities don't mean what they say. If the things you call "70% likely" actually happen half the time, a beautifully decomposed 70% is worthless. This property — your stated confidence matching how often you turn out to be right — is called calibration, and it's the floor the whole exercise stands on.
Most people are worse at this than they think. Asked for things they're "90% sure" of, people are typically right closer to 70% of the time. The good news, unlike most skills, is that you can measurably improve your calibration in a single evening — because you can practise on questions where the answer is already known and get told immediately whether your confidence was warranted.
Do this now: Calibrate Your Judgment — Clearer Thinking (~20 mins). Required, not optional. It quizzes you on questions with known answers, asks how confident you are each time, and scores you on both whether you were right and whether your confidence was earned — you lose points for saying 95% and being wrong, and equally for hedging to 55% on something you actually knew. Complete one set. At the end it tells you whether you run overconfident, underconfident, or about right. Write your result down — you'll want to know your own bias when you're putting probabilities into your BOTEC in Lesson 6.
(Optional, if you catch the bug: Metaculus and Good Judgment Open let you forecast live questions — ones not yet resolved — and track your accuracy over months.)
You've now got both numbers: a value from decomposition, and a probability you've anchored, adjusted, triangulated and stress-tested. Multiply them and you have a raw expected value. But that raw number still flatters you — it quietly assumes that none of this good would have happened without you, that it happens the moment you act, and that it lasts forever. The next lesson is about the three adjustments that turn it into something honest.
Exercise: forecasting practice
~10 minutes. Write your answers somewhere you can find them again. An example answer is in the appendix at the end of this week — don’t read it until you’ve done the exercise yourself.
Throughout these lessons we'll use one running example: Feather Forward, a (fictional) animal-welfare organisation running a corporate campaign to persuade a large European fast-food chain to adopt higher broiler-chicken welfare standards. "Success" means the chain publicly commits to the standard and actually implements it across its supply chain. It's a good test case because none of the numbers are obvious — how many chickens are affected, how likely the campaign is to work, how much credit Feather Forward can claim — which is exactly the situation a BOTEC is built for.
You’re thinking about whether a €500 donation to Feather Forward would be a good use of money. Before you build a full BOTEC in Lesson 6, practise the forecasting moves.
(a) Fermi decomposition — the value side. Estimate how many broiler chickens in Europe would be affected per year if Feather Forward’s current campaign fully succeeded. Decompose into 2–3 components, show your arithmetic, give a final number.
(b) Base rate + adjustment — the probability side. Research suggests well-funded corporate welfare campaigns targeting a single large company in a democratic country succeed about 20–30% of the time within five years.
What’s one factor about Feather Forward’s specific situation that would push your probability up from this base rate?
What’s one factor that would push it down?
Give your final probability estimate in a single number.
(c) Pre-mortem. In one sentence: if your forecast turns out to be wrong, what’s the most likely reason?