Skip to main content
Intelligent and Adaptive Finance
Intelligent and Adaptive Finance
The Diagram That Broke the Taboo — A Synopsis of The Book of Why | Intelligent Adaptive Finance
Synopsis · Research & Methodology

The DiagramThat Brokethe Taboo

The New Science of Cause and Effect

Judea Pearl and Dana Mackenzie spent a book arguing that data cannot answer the questions enterprises most want answered. Two decades of analytics investment suggest they were right.

BookThe Book of Why
AuthorsJudea Pearl · Dana Mackenzie
Synopsis written byKrishnendu Pal

For most of the twentieth century, a statistician who wrote that one thing caused another was committing a professional error. The word had no standing in the formal vocabulary. Karl Pearson, who built much of the apparatus the field still runs on, treated causation as a superstition that correlation had superseded. Ronald Fisher, who gave science randomisation, spent his final years insisting that the association between smoking and lung cancer might be an artefact of genetic predisposition. People died in large numbers while the one discipline best equipped to settle the question maintained that the question was improper. Pearl and Mackenzie tell the story of how that prohibition formed, what it cost, and how it eventually broke. What broke it was not more data. It was a drawing.

01 · The argumentWhat the book actually claims

The central claim is a hierarchy. Questions about the world sort into three levels, and each level requires machinery the level below cannot supply. Seeing comes first: what does the data say about how often A and B occur together? Doing comes second: what happens if I intervene and set A myself? Imagining comes third: what would have happened to this particular customer, patient or portfolio had I acted differently, given that I acted as I did? Pearl calls this the Ladder of Causation, and the argument that follows is uncompromising. No amount of observational data, and no algorithm however sophisticated, can climb from one rung to the next without something extra. That something is a causal model, and the model has to come from outside the data.

The second claim is the one that makes the book more than a philosophy lecture. Pearl says the missing ingredient can be written down. A causal diagram, a set of variables joined by arrows encoding what someone believes influences what, turns vague assumptions into a mathematical object. Once drawn, the diagram can be interrogated. It will tell you which effects can be estimated from the data you already hold, which cannot be estimated at any sample size, and precisely which variables you must adjust for and which you must leave alone. That last capability is the quiet revolution. Statistics spent a century unable to say what "controlling for confounders" formally meant.

Pearl's warning to the analytics era compresses into four words: data are profoundly dumb. Records can show that treated patients recovered faster. They cannot say whether the treatment did it. Pearl & Mackenzie, The Book of Why, Basic Books, 2018
Exhibit 1
The Ladder of Causation, with the enterprise systems that occupy each rungSelect any rung for detail
3IMAGINING · CounterfactualsP(yₓ | x′, y′) "What if I had acted differently?"REQUIRESA fully specified structural model of the mechanismENTERPRISEIncrementality attribution, root-cause claims, liability2DOING · InterventionP(y | do(x)) "What happens if I set the lever myself?"REQUIRESA causal diagram, or a randomised experimentENTERPRISEA/B and geo-lift testing, uplift models, policy simulation1SEEING · AssociationP(y | x) "What does the record say tends to go together?"REQUIRESData alone. Nothing else.ENTERPRISEDashboards, forecasting, churn scoring, most machine learningEACH RUNG IS STRICTLY MORE POWERFUL THAN THE ONE BELOWNo amount of rung-1 data answers a rung-2 question. The gap is logical, not a matter of sample size.
The three levels, their formal query notation, and the analytics infrastructure that sits on each. Most enterprise investment over the past decade has gone into the bottom band, while most enterprise questions come from the top two. Structure after Pearl & Mackenzie (2018), ch. 1.

02 · Author's vantageWho is making it, and from where

Judea Pearl is a computer scientist at UCLA who won the 2011 Turing Award for building a calculus of probabilistic and causal reasoning. He arrived at causality sideways. In the 1980s he invented Bayesian networks, the machinery that let machines reason under uncertainty and that still underpins a good deal of applied AI, and he then concluded that his own creation was insufficient because it handled belief rather than action. Dana Mackenzie, a mathematician turned science writer, supplies the narrative craft; the history of Galton, Pearson, Wright and Fisher reads like a proper story rather than a literature review.

03 · Key insightsFive things the book gets right that most analytics practice still gets wrong

The ladder is a hierarchy of impossibility, not a hierarchy of difficulty

The rungs are often read as a maturity model: get good at prediction, then get good at experimentation, then get good at counterfactuals. That reading is too soft. The relationship is one of strict logical containment. A question at rung two cannot be answered by any function of rung-one data alone, no matter how much of it you have or how clever the estimator. Bareinboim and colleagues later gave this a formal name, the Causal Hierarchy Theorem, and proved it holds almost everywhere. The practical consequence for any organisation is blunt. A model trained purely on historical observation is answering a different question from the one the decision-maker asked, and no amount of accuracy on the first question licenses an answer to the second.

The diagram makes assumptions arguable

Causal claims always rest on assumptions the data cannot verify. Every serious methodologist knew this. Pearl's contribution was insisting the assumptions be drawn rather than muttered. Once you commit to a graph, three things become possible that were not possible before. The graph implies testable conditional independences, so parts of it can be falsified by the very data it is meant to interpret. It settles the adjustment question mechanically through the back-door criterion. And it makes disagreement productive, because two analysts who reach different conclusions can now point at the specific arrow they dispute rather than trading intuitions about robustness.

Exhibit 2Graph surgery: what the do-operator actually does to a model
SEEING · P(Y | X)Price is set by the demand team, who watch demand.ZDemand conditionsXPriceYUnits soldDOING · P(Y | do(X))Price is set by us, at random, ignoring demand.ZDemand conditionsXPrice, set by usYUnits sold
Intervention is modelled as deletion. Every arrow pointing into the manipulated variable is cut, because the variable no longer obeys its old causes. Estimating price elasticity from historical transactions answers the left-hand diagram; the pricing committee is asking about the right-hand one, where the observed positive price-volume correlation vanishes.

Adjusting for more variables makes the answer worse surprisingly often

The instinct drilled into every analyst is that a regression with more controls is more credible. Pearl demolishes this. Inclusion depends entirely on where the variable sits in the causal structure, and two positions actively manufacture bias. Condition on a mediator, a variable sitting on the path you are trying to measure, and you erase part of the effect you came to find. Colliders are stranger. A collider is a variable that both the cause and the outcome influence, and conditioning on one invents a correlation between them that exists nowhere in the world. Selection is silent conditioning: the moment you analyse only converted customers, only funded loans or only hired candidates, you have conditioned on a collider without ever deciding to.

Exhibit 3The three junctions, and the control rule each one implies
CHAIN · MediatorXMYX acts on Y through M.DO NOT ADJUST FOR MControlling for M erases the veryeffect you set out to measure.Business: adding "engagement"as a control in a feature test.FORK · ConfounderZXYZ drives both X and Y.ADJUST FOR ZThe back-door path is open untilyou block it.Business: seasonality drivingboth price and volume.COLLIDER · SelectionXYCX and Y both feed into C.DO NOT ADJUST FOR CConditioning invents a correlationthat does not exist in the world.Business: analysing only won deals.
Three structures exhaust the ways information flows through a junction, and each carries a different rule. The middle case is the only one where the analyst's instinct to add controls is correct. Getting the other two wrong produces bias that grows with sample size rather than shrinking.

Simpson's paradox is a question about mechanism disguised as a question about arithmetic

A treatment can help every subgroup and appear to harm the population. The numbers are not in error and no amount of staring at them resolves the contradiction, because the resolution does not live in the numbers. It lives in whether the grouping variable is a common cause of treatment and outcome, in which case the segment view is correct, or a consequence of treatment, in which case the aggregate view is correct. Same table, opposite conclusions, and the deciding evidence is structural knowledge no dataset contains. This is the cleanest demonstration in the book that data cannot arbitrate its own interpretation.

Exhibit 4Simpson's paradox on a checkout redesign: better in every segment, worse in the roll-up
0%10%20%30%OLD CHECKOUT (A)NEW CHECKOUT (B)CONVERSION RATE 30.0%32.0%Desktop 12.0%14.0%Mobile 26.4%16.7%All trafficIllustrative figures. B was released disproportionately to mobile traffic,which converts lower for reasons unrelated to checkout design.
Both segment lines rise. The aggregate line falls, because the mix shifted towards the weaker-converting device. If device causes both exposure and conversion it is a confounder and the segment view rules; if the redesign changed which devices people used, device is a mediator and the aggregate view rules. The table cannot tell you which world you are in. Exhibit 4 uses illustrative figures constructed to demonstrate the structure, not observed data.

Counterfactuals are where credit, blame and attribution actually live

The third rung handles the retrospective question: given what happened, what would have happened otherwise? Pearl shows this is not a philosophical indulgence but the operative question in law, in medicine and in every attribution argument a business has ever had. Would this customer have bought without the discount? Would the outage have occurred had we not shipped on Friday? These require a fully specified structural model, because they ask about a world that never existed and therefore left no trace in any log. The formal machinery, probability of necessity and probability of sufficiency, gives shape to a conversation most organisations conduct entirely through assertion.

04 · Where the argument echoesThree fields that solved this problem before statistics did

Pearl's structural claims rhyme with patterns well outside inference. Each parallel sharpens something his own framing leaves implicit.

In reliability engineering, the fault tree has been standard practice since the 1960s. An engineer analysing a launch failure does not run a regression across historical launches; she draws the mechanism, then asks which branch, if severed, would have prevented the top event. The parallel reveals what Pearl understates about adoption. Engineers never needed persuading that a diagram precedes an estimate, because their systems were designed rather than observed. The causal revolution was hard in statistics precisely because statisticians inherited data from processes nobody had drawn.

Control theory arrived at the same conclusion through a theorem. Conant and Ashby proved in 1970 that every good regulator of a system must be a model of that system. A controller cannot steer what it does not represent. Set beside the ladder, this converts Pearl's philosophical point into an engineering constraint: any agent that acts, rather than merely reports, is committed to a rung-two model whether or not it has one written down. An enterprise deploying autonomous agents on rung-one models has not avoided the modelling problem, only hidden it inside the weights.

Thermodynamics supplies the deepest rhyme. Correlation is symmetric; causation is not, and the asymmetry has to come from somewhere. Physics faced this exact difficulty and answered it with entropy, an arrow imposed on equations that were themselves time-reversible. Pearl's directed graph performs the same manoeuvre for inference. Seen this way, the arrowheads stop looking like notation and start looking like what they are: an ontological commitment about the world, made before the data speaks, of the same species physics makes when it says the future differs from the past.

05 · The enterprise translationWhat the ladder costs when an organisation ignores it

Pearl wrote for scientists, and the book's examples run through epidemiology, genetics and clinical trials. The structural argument transfers to commercial decision-making with almost no loss, because the pathology it describes is the standard operating model of enterprise analytics: a rung-one system answering rung-two questions, with nobody in the room aware a substitution has occurred.

The most-cited demonstration comes from eBay. Tom Blake, Chris Nosko and Steven Tadelis ran a series of large-scale field experiments on paid search, published in Econometrica in 2015, by switching off branded keyword advertising in some markets and leaving it on in others. Almost all of the forgone click traffic and attributed sales were recaptured by natural search; substitution between paid and unpaid traffic was close to complete. Brand keyword ads showed no measurable short-term benefit. Returns on the other keywords came in at a fraction of what the non-experimental estimates had promised, and set against the experimental result an observational regression inflated the estimated return by somewhere between 871 and 1,198 per cent.

Exhibit 5The attribution gap: what an observational estimate reported versus what the experiment found
ESTIMATED RETURN, INDEXED TO THE EXPERIMENTAL RESULT = 1×Experiment(causal, rung 2)Observation(regression, rung 1)9.7×13.0×14×eBay paid-search field experiments. Blake, Nosko & Tadelis, Econometrica 83(1), 2015.Shaded band shows the reported range of overstatement.
The observational estimate was not slightly optimistic. It was wrong by roughly an order of magnitude, and it was wrong in the direction that justified continued spending. Branded search intercepts customers already walking through the door, so clicks and purchases correlate almost perfectly while the advertising causes very little.

The same structure recurs across functions, wearing different labels. Pricing teams estimate elasticity from transaction histories in which price was itself set in response to demand, which is Exhibit 2 exactly. Credit teams fit risk models on funded loans only. That conditions on a collider the approval process created. Marketing runs last-touch attribution: a rung-one accounting identity dressed as a rung-three claim about what would have happened otherwise. Finance decomposes a variance arithmetically, then narrates the decomposition as though arithmetic had identified causes. Nobody in these meetings says the word cause. Everyone acts on it.

The question actually being askedWhat most organisations runWhat the rung requiresEvidence in practice
Rung 1
Which accounts are likely to churn next quarter?
Supervised churn model on historical featuresNothing more. This is a prediction problem and the model is fit for it.Well served by current practice. The failure begins when the score is used to target.
Rung 2
Which accounts should receive the retention offer?
The same churn score, sorted descendingUplift modelling or a randomised holdout. High-risk is not the same as persuadable.Spend concentrates on accounts that would have stayed anyway, and on some the offer annoys.
Rung 2
What does an extra unit of ad spend return?
Regression of sales on spend, or last-touch attributionGeo-lift, ghost ads or a switch-off experimenteBay: observational estimate overstated by roughly 9× to 13× (Blake et al., 2015).
Rung 2
What happens to volume if we raise price 4%?
Elasticity fitted to historical price and volumePrice experiments, or a diagram plus a defensible instrumentDemand-driven pricing confounds the estimate, often reversing its sign.
Rung 3
Did the campaign cause this quarter's uplift?
Variance bridge, multi-touch attribution modelA structural model plus an explicit counterfactual baselineAttribution disputes become political because no method in the room can settle them.
Rung 3
Would the outage have happened without the release?
Post-incident review, narrative reconstructionProbability of necessity over a specified fault modelEngineering already does this informally through fault trees; finance rarely does.
Exhibit 6Decision authority: matching the rung of the evidence to the rung of the question
RUNG OF THE EVIDENCE THE SYSTEM HOLDSObservationalExperimentalStructural modelRUNG OF THE QUESTIONCounterfactualUNSOUNDAttribution theatreUNSOUNDAverage ≠ individualSOUNDCredit, blame, liabilityInterventionalUNSOUNDThe eBay failure modeSOUNDTest, then decideSOUNDIdentify, then estimatePredictiveSOUNDForecasting, scoringOVERBUILTCostly but validOVERBUILTCostly but valid
The three red cells are where enterprise analytics spends most of its credibility. Nothing is wrong with a rung-one system; the error is granting it authority over a question one or two levels above the evidence it holds. The rule generalises to autonomous agents: cap the agent's decision rights at the rung of its underlying model, not the rung of the question it was handed.

06 · DisciplinesFour things to do differently on Monday

07 · RelevanceWhy a 2018 book about statistics reads as an AI book in 2026

Pearl finished writing before transformers, and his target was the deep-learning enthusiasm of the mid-2010s. Eight years on the argument has aged into something sharper than he intended. Enterprises are now deploying agents that do not merely score but act: reprice inventory, reallocate media budget, adjust safety stock, approve credit. The moment a system acts, it has left rung one, and the question of what model licenses that action stops being academic.

The evidence on whether language models can climb is genuinely unsettled. Benchmarks built directly on the ladder, CLadder among them, report GPT-4 at 62% on a Pearl-hierarchy battery against 44% for LLaMa-1, a gap that admits two readings: emergent causal competence, or richer lexical priors over familiar variable names like smoking and cancer. Work published in early 2026 tested the same numerical evidence under meaningful and coded variable labels and found performance moved with the semantics, which points towards the second reading. That result does not settle anything, and it should not be read as settled. It does mean an organisation cannot outsource its causal model to a language model on the assumption that fluency about causes implies knowledge of them.

08 · Where the argument strainsFive lenses that sharpen the case rather than dismiss it

The book invites engagement from the disciplines it draws on. Five lenses bring its genuine claims into relief.

Through the lens of the potential-outcomes framework, Pearl's claim to have supplied the missing language looks overstated. Rubin, Imbens and Angrist built a rigorous apparatus for the same problems using a different notation, and design-based identification through natural experiments earned the 2021 Nobel in economics. Imbens's 2020 review essay in the Journal of Economic Literature is the fairest arbitration: the graph excels at communicating and validating assumptions; the potential-outcomes tradition excels at connecting an assumption to a credible research design. Practitioners need both, and the book only sells one.

Through the lens of structural econometrics, the novelty claim thins further. Trygve Haavelmo's 1943 work on simultaneous equations already treated structural parameters as invariant to intervention, which is the do-operator without the notation. James Heckman has argued at length that econometrics never lost causality and that do-calculus adds less than advertised, particularly for policy questions requiring extrapolation beyond the observed support. A reader who takes the book's history at face value will underestimate how much of this apparatus predates the revolution it announces.

Through the lens of statistical estimation, the graph answers only half the problem. Identification tells you whether an effect is recoverable in principle from infinite data; it says nothing about estimating it well from the finite, high-dimensional, badly measured data an actual enterprise holds. That gap is where the productive work of the past decade has happened, in double machine learning and targeted maximum likelihood. Pearl's framework and Chernozhukov's estimators are complements, and treating the first as sufficient produces confidently identified estimates with uselessly wide intervals.

Through the lens of machine-learning research, the book's verdict on deep learning has become harder to score. Pearl declared such systems stuck on rung one, curve-fitters incapable of imagining. Language models now produce fluent counterfactual reasoning across an enormous range of scenarios, which either falsifies the claim or confirms it depending on whether the fluency rests on structure or on pattern. The honest position in 2026 is that the question is live, the benchmarks are contested, and Pearl's framework remains the sharpest available instrument for asking it, even if his answer may not survive.

Through the lens of organisational epistemology, the diagram runs into people. Pearl assumes a modeller who knows the mechanism and can commit to it. Inside a firm, the arrows encode contested beliefs about how the business works, and drawing them exposes disagreements that ambiguity had been protecting. Chris Argyris spent a career on precisely this gap between espoused theory and theory-in-use. The graph is a superb instrument for making disagreement explicit, which is exactly why organisations resist drawing one, and the book has nothing to say about that resistance.

Refracted through these lenses, the friction above are not objections to Pearl and Mackenzie's perspective. They are scaffolding the reader can build around the book's argument, places to anchor their own engagement and test the framework against disciplines it touches. What the authors built survives the engagement, and the reader gains a firmer grip on why it does. The ladder itself is not in dispute; what the lenses expose is that the ladder is a diagnostic instrument rather than a complete method, and that the estimation, design and organisational work it leaves undone is where most of the effort actually lies.

09 · Adjacent readingWhere to go next

Neighbours · same territory, different angle

  • Hernán & Robins, Causal Inference: What If (2020) — the graph-native treatment worked through real epidemiological designs, and freely available.
  • Cunningham, Causal Inference: The Mixtape (2021) — the same terrain with code, aimed squarely at analysts who need to ship an estimate.
  • Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments (2020) — rung two as an operating discipline at industrial scale, written by people who ran it.

Productive adversaries · books that complicate this one

  • Imbens & Rubin, Causal Inference for Statistics, Social, and Biomedical Sciences (2015) — the potential-outcomes case, made by its architects rather than characterised by a rival.
  • Angrist & Pischke, Mostly Harmless Econometrics (2009) — identification through research design instead of graph surgery, and a reminder that credibility often comes from the setting rather than the notation.

Deeper roots · what the book stands on

  • Pearl, Causality (2nd ed., 2009) — the technical original, for readers who want the proofs rather than the story.
  • Fisher, The Design of Experiments (1935) — the randomisation argument in its first and still-clearest form, from the man the book casts as antagonist.
Start here If you run a commercial function rather than a research group, read Kohavi, Tang and Xu next. Pearl tells you why your attribution numbers are wrong; Kohavi tells you what to build on Monday so they stop being wrong, and the two books together convert a philosophical objection into an operating change.

10 · ClosingThe drawing that ended the prohibition

Pearson banished the word because he could not formalise it, and a discipline that cannot formalise a concept eventually decides the concept is unscientific. That instinct was not stupid; it was the cost of rigour paid in the wrong currency. What Pearl supplied was not permission to speak loosely about causes but a grammar strict enough that causal claims could finally be wrong in public. The arrows commit you. They can be contradicted by the data, disputed by a colleague, and shown to render your question unanswerable at any sample size. A century of statistical caution was, in the end, waiting for exactly that. The modern enterprise has inherited Pearson's discomfort without inheriting his rigour: it avoids the word "cause" in its documentation and assumes it in every decision.

The authors' charge Stop treating the pursuit of more data as the path to understanding, and start writing down what you believe about how the world works. The model is not a concession to ignorance that better data will eventually retire. It is the thing that makes the data mean anything at all.

11 · What to carry away

The rungs are a logical hierarchy, not a maturity curve. Rung-one data cannot answer a rung-two question at any sample size, and better models do not close the gap.

The diagram's value is that it can be wrong. Drawing assumptions makes them testable, falsifiable and arguable; muttering them makes disagreement unresolvable.

More controls is not more rigour. Adjust for confounders, never for mediators or colliders, and remember that filtering your sample is silent conditioning.

Enterprise analytics systematically substitutes rungs. Attribution, elasticity and retention targeting are rung-two or rung-three questions routinely answered with rung-one machinery, and eBay's experiments put a number on the cost.

An agent that acts is committed to a causal model. Autonomy should be capped at the rung of the system's underlying model, not the rung of the question it was given.