Potential Outcomes and the Fundamental Problem

Every unit has an outcome under treatment and an outcome without it. You observe exactly one of them, ever. Everything in causal inference follows from that.

That sounds like bookkeeping. It is the distinction the whole subject rests on: every design, estimator and test that follows is a way of standing in for a number that was never observed, and a learner who cannot see which number is missing cannot judge whether the substitute is credible.

Give every unit two sealed envelopes. One says what happens to it if treated; the other says what happens if it is not. Both are written before anyone decides anything. They are facts about the unit, not predictions.

Assigning a treatment opens one envelope and destroys the other.

The causal effect for that unit is the difference between the two numbers. It is well defined, and it is not observable for any unit in any study, at any sample size.

This is why causal statements are about averages. Not because averages are more interesting than individuals, but because the individual quantity is structurally unavailable and the average is not.

For each unit $i$ and a binary treatment, define two potential outcomes:

Both are defined for every unit, whichever treatment is actually assigned. The individual treatment effect is their difference:

Let $W_i\in \{0,1\}$ record the assignment. The observed outcome is selected by it:

When $W_i=1$ this returns $Y_i(1)$; when $W_i=0$ it returns $Y_i(0)$. The other potential outcome is not merely unknown. It is not generated by the study at all.

The fundamental problem of causal inference is that ${\tau }_i$ requires both halves and every study supplies one. Averages remain reachable: for a finite sample of $N$ units,

and for a population, $\mathrm{ATE}=E[Y(1)-Y(0)]$.

Six units, with the potential outcomes stipulated so the arithmetic is visible. In a real study the shaded half of this table does not exist.

The finite-sample average effect is ${\tau }_S=(4+3+2+3+2+4)/6=3$.

Now discard what a study never sees. The treated units are 1, 3, 6 with observed outcomes 14, 11, 17, giving ${\overline{Y}}_1=14$. The control units are 2, 4, 5 with 12, 15, 11, giving ${\overline{Y}}_0\approx 12.67$. The difference in means is $\hat{\tau }\approx 1.33$.

The estimate is not 3, and nothing went wrong. This assignment happened to place units with high $Y(0)$ in control. Across every allowed assignment the difference in means averages to ${\tau }_S$; on any single one it need not. The gap is randomization error, not a flaw in the design and not evidence of bias.

Problem. A clinic enrols four patients in a trial of a new analgesic. The outcome is pain score after two hours, lower being better. For the purpose of this example the full schedule of potential outcomes is stipulated; a real trial would supply only one column.

Goal. Report what the trial observes, what it can estimate, and what it cannot.

Relevant principle. The assignment selects one potential outcome per unit; the other is not measured, not zero, and not recoverable from other units.

Step 1: apply the selection identity.
$Y_i^{\mathrm{obs}}=W_i{Y}_i(1)+(1-W_i)Y_i(0)$.
Patient A: $W=1$, so $Y^{\mathrm{obs}}=4$, and $Y_A(0)=7$ goes unobserved.
Patient B: $W=0$, so $Y^{\mathrm{obs}}=6$, and $Y_B(1)$ goes unobserved.
Patient C: $W=1$, so $Y^{\mathrm{obs}}=8$, and $Y_C(0)$ goes unobserved.
Patient D: $W=0$, so $Y^{\mathrm{obs}}=5$, and $Y_D(1)$ goes unobserved.
Reason: the indicator picks a column; nothing about the other column is measured.

Step 2: compute the estimand from the stipulated table.
${\tau }_A=-3$, ${\tau }_B=-1$, ${\tau }_C=0$, ${\tau }_D=-3$, so

Reason: the finite-sample average effect needs both columns, which is exactly why it is unavailable in practice.

Step 3: compute what the trial can actually compute.
Treated: A and C, with observed $4$ and $8$, so ${\overline{Y}}_1=6$.
Control: B and D, with observed $6$ and $5$, so ${\overline{Y}}_0=5.5$.

Reason: only the observed column is available, and the estimator compares groups rather than a unit with itself.

Result. The trial reports $\hat{\tau }=0.5$ while ${\tau }_S=-1.75$. The estimate has the wrong sign.

Check. Nothing was miscalculated. Patient C, for whom the drug does nothing, was assigned to treatment and has the worst pain score in the table; patient D, who benefits most, was assigned to control and never received the drug. The assignment happened to pair a high baseline with treatment.

Interpretation. With four units this is unremarkable: the difference in means is unbiased over repeated assignments, not correct on any one of them. The lesson is not that the estimator is broken but that a single small experiment carries real randomization error, and that ${\tau }_S$ was never observable to check it against. Precision comes from units, not from better arithmetic.

A before-and-after comparison. A clinic measures pain before the drug and two hours after, and reports the change as the effect. The two measurements are the same patient at two times, not the same patient under two treatments. $Y_i(0)$ is what the pain would have been at two hours without the drug, which is not the pain before taking it, pain drifts, resolves, and responds to rest. A before/after difference is a potential-outcomes contrast only under the additional assumption that nothing else changed, which is usually the assumption in question.

A variable measured after treatment. A trial reports the effect of the drug on pain among patients who reported no side effects. Side effects are caused by the treatment, so conditioning on them splits the sample by something the treatment determined. There is no $Y_i(0)$ for "this patient, untreated, in the no-side-effect group", because the group itself does not exist without the treatment.

An outcome that depends on another unit's assignment. In a vaccine trial in one household, an untreated person's infection risk falls when a housemate is vaccinated. Then $Y_i(0)$ is not well defined: the notation assumes a unit's outcome depends on its own assignment alone. This is a failure of the stable-treatment assumption, and it requires richer notation rather than more data.

A prediction from a model. A regression predicts what an untreated patient "would have scored" and the fitted value is called the counterfactual. The model output is an estimate of a conditional mean over units with similar covariates. It may be a reasonable stand-in, but it is not $Y_i(0)$, and calling it so hides every assumption that makes the substitution credible.

The missing potential outcome is not any of the things learners reach for to fill it.

Not the other group's outcome. Unit 1 was treated and scored 14. Unit 2 was a control and scored 12. The difference, 2, is not unit 1's effect and not unit 2's: it compares two different units, and ${\tau }_1$ happens to be 4.

Not the other arm's mean. Substituting ${\overline{Y}}_0$ for each treated unit's $Y_i(0)$ produces a number, and that number is the difference in means again. A sample average, carrying no information about any individual.

Not recoverable with more data. A thousand more units supply a thousand more half-rows. They sharpen the estimate of an average; they do not complete a single row already in hand.

Not a measurement problem. Better instruments measure what happened more precisely. No instrument measures what would have happened under a treatment that was not given.

Work these in order. The first supplies most of the structure; the last supplies none.

1: fully structured. Three units, with the table given in full.

(a) Write $Y_i^{\mathrm{obs}}$ for each unit. (b) Which potential outcome is missing for unit 2? (c) Compute ${\tau }_S$. (d) Compute $\hat{\tau }$ from the observed data.

Check: observed are $15$, $9$, $11$; unit 2 is missing $Y_2(1)=14$; ${\tau }_S=(3+5+0)/3\approx 2.67$; $\hat{\tau }=13-9=4$.

2: partly structured. A study of six units reports ${\overline{Y}}_1=22$ from four treated units and ${\overline{Y}}_0=19$ from two control units. A reader asks what the effect was for the third treated unit.

Answer the reader, and say which quantity the study does support.

Check: the study supports $\hat{\tau }=3$ as an estimate of an average effect. It supports nothing about the third treated unit, whose $Y(0)$ was never generated.

3: unstructured. A colleague proposes: "Run the experiment, then for every treated unit subtract the mean of the controls. That gives each treated unit's individual effect, and averaging them recovers the average effect."

The second half of the claim is arithmetically fine. Explain precisely what is wrong with the first half, and why the two halves can differ in validity even though the same subtraction appears in both.

Check: subtracting a group mean produces a number per unit, but that number estimates nothing about the unit, its $Y(0)$ was never observed, and the control mean carries no unit-specific information. Averaging those numbers gives ${\overline{Y}}_1-{\overline{Y}}_0$, which is a valid estimator of the average effect. A valid average can be assembled from quantities that are individually meaningless.

The two quantities. Every unit has $Y_i(1)$ and $Y_i(0)$. Both are properties of the unit; both are defined whether or not the unit is assigned that treatment.

The effect. ${\tau }_i=Y_i(1)-Y_i(0)$, for that unit.

What a study sees. $Y_i^{\mathrm{obs}}=W_i{Y}_i(1)+(1-W_i)Y_i(0)$. The assignment selects one; the other is never generated.

The fundamental problem. No individual effect is identified, in any study, at any sample size. This is structural, not statistical.

What remains reachable. Averages: ${\tau }_S=\frac{1}{N}\sum _i[Y_i(1)-Y_i(0)]$ for a fixed set of units, or $\mathrm{ATE}=E[Y(1)-Y(0)]$ for a population.

The three words to keep apart. The estimand is the quantity wanted; the estimator is the rule applied to the sample; the estimate is the number that rule returns. "The effect" names all three in careless writing and none of them precisely.

The recurring error. Filling a missing $Y_i(0)$ with the other arm's outcome, the other arm's mean, a pre-treatment measurement, or a model's fitted value, and then speaking about an individual.