Practice: Potential Outcomes and the Fundamental Problem

(a) Unit 1 is treated: $Y^{\mathrm{obs}}=26$, with $Y_1(0)=20$ unobserved. Unit 2 is control: $18$, with $Y_2(1)$ unobserved. Unit 3 is treated: $25$, with $Y_3(0)$ unobserved. Unit 4 is control: $22$, with $Y_4(1)$ unobserved.

(b) The individual effects are $6,3,0,8$, so ${\tau }_S=17/4=4.25$.

(c) Treated observed outcomes are $26$ and $25$, giving ${\overline{Y}}_1=25.5$. Control observed outcomes are $18$ and $22$, giving ${\overline{Y}}_0=20$. So $\hat{\tau }=5.5$, against ${\tau }_S=4.25$. The two differ because the estimator compares different units rather than the same unit under both treatments: this assignment placed unit 4, whose $Y(1)$ is high, in control, and unit 3, whose effect is zero, in treatment. Over all allowed assignments the difference in means averages to ${\tau }_S$; on this one realisation it does not.

(d) Nothing about unit 3's effect. The study observes $Y_3(1)=25$ and nothing else for that unit; ${\tau }_3=0$ is visible here only because the table stipulates $Y_3(0)$, which a real study would not supply. No amount of data on the other three units identifies it.

(d) The target is the average treatment effect $\tau =\frac{1}{4}\sum _i(Y_i(1)-Y_i(0))$, a property of these four units. The rule is the difference in means $\hat{\tau }={\overline{Y}}_1-{\overline{Y}}_0$, applied to whichever units happened to be treated. Because the table stipulates both potential outcomes, $\tau$ can be computed exactly here, which is precisely what no real experiment permits, and why the estimator exists.

(a) A contrast between potential outcomes. Random assignment makes the treated group's observed mean an estimate of $E[Y(1)]$ and the control group's an estimate of $E[Y(0)]$ for the same population, so the difference estimates an average causal effect.

(b) Not a causal contrast. This compares the same patient at two times, so the second term is the intake score rather than $Y_i(0)$, what the score would have been at follow-up without the programme. Those differ whenever anything else changes over the period, including natural recovery.

(c) Not a causal contrast, and the worst of the four. Completion is determined after treatment begins and is plausibly caused by how the patient was responding, so the groups are defined by a post-treatment variable. There is no well-defined $Y(0)$ for 'this patient, untreated, in the completer group'.

(d) A contrast between groups that may differ for reasons other than the programme, because the clinics selected themselves. It compares $E[Y\mid \text{adopting clinic}]$ with $E[Y\mid \text{non-adopting clinic}]$. It becomes a causal contrast only under an identification assumption about how clinics chose, which the design does not supply.

Targets and rules. A contrast between potential outcomes for the same units targets an average treatment effect, with a difference in means as its estimator. The others target something else entirely, a difference between populations, or a change over time, and no rule applied to them approximates a causal effect. Saying which quantity is wanted before which rule is applied is what keeps the two from merging into "the effect".

First error: a between-unit difference is called a within-unit effect. Student A's effect is ${\tau }_A=Y_A(1)-Y_A(0)$, both terms belonging to Student A. The figure 7 is $Y_A^{\mathrm{obs}}-Y_B^{\mathrm{obs}}$, a comparison of two different students who differ in more than tutoring. It is not ${\tau }_A$, and nothing licenses treating it as such.

Second error: the averaging story is backwards. The difference in means is not an average of computed individual effects, because no individual effect was ever computed. It is a difference of two group averages, and randomization makes it unbiased for the average effect directly, not by averaging quantities that were each separately identified.

Third error: sample size is offered as the remedy. More students do not complete any existing student's row. Each additional unit contributes one observed potential outcome and one missing one. Sample size improves precision about an average; it never identifies an individual effect, because the obstacle is structural rather than statistical.

What the trial can report about Student A: her observed score under tutoring, $Y_A(1)=78$, and nothing else about her personally. Any statement about what she would have scored untutored is an inference from other students, justified, if at all, by the design, and it is a statement about an average rather than about her.

Estimand against estimator. The report uses "the effect" for three different things: the quantity it wants (the average of $Y_i(1)-Y_i(0)$ over students), the rule it applied (a difference between two students' scores), and the number that came out. Naming them separately is what exposes the error. The rule it used does not approximate the quantity it wants, because the two students are not each other's counterfactuals.

(a) A difference between two group averages: the average change among districts that built lanes and the average change among those that did not. Both are averages over observed districts.

(b) The question asks what would have happened in the lane-building districts had they not built lanes. That quantity was never observed for those districts, and the non-building districts are a stand-in for it rather than a measurement of it.

(c) The 22-point figure is an average over districts. The claim that each district gained roughly 22 points asserts something about every individual district, which requires knowing what each would have done without lanes. A quantity no study of this kind produces. Districts almost certainly differ: a dense district with existing cycling culture and a car-dependent suburb need not respond alike, and the average can be composed of very different district-level changes, including some near zero.

(d) The comparison supports the weaker average claim only if the districts that built lanes would, absent the programme, have changed like those that did not. Since the city chose where to build, plausibly where cycling was already growing, or where campaigning was strongest, that is an assumption about the selection process, not something the data establish. Without it the 22 points mixes the effect of the lanes with whatever made those districts get lanes first.

What was wanted and what was done. The claim needs the average over districts of $Y_d(1)-Y_d(0)$: cycling with the lanes against cycling in the same districts without them. What was computed is a difference between districts that got lanes and districts that did not. A rule that approximates the wanted quantity only if the two groups would have moved alike, which is exactly what choosing where to build lanes makes doubtful.