Practice: Neyman Repeated-Sampling Inference

(a) $\widehat{\mathrm{Var}}(\hat{\tau })=160/80+180/120=2+1.5=3.5$, so $SE=\sqrt{3.5}\approx 1.87$.

(b) $2.5\pm 1.96(1.87)=(-1.17,6.17)$. The interval includes zero, so the trial does not establish a non-zero average effect at this level.

(c) Exactly, $\mathrm{Var}(\hat{\tau })=\frac{S_1^2}{80}+\frac{S_0^2}{120}-\frac{S_{\tau }^2}{200}$. The calculation omits $S_{\tau }^2/200$, the variance of the individual treatment effects scaled by $N$.

(d) Wider, or at worst equal. $S_{\tau }^2$ is a variance and so is non-negative, and it enters the exact expression with a minus sign; omitting it therefore cannot reduce the reported variance. This holds whatever its value, which is why the answer does not depend on knowing it. The two coincide exactly when every unit has the same treatment effect.

(c) The omitted term is $S_{\tau }^2/N$, the variance of the unit-level effects. Each ${\tau }_i=Y_i(1)-Y_i(0)$ requires both potential outcomes for the same unit, and each unit was assigned once, so one of the two is never observed. Enlarging the trial multiplies half-observed units; it never completes a pair. The interval is therefore conservative by construction, not by approximation.

(i) Appropriate, and conservative. The interval refers to the finite-sample average treatment effect over the units in the trial, with randomness coming from the assignment.

(ii) The same arithmetic, a different justification. Randomness comes from sampling, the units are not fixed, and the interval refers to a difference of population means. Numerically identical, conceptually distinct, and the finite-population correction that appears in (i) has no analogue here.

(iii) Not appropriate. Pairing makes the two arms dependent by construction: the randomization distribution is over one choice per pair, $2^{30}$ allocations, not over all ways of splitting 60 units. The pair-difference standard error $s_D/\sqrt{J}$ should be used. Ignoring the pairing generally overstates the standard error, discarding the precision the design was built to gain.

(iv) Appropriate but no longer conservative. It becomes exact. Constant effects make $S_{\tau }^2=0$, so the omitted term is zero rather than merely unknown. The assumption is not checkable from the data, so reporting on that basis should say so.

Why the term is missing. In the design-based cases the omission traces to the fundamental problem: $S_{\tau }^2$ is a variance of unit-level effects, and no unit reveals both of its potential outcomes. In the two-population survey there are no unit-level effects to have a variance, the two samples describe different people, so the same formula is exact there for a different reason, not conservative.

What the plan computes. For a treated unit, outcome minus the control-arm mean is $Y_i(1)-{\overline{Y}}_0$. That is not ${\tau }_i=Y_i(1)-Y_i(0)$: the second term belongs to different units. Its sample variance mixes the variation in $Y(1)$ across units with nothing about how effects differ, so it does not estimate $S_{\tau }^2$. Substituting it would produce a number, and subtracting that number would yield an interval narrower than the design supports.

Sample size. Four thousand units give four thousand rows with one entry each. No ${\tau }_i$ is formed for any of them. The obstacle is that the quantity is not identified, and identification is not something more observations supply.

"Approximation" is the wrong word. An approximation is close to a target and improves with effort. This estimator is conservative: its expectation is at least the true variance, so intervals are at least as wide as warranted and tests reject no more often than their level. That is a guarantee, in a known direction, not an error term. It is exact when every unit shares the same effect. A condition the data cannot confirm.

What to do instead. Report the conservative interval. If there is a substantive reason to believe effects are near-constant, say so explicitly as an assumption, and note that the data cannot check it.

Why no sample size rescues it. $S_{\tau }^2$ is the variance of the unit-level effects ${\tau }_i=Y_i(1)-Y_i(0)$. Computing a single ${\tau }_i$ needs both potential outcomes for that unit, and assignment reveals exactly one. Adding units adds more half-observed pairs, never a complete one, so the term is unidentified by the structure of the problem rather than by a shortage of data or a defect in the estimator.

First claim — 'worth between £310 and £3,370 to a participant'. The interval covers the average effect across the 1,200 applicants, not any individual's gain. It is compatible with every participant gaining about £1,840, and equally with half gaining £3,700 and half gaining nothing. To say what a participant gained, the journalist would need that participant's ${\tau }_i$, which requires their earnings both with and without the programme. One of which never happened.

Second claim — 'the study was too small to detect' individual variation. This misidentifies the obstacle. Sample size determines the width of the interval around the average; it does nothing about the spread of individual effects. The quantity describing that spread is $S_{\tau }^2$, and it is not identified at any sample size, because forming even one ${\tau }_i$ needs both potential outcomes for one person. A trial of 120,000 applicants would give a much narrower interval around the average and exactly as little information about who gained most.

What the trial supports. A positive average effect on twelve-month earnings for this population of applicants, with the stated uncertainty, and that uncertainty is conservative, since the reported variance omits a non-negative term.

What would be needed instead. Statements about who benefits require either strong assumptions about effect heterogeneity, or a design targeting subgroup effects defined by pre-treatment characteristics, which estimates averages within subgroups, still not individual effects.

Would a larger trial pin it down? Not that component. A larger trial shrinks $s_1^2/N_1+s_0^2/N_0$, so the interval does narrow. But the piece deliberately left out, $S_{\tau }^2/N$, is the variance of individual effects, and an individual effect needs both of that person's potential outcomes. Each applicant was either trained or not. No trial size changes that, so the conservatism survives at any scale.