Practice: Unconfoundedness and Overlap

Unconfoundedness here. Among students with the same prior grade, the same parental education and the same school, whether a student attended the camp is independent of the programming grade they would obtain under either attendance state. Equivalently: within those three variables, attendance is as good as randomly assigned.

Overlap here. At every combination of prior grade, parental education and school that appears in the population the estimate is meant to describe, both attending and not attending must have positive probability. If, say, every high-prior-grade student at one school attended, no comparable non-attender exists there.

Which the data can address. Overlap, by estimating the attendance probability from the covariates and inspecting its distribution within each group, looking for values near 0 or 1 and for regions occupied by only one group. Post-adjustment balance is also checkable. Unconfoundedness is not: it concerns grades never observed and variables never recorded.

Why to doubt it here. Enrolment was voluntary, so whatever led a student to sign up is a candidate confounder. Prior interest in programming, family encouragement and access to a home computer plausibly raise both the chance of attending and later grades, and none of the three covariates captures them. The estimate would then attribute to the camp what belongs to the disposition that produced attendance.

Timing of the covariates. Prior grade and parental education are fixed before the camp, so they are admissible. Anything recorded during or after it, attendance, engagement, teacher reports from that term, is not, however predictive: the camp could have influenced it, and adjusting for it would remove part of the effect being estimated.

The error. The argument runs from a fact about 14 measured variables to a conclusion about all variables. Nothing in the observed data supports that step.

What the balance table establishes. That the matching procedure succeeded at what it was asked to do: the matched groups have similar distributions on the 14 covariates supplied to it. This is a diagnostic of the procedure, and worth reporting.

What it cannot establish. Unconfoundedness is the claim that those 14 are sufficient, that no further common cause of treatment and outcome is missing. A study balanced on every measured covariate can still be confounded by any unmeasured one, and the balance table would look exactly the same. Balance is necessary for a credible analysis, not sufficient for a causal one.

What the report should contain instead. The assumption stated explicitly, with an argument from subject-matter knowledge about what drove treatment and whether the 14 capture it; overlap diagnostics, since matching can achieve balance by discarding units and thereby change the estimand; a statement of which population the matched sample now represents; and a sensitivity analysis asking how strong an unmeasured confounder would have to be to overturn the conclusion. What cannot be offered is a demonstration that confounding was eliminated.

A second question the report does not ask. Whether all fourteen covariates were measured before treatment. Balance on a post-treatment variable is not reassurance, adjusting for something the treatment influenced can remove part of the effect or open a non-causal path. The adjustment set has to be pretreatment common causes, not the widest set available.

Overlap — and the evidence does speak to it. The reported figure is a direct measurement of a problem. For 22% of non-adopters, the model says adoption was all but impossible given their sector, size, growth and region; there are correspondingly few adopting firms resembling them. Overlap fails over a substantial part of the sample.

Consequence. Any estimate covering those firms rests on extrapolation from a fitted model rather than on comparison with similar treated firms, and if they are retained with weights their influence becomes extreme, so the estimate turns on a handful of observations. Trimming them restores a comparison supported by data but changes the estimand: the result then describes the subpopulation with genuine overlap, and the paper must say so rather than continue to claim an average over all firms.

Unconfoundedness — and the evidence does not speak to it. Adoption was the firm's own decision, which is the signal to look for. Management quality, owner ambition, organisational slack and the capacity to absorb change plausibly drive both the decision to adopt and subsequent performance. None is among the four covariates, and none is easy to measure.

Consequence. The comparison confounds the practice's effect with the qualities that led a firm to adopt it, most plausibly inflating the estimate, since capable firms both adopt more readily and perform better anyway. No diagnostic in the paper would reveal this: a balance table on the four covariates would look fine.

The asymmetry. One assumption was measured and found wanting; the other was never measurable. They call for different responses, trimming and restating the estimand for the first, subject-matter argument and sensitivity analysis for the second, and a single verdict on 'the assumptions' would obscure both.

Which variables may enter. Sector, prior revenue growth and region precede adoption and are admissible. Firm size is admissible only if measured before adoption; if it is current size, the practice could have affected it, and conditioning on it would block part of the pathway under study.