Practice: Regression Adjustment in Experiments

Admissible covariates. Baseline score and school are fixed before assignment and belong. Baseline score in particular should explain a large share of outcome variation, so it is the one most likely to sharpen the estimate.

Inadmissible covariate. Books read during term is measured after the intervention began, and a reading intervention plausibly changes it. It is post-treatment and must be dropped.

What including it would do. Books read is very likely a mediator, part of how the intervention works is by increasing reading. Conditioning on it removes that pathway, so $\tau$ would capture only the effect not operating through books read: a different quantity, and probably a much smaller one. Conditioning on a consequence of treatment can also induce association between treatment and unobserved determinants of the outcome, so the remainder need not be interpretable even as a partial effect.

What adjustment contributes. Precision only. The arms are already comparable in expectation because of randomization, so the estimate should not move much; what changes is the residual variance and hence the standard error.

Where the causal claim comes from. The assignment mechanism. It would be equally causal with no covariates at all, using the raw difference in means, merely less precise. No covariate in this specification is carrying identification.

Matching the analysis to the design. With individual randomization, heteroskedasticity-robust standard errors are appropriate; conventional ones assume a homoskedasticity there is no reason to expect. Had assignment been blocked, the block indicators belong in the regression; had it been clustered, the errors must be clustered at the unit of assignment; unequal assignment probabilities would call for weights.

The error. Covariates are being credited with repairing a failed design. For the affected period, treatment was not assigned by a known mechanism, so nothing about the design licenses a causal reading, and adding controls does not restore it.

What the analysis has silently become. An observational study. The five covariates are now carrying identification, which means the estimate rests on unconfoundedness given site, week, age, sex and baseline severity. That assumption was never planned for, and it is implausible here: staff allocating manually may have steered patients on clinical judgement, urgency or perceived likely benefit, none of which is recorded. If any such consideration affects the outcome, the adjusted estimate remains confounded.

Why significance is not reassurance. A confounded estimate can be precisely estimated and significant. The p-value speaks to sampling variability, not to whether the comparison identifies an effect.

What the report should state. The breakdown, plainly, with the number of participants affected. The primary analysis restricted to the properly randomized period, where the design still supports the claim. The full-sample adjusted analysis presented as a secondary, observational estimate with its assumption stated explicitly. A sensitivity analysis for the manually allocated portion, asking how strong an unmeasured confounder would need to be to overturn the conclusion. And, if possible, an account from the staff of the criteria they actually used, which is evidence about the assignment mechanism that no statistical adjustment can substitute for.

A further point about the specification. Even where randomization held, the regression has to match the mechanism: site-level allocation makes site a design feature rather than a nuisance covariate, and errors would need clustering at the level assignment occurred. Controlling for site as though it were an ordinary covariate treats a design fact as a statistical adjustment.

Two of the three controls are fine. Account age and device type are fixed before the user enters the experiment and are admissible precision covariates.

Pages viewed in the session is not. It is measured during the session, after exposure to the new checkout flow, and the flow plausibly changes how many pages a user views. Conditioning on it removes part of the effect, if the new flow works partly by reducing the pages needed to complete a purchase, that pathway is being held fixed, and can induce association between the variant and unobserved user characteristics. Much of the apparent precision gain is likely coming from this variable, which makes the tightened standard error misleading rather than reassuring.

On reporting only the adjusted figure. Both should be reported. Showing +1.2pp raw beside the adjusted estimate lets a reader see how much of the conclusion depends on the modelling. Suppressing the raw difference hides exactly what most needs checking, particularly when a post-treatment control is in the specification.

Where the causal claim comes from. The randomization, which was sound. The raw +1.2pp was already a valid causal estimate; the covariates were only ever going to sharpen it.

What I would change. Drop pages viewed. Refit with account age and device type. Report the raw difference alongside the adjusted estimate. If session engagement is itself of interest, analyse it as a secondary outcome. A legitimate question, and a different one from the effect on purchase completion.

What the design requires here. Individual randomization with a binary outcome: robust standard errors, not conventional ones, since a binary outcome is heteroskedastic by construction. No block indicators or clustering are needed because assignment was neither blocked nor grouped, which should be stated rather than assumed.