Regression Adjustment in Experiments

Adding pretreatment covariates to a regression on experimental data can reduce the variance of the estimate. It is not what makes the estimate causal, and treating it that way is how observational studies come to sound like trials.

The regressions are elementary. The competence is knowing that the treatment coefficient's meaning was settled before the model was fitted, and that no covariate can change it.

Regress the outcome on an intercept and a treatment indicator, nothing else. The coefficient that comes back is the difference in means, exactly, as algebra, not as an approximation. The regression has introduced no new information. It has restated a comparison you could have computed with two averages.

That settles where the causal content lives. If the difference in means deserved a causal reading, it was the assignment mechanism that earned it. Running the same number through least squares changes the notation and nothing else.

Now add a baseline measurement of the outcome as a covariate. Randomization already made the arms comparable on it in expectation, so the estimate should not move much, and if it lurches, that is a signal about the draw rather than a correction of it. What adjustment does is soak up outcome variation that treatment had nothing to do with. Less residual noise, a smaller standard error, a tighter interval around the same quantity.

So in an experiment, adjustment is for precision. Identification was already secured, and the covariates are optional.

The trouble starts when that habit meets a study where identification was not secured. There, covariates are the entire basis for believing the comparison. The regression looks the same in both places, which makes it easy to believe the tool does the same work, and from there it is a short step to thinking a few controls can repair a design that failed. They cannot. Adjusting a confounded comparison yields an adjusted estimate of a confounded quantity, presented with more apparatus.

Bare treatment regression.

With an intercept and a binary indicator, OLS gives the algebraic identity

The regression reproduces the difference in means. Random assignment is what gives that coefficient its causal interpretation.

Additive adjustment.

with $X_i$ containing pretreatment covariates. Adjustment absorbs outcome variation explained by $X$ and typically improves precision. It constrains the covariate-outcome relationship to be identical in both arms.

Fully interacted adjustment.

Interactions let that relationship differ by arm. If the covariates are centred at the full-sample mean, $\tau$ is directly interpretable at that reference point, and the arm-specific fitted means can be averaged to estimate an average effect.

In all three, the causal reading rests on the assignment mechanism.

A randomized trial of a tutoring programme has 400 students. The outcome is an end-of-year test score; a baseline score from the previous year is available.

Unadjusted. ${\overline{Y}}_1-{\overline{Y}}_0=4.2$ points, $SE=1.9$.

Adjusted for baseline score. $\hat{\tau }=4.3$ points, $SE=0.8$.

Read the two numbers. The estimate barely moved, which is what randomization leads you to expect: the arms were already comparable on baseline score, so removing its influence does not shift the comparison. The standard error more than halved, because baseline score explains much of the variation in end-of-year score, and that variation was previously sitting in the residual.

That is adjustment working exactly as advertised, same quantity, sharper estimate.

A contrasting case. Suppose the adjusted estimate had come back at 7.1 instead. Nothing has been corrected; something has been revealed. A shift that large says the arms differed substantially on baseline score, which in a properly randomized trial of 400 students is an unlucky draw worth reporting rather than quietly adjusting away. The pre-specified analysis still stands, and the discrepancy belongs in the write-up.

Why the unadjusted number is still reported. Presenting only the adjusted figure makes the modelling invisible. Showing both lets a reader see how much of the result depends on the specification, which is precisely the thing a reader cannot otherwise check.

Problem. A team runs a randomized field experiment on a job-placement service. 1,200 unemployed applicants are individually randomized to the new service or the standard one. The outcome is employment status at six months. They propose:

and plan to report $\hat{\tau }$ with conventional standard errors.

Goal. Say which covariates belong in the specification, what the adjustment contributes, and what must change.

Relevant principle. In a randomized experiment, identification of the treatment effect comes from the assignment mechanism. Adjusting for pretreatment covariates may reduce the variance of the estimate; it does not supply identification. A covariate that the treatment could have influenced is not admissible.

Step 1: screen the covariates by timing. Age and prior earnings are determined before assignment and are admissible. Sessions attended and CV completed are measured after the service began.

Reason: the new service can change how many sessions a person attends and whether they complete a CV, so both are post-treatment variables.

Step 2: state the effect of including a mediator. Sessions attended is plausibly a mediator, since part of the service's effect operates through increased engagement. Conditioning on it removes that pathway, so $\hat{\tau }$ would estimate only the effect not operating through attendance, which is a different estimand from the one the trial was designed to measure.

Reason: holding a mediator fixed asks what the treatment does among units whose engagement did not change.

Step 3: note the second problem with post-treatment adjustment. Conditioning on a post-treatment variable can induce association between treatment and unobserved determinants of the outcome, so the remaining estimate need not be interpretable as a partial effect either.

Reason: the covariate is a common consequence of treatment and of unobserved characteristics, and conditioning on a common consequence induces dependence between them.

Step 4: use heteroskedasticity-robust standard errors. With individual randomization and a binary outcome, the residual variance plausibly differs by arm and by covariate value, and conventional standard errors assume it does not.

Step 5: fix the specification before examining treatment effects. Fix the covariate list in advance and report the unadjusted difference in means beside the adjusted estimate.

Reason: selecting covariates after observing their effect on $\hat{\tau }$ makes the specification a function of the outcome data, which invalidates the reported uncertainty.

Result. Retain age and prior earnings. Drop sessions attended and CV completed. Use robust standard errors. Report the unadjusted difference alongside the adjusted estimate.

Check. Dropping the two post-treatment covariates may widen the confidence interval. The narrower interval obtained by including them corresponds to a different estimand, so it is not a gain in precision for the quantity of interest. If engagement is itself of interest, it can be analysed as a secondary outcome, which is a separate analysis supporting a separate claim.

Interpretation. The corrected specification estimates the effect of being offered the new service on employment at six months. Adjustment for the two admissible covariates may reduce variance; identification rests on the randomization.

Adjusting after a compromised randomization. If assignment was subverted, staff steering certain applicants, a broken allocation sequence, covariates do not restore it. The result is an adjusted estimate of a confounded comparison, and the assumption it now rests on is unconfoundedness, which the experiment was designed to avoid needing.

Adjusting for differential attrition. When dropout differs by arm, the units remaining are a treatment-affected selection. Controlling for baseline characteristics of the survivors does not undo that selection, because the selection operated on things the baseline does not capture.

Controlling for a mediator to 'isolate the direct effect'. Conditioning on a post-treatment variable does not generally identify a direct effect. It removes one pathway and can simultaneously induce association through unobserved common causes.

Choosing the covariates that most raise significance. A specification selected by the estimate it produces has a reported uncertainty that no longer describes the procedure actually followed.

Reading a large shift in the estimate as a correction. In a properly randomized trial, adjustment should barely move the estimate. A large movement is information about the draw, to be reported rather than absorbed.

Reporting only the adjusted figure. Suppressing the unadjusted difference hides how much of the finding is the modelling, which is exactly what a reader most needs to judge.

Why the confusion is natural. The command is the same in every statistical package. Nothing in the syntax records which of these two situations you are in, and the output looks identical.

The consequential asymmetry. In an experiment, getting the covariate set wrong costs precision. In an observational study, getting it wrong costs correctness. A single habit applied to both makes the difference invisible, and it is the difference between a suboptimal analysis and a wrong answer.

Where the misconception bites hardest. A trial whose randomization broke down is not converted into an observational study that happens to be well adjusted. It becomes a study relying on an assumption nobody planned for, about covariates nobody chose with identification in mind. The correct report says the design failed and states what is now being assumed.

1: fully structured. A randomized experiment reports an unadjusted difference of 2.4 ( $SE=1.1$) and, adjusting for two pretreatment covariates, 2.5 ( $SE=0.6$).

(a) Why did the estimate barely move? (b) Why did the standard error fall? (c) Which should be reported?

Check: (a) randomization made the arms comparable on those covariates in expectation, so removing their influence does not shift the comparison; (b) the covariates explain outcome variation that was previously in the residual, so less noise surrounds the same quantity; (c) both. The unadjusted difference shows the result does not depend on the modelling, and the adjusted one is the more precise estimate of the same effect.

2: partly structured. A trial of a mentoring programme adjusts for baseline grade, school, and number of mentoring sessions attended.

(a) Which covariate is inadmissible and why? (b) What does including it do to the estimand? (c) What if the team argues that attendance measures the dose received?

Check: (a) sessions attended is measured after assignment and is plausibly affected by the programme; (b) conditioning on it removes the pathway running through attendance, so the coefficient no longer estimates the effect of being offered the programme; (c) dose is a real question, but it is a different one. It requires an analysis that treats attendance as an outcome or uses a design that varies dose, not a control variable in the main specification.

3: unstructured. A team reports: "Randomization was compromised for the first three weeks, when site staff were allocating participants manually. We addressed this by controlling for site, week of enrolment, age, sex and baseline severity, and the adjusted treatment effect remains significant."

Assess the claim, and say what the report should state instead.

Check: for the affected period, assignment was not made by a known mechanism, so the causal reading is no longer supplied by the design; the covariates are now carrying identification, which makes this an observational analysis resting on unconfoundedness given those five variables. An assumption nobody planned for and which the staff's actual criteria may well violate, since they may have steered participants on judgements never recorded. Significance of the adjusted estimate is not evidence the problem was fixed. The report should state the breakdown plainly, present the analysis restricted to the properly randomized period as the primary result, present the full-sample adjusted analysis as a secondary observational estimate with its assumption stated, and include a sensitivity analysis for the manually allocated portion.

The identity. Regressing the outcome on an intercept and a treatment indicator gives $\hat{\tau }={\overline{Y}}_1-{\overline{Y}}_0$ exactly. The regression restates the difference in means and adds nothing.

Where identification lives. In the assignment mechanism, in every specification. Covariates never supply it in an experiment.

What adjustment supplies. Precision, by absorbing outcome variation unrelated to treatment.

Additive vs interacted. $Y_i=\alpha +\tau W_i+X_i^{\mathrm{\top }}\beta +{\epsilon }_i$ holds the covariate-outcome relationship fixed across arms; adding $W_i{X}_i^{\mathrm{\top }}\gamma$ lets it differ, and centring the covariates makes $\tau$ interpretable at the sample mean.

Admissibility. Pretreatment covariates only. Post-treatment controls change the estimand and can introduce bias.

Specify in advance. Choosing covariates by their effect on $\hat{\tau }$ invalidates the reported uncertainty.

Match the design. Blocking, clustering and unequal assignment probabilities constrain the regression and its standard errors; robust errors are usual for individual randomization.

Report both. The unadjusted difference beside the adjusted estimate, so the modelling's contribution is visible.

The recurring error. Believing covariates can repair a design that failed.