Practice: Randomized Assignment

(a) Complete randomization with $N=40$, $N_1=20$: each of the $(\genfrac{}{}{0ex}{}{40}{20})$ allocations is equally likely. The rule consults a random draw, not the schools, so it is independent of the potential outcomes. The difference in means estimates the finite-sample average treatment effect over these forty schools.

(b) Bernoulli assignment with $p=0.4$. The treated count is random with mean 16. It is likewise independent of the potential outcomes, so the difference in means again estimates an average effect, though its randomization distribution differs from (a), so a standard error computed for one design does not transfer to the other.

(c) Not an assignment mechanism. Speed of returning a form is a property of the school, its administrative capacity, its enthusiasm, perhaps its existing resources, and those plausibly bear on the outcome. The probability of each vector is unknown. The difference in means estimates the contrast between prompt and slow responders, which is a description rather than a causal effect.

What each supports. Where a known chance mechanism assigned treatment independently of the potential outcomes, the difference in means is unbiased for ${\tau }_S$, the average effect over these units, unbiased in the sense that it averages to ${\tau }_S$ across the assignments the design could have produced, not that any single run returns it. Where no such mechanism exists, the difference in means estimates a contrast between whoever ended up in each group, and names no causal quantity.

(i) and (ii) permit different vector sets: (i) allows only the $(\genfrac{}{}{0ex}{}{20}{10})=\mathrm{184,756}$ vectors with exactly ten treated, while (ii) allows all $2^{20}$. Both are valid mechanisms independent of the potential outcomes, but a standard error derived for (i) does not apply to (ii), whose treated count varies.

(iii) permits a strict subset of (i)'s vectors, those meeting the balance criterion, with the excluded ones having probability zero. It is still independent of the outcomes, so it is a legitimate design, but it is a different mechanism. Inference computed as though all $(\genfrac{}{}{0ex}{}{20}{10})$ allocations were possible is wrong: the reference distribution is the restricted one.

(iv) permits $2^{10}=1024$ vectors, one choice per pair. Exactly ten units are treated, so its vectors are a subset of (i)'s, but the probabilities differ. An allocation treating both members of a pair is impossible here and possible under (i). A standard error from (i) ignores the pairing and is generally too large.

Only designs sharing a mechanism share a randomization distribution. Three of these four fix ten treated units, which is not sufficient for their inference to coincide.

What each estimator claims. Under complete randomization and under Bernoulli assignment, the difference in means is unbiased for the average effect over the twenty units, averaging across the allocations the design permits. Under the re-draw-until-balanced rule it is not: the allocations actually reachable are a subset chosen by looking at covariates, and the estimator averages over a different set than the one its unbiasedness argument assumes.

The first error: imbalance is read as failure. Complete randomization does not produce balanced groups; it produces an allocation drawn uniformly from those the design permits, and some of those are unbalanced on any given covariate. With enough baseline variables, some imbalance is near-certain. A 22% against 13% gap in 30-unit arms is roughly three patients, which is unremarkable. Nothing about it indicates the mechanism misfired.

The second error: the mechanism was silently changed. Drawing repeatedly and keeping the most balanced allocation is a different rule, uniform over allocations meeting a balance criterion, not uniform over all $(\genfrac{}{}{0ex}{}{60}{30})$. That rule is still independent of the potential outcomes, so the design is not invalid. But every subsequent standard error, confidence interval and randomization test refers to the distribution of assignments the mechanism allows, and analysing it 'as a standard completely randomized trial' uses the wrong reference distribution. The reported uncertainty will not be the uncertainty the design actually carries.

What should have happened. If diabetes, age and sex were expected to matter, they should have been blocked on before assignment. Blocking is a deliberate, pre-specified restriction of the mechanism, and its analysis is known.

What must now be reported. The actual procedure, including the rejection criterion and the number of draws, so that inference can be computed under the mechanism used, for instance by a randomization test that redraws subject to the same balance restriction.

What the original randomization supported. Before the re-run, the difference in means was unbiased for the average effect over these 60 patients, because the mechanism was known and independent of the potential outcomes. That property belongs to the procedure across its possible assignments; the 22%-against-13% split is one draw from it and does not revoke it. Re-drawing until the covariates look balanced replaces the known mechanism with one that consults the data, and the claim goes with it.

The mechanism. Allocation of 300 places among 900 applicants by public lottery is complete randomization with $N=900$ and $N_1=300$: each allocation of places is equally likely and the draw consults no property of the applicants. That it was run for fairness rather than for research does not weaken it: the requirement is that the rule is known and independent of the potential outcomes, and both hold.

What it licenses. A comparison of lottery winners with losers estimates the average effect of winning the lottery among the 900 applicants. That is a real causal estimand, and the design supports it without any assumption about how applicants differ.

First threat: winning is not the same as receiving. Some winners will decline the place, move away, or fail eligibility checks; some losers may obtain housing by another route. Comparing winners with losers still estimates the effect of winning, which is what randomization protects. Comparing those who actually moved with those who did not abandons the randomization, because taking up the offer is a choice made by the applicant.

Second threat: differential loss to follow-up. Health five years later is only observed for people the researcher can trace. If winners are easier to trace because they have a stable address, the observed groups are no longer the randomized groups, and the comparison is between those who could be found. A selection the lottery does not govern.

What remains true. The lottery licenses a claim about the effect of winning, on the applicants who entered, provided outcomes are observed for both arms alike. It licenses nothing about the effect of housing on people who never applied.

Which claim the lottery supports. The lottery is a known chance mechanism applied to the 900 applicants, so among them the comparison is unbiased for the average effect of receiving a place, averaging over the draws the lottery could have produced. It supports nothing about people who never applied: they were never in the mechanism. The estimand is the average effect for applicants, and the unbiasedness is a property of the allocation procedure, not of the single draw that occurred.