Practice: Blocked and Paired Randomized Experiments

Size-weighted. $\hat{\tau }=\frac{300}{400}(5)+\frac{100}{400}(13)=3.75+3.25=7.0$.

Equally weighted. $(5+13)/2=9.0$.

What each targets. The size-weighted figure estimates the average effect over the 400 units actually in the trial: each unit counts once, so the larger region dominates in proportion to its size. The equally weighted figure estimates the average of the two regional effects, giving a region of 100 the same influence as a region of 300.

Which to report. If the question is what the intervention did for the people studied, 7.0 is the answer. The 2.0 gap is a difference of estimand rather than of precision, so reporting 9.0 while describing the average effect on the trial's units misstates what was estimated. Equal weighting is defensible only when the regions are themselves the units of interest, for instance if each region is a site whose policy is decided separately.

Was blocking worth it here? It depends on how strongly region predicts the outcome. Blocking removes between-region variation from the comparison, so it reduces variance in proportion to how much of the outcome region explains. If the two regional effects had been close and region weakly prognostic, the design would have committed the analysis to a stratified estimator for little gain.

The error. The claim reverses the relationship between design and analysis. Pairing is not a feature of data collection that can be set aside afterwards; it is a restriction on which allocations were possible, and that restriction governs every later calculation.

What the design permitted. One choice per pair, so $2^{20}$ allocations, about a million, not $(\genfrac{}{}{0ex}{}{40}{20})$, which is roughly 138 billion. The two-sample test refers to a reference distribution over allocations the experiment could never have produced, including every allocation treating both members of some pair.

What it costs in precision. The two-sample standard error is computed from the spread of raw outcomes across all 40 units. That spread contains the between-pair variation the matching was designed to remove: whatever the two members of a pair shared cancels exactly in their difference but not in a pooled comparison. If the pairs were well matched, the raw outcomes vary considerably more than the differences do, so the standard error is inflated and the result looks weaker than the design earned.

The correct analysis. Work on the 20 differences $D_j=Y_{j,T}-Y_{j,C}$: $\hat{\tau }=\overline{D}$, $SE=s_D/\sqrt{20}$, compared against $t$ on 19 degrees of freedom, or a randomization test enumerating the $2^{20}$ sign flips.

Why the design cannot be set aside. Pairing was chosen because the pairing variable was expected to predict the outcome; that expectation is what the precision gain rests on. Having accepted the constraint, the analysis is committed to it, which is exactly what the two-sample test ignores. A design is not advice that can be declined at the analysis stage.

The design. A matched-pair experiment with the field as the block: 8 fields, one coin per field, $J=8$ pairs. The vocabulary is agricultural and the structure is the one this unit covers.

The analysis it requires. Work on the 8 within-field differences $D_j$, giving $\hat{\tau }=\overline{D}$ and $SE(\hat{\tau })=s_D/\sqrt{8}$, against $t$ on 7 degrees of freedom, or a randomization test over the $2^8=256$ allocations the design permitted.

Why the field is the right block. Soil composition, drainage, slope and microclimate are shared by the two halves of a field and differ substantially between fields. Those shared characteristics cancel exactly in a within-field difference. The pooled spread across 16 half-fields contains all of that between-field variation, which is precisely what the design arranged to remove.

What the reported standard error describes. An experiment in which 16 half-fields were independently assigned to fertiliser, which is not what happened, since assigning one half of a field determined the other. It will typically overstate the uncertainty, understating what the trial established, and its reference distribution belongs to an experiment that was never run.

Why halves of a field. Soil and drainage vary far more between fields than within one, so the field is strongly prognostic and splitting it removes that variation by construction. That is the case where blocking earns its constraint, and having earned it, the analysis owes the paired estimator rather than a pooled comparison.