Practice: Fisher Randomization Inference

(a) $H_0:Y_i(1)=Y_i(0)$ for every unit, treatment changed nobody's outcome, so each observed value is fixed across allocations. ${\overline{Y}}_1=(9+7)/2=8$, ${\overline{Y}}_0=(6+4+2)/3=4$, so $T^{\mathrm{obs}}=4$.

(b) Complete randomization with $N=5$, $N_1=2$ permits $(\genfrac{}{}{0ex}{}{5}{2})=10$ allocations, each with probability $1/10$.

(c) Holding the five outcomes fixed and relabelling: treating $\{A,B\}$ gives $T=8-4=4$; $\{A,C\}$ gives $7.5-4.33\approx 3.17$; $\{D,E\}$ gives $3-7.33\approx -4.33$. The most extreme is $\{D,E\}$ at $|T|\approx 4.33$, then $\{A,B\}$ at 4. Only these two satisfy $|T|\ge 4$, so $p=2/10=0.20$.

(d) The sharp null is not rejected at conventional levels: these data are consistent with treatment having done nothing to anyone. The smallest attainable two-sided p-value is $2/10=0.20$, so this design could never have produced a significant result. The finding reflects the design's resolution, not the treatment.

(d) It does not. The randomization test nulls $Y_i(1)=Y_i(0)$ for every unit, that treatment did nothing to anyone. Zero average effect is a weaker claim, and the two come apart: if treatment raised A and B by 3 and lowered D and E by 3, the average effect is zero while the sharp null is false for all four. Failing to reject the sharp null leaves the data consistent with no effect for anyone; it says nothing directly about the magnitude of an average, which is Neyman's question.

(i) $(\genfrac{}{}{0ex}{}{16}{8})=\mathrm{12,870}$. Using all splits is correct here. This is the design that permits them.

(ii) $2^8=256$: one independent choice per pair. Using all 12,870 splits admits allocations the design forbids, such as both members of a pair treated. Pairing removes between-pair variation, so the true randomization distribution is tighter; the inflated reference set makes the observed statistic look less extreme and the p-value too large.

(iii) All $2^{16}=\mathrm{65,536}$ vectors, since the treated count is random. Restricting to the 12,870 with exactly 8 treated conditions on something the design left free, and the resulting p-value refers to a different experiment.

(iv) ${(\genfrac{}{}{0ex}{}{8}{4})}^2={70}^2=\mathrm{4,900}$: an independent choice within each stratum. Using all splits again includes impossible allocations, for example 8 treated all drawn from one stratum, and inflates the reference distribution for the same reason as in (ii).

The common principle: the reference set is whatever the mechanism could have produced. Enumerating more than that answers a question about an experiment nobody ran.

Sharp null against average effect. In all four. Rejecting the sharp null says treatment changed something for somebody, in whichever design produced the reference set. It does not establish a nonzero average effect in any of them: effects that cancel across units leave $\tau =0$ while the sharp null is false. The design fixes which allocations are permitted, not which hypothesis the test addresses. An average effect needs Neyman's estimate and interval.

First error: the wrong null. The randomization test nulls $Y_i(1)=Y_i(0)$ for every unit, not a zero average effect. These are different hypotheses: a treatment helping half the units and harming the other half equally has a zero average effect while the sharp null is false. A large p-value bears on the sharp null.

Second error: accepting a null. Failing to reject means the data are consistent with no effect for anyone; it is not evidence that no effect exists. With 200 units the design has resolution, so this is less severe than in a tiny experiment, but the inference is still 'not rejected', not 'true'.

Third error: treating the two methods as interchangeable. Both draw randomness from the assignment mechanism, and there the similarity ends. Fisher returns a p-value about a per-unit null; Neyman returns an estimate, a standard error and an interval for the average effect. Neither determines the other, and the interval carries information the p-value does not, in particular the magnitude and precision of the estimated average.

What the report should contain. The randomization p-value with its null stated as a per-unit hypothesis, the estimated average effect with a conservative standard error and interval, and, if a null result is to be interpreted, some indication of what size of effect the design could have detected.

Is an exact test available? Only if there is a mechanism with known probabilities. Hashing a user id is deterministic: given the id, the assignment is fixed, and it could not have come out otherwise. There is no set of allocations the design might have produced, so there is no reference distribution and no exact randomization test. The procedure is closer to assignment by an arbitrary but fixed characteristic than to randomization.

What the proposed reference set would be. Permuting labels freely assumes every split of 5,000 users into the observed group sizes was possible. Under the hash rule none of them was possible except the one observed. The resulting p-value would describe a randomized experiment that was never run.

Is it salvageable? Sometimes. If the hash includes a seed chosen at random per experiment, then re-drawing the seed is a genuine mechanism, and the reference set is the allocations generated by the seeds that could have been drawn, not free permutations. That is a real randomization test, but it must enumerate seeds rather than labels.

Recommendation. Either introduce an explicit random component to the assignment and enumerate over it, or drop the claim of exactness and analyse the comparison as observational, stating the assumption that hash parity is unrelated to the outcome. That assumption is plausible and is still an assumption, which is the distinction the team's proposal obscures.

Which question they are asking. It changes the test entirely. An exact randomization test addresses the sharp null, did B change anything for anyone, and answers with a p-value. A claim about the average is Neyman's question, answered with an estimate, a standard error and an interval. The two can disagree: gains for some users offset by losses for others leave the average near zero while the sharp null is plainly false. Since the hashing gives them no mechanism, neither test is available until assignment is actually randomized.