Practice: Hypothesis Tests for Experimental Research

The test. Two independent groups of different patients, so a two-sample test. Welch's version is the default, since nothing here justifies assuming equal variances, and the sample standard deviations do differ.

The standard error.

The statistic.

Reference distribution. A $t$ distribution with degrees of freedom from the Welch–Satterthwaite approximation, here roughly 95.

Conclusion. The two-sided critical value at $\alpha =0.05$ is about 1.99, and $|-2.53|$ exceeds it, so the null of no difference is rejected; $p\approx 0.013$. The new process is associated with a mean waiting time about 4.5 minutes shorter.

What to add. An interval for the difference, roughly $-4.5\pm 1.99\times 1.776=[-8.0,-1.0]$ minutes, which reports the magnitude the data support rather than only that a threshold was crossed. Because patients were randomized, the comparison supports a causal reading.

On pre-testing the variances. Do not. Welch's test makes no equal-variance assumption and is the safer default, and here $s=8.1$ against $9.6$ gives no reason to pool anyway. The deeper objection is procedural: a preliminary variance test uses the same data to choose the test and then to run it, so the reported error rate no longer describes what was done. Pool only on substantive grounds known before the data, never on the strength of a pre-test.

(a) The statistic. The standard error of the difference, without pooling:

Referred to a $t$ distribution with Welch's degrees of freedom:

So $t\approx 1.53$ on about $25$ degrees of freedom.

(b) No pooling. The sample standard deviations differ by nearly a factor of two and the groups are unequal in size. The case in which pooling is least safe, because the larger group's smaller variance dominates the pooled estimate and understates the standard error. Welch's test is the default regardless: it costs almost nothing when variances are equal and protects the error rate when they are not.

Note what was NOT done: a preliminary $F$ test of equal variances, used to decide whether to pool. That two-stage procedure distorts the error rate of the test that follows, because the second test is chosen using the same data it is applied to.

(c) What the result establishes. , so the data do not distinguish the hypothesis of no difference from the alternatives nearby. It does not establish that the groups are the same. The observed difference is $4.5$ with a standard error of $2.9$, so the compatible range runs from roughly $-1.6$ to $10.6$, comfortably including zero, and also including differences large enough to matter. The correct report is that this study was not big enough to settle the question. Supporting a claim of no difference would require an equivalence test against a margin stated before the data were seen.

The error. Failure to reject is being converted into a positive claim of equivalence. A p-value of 0.31 says the data would not be surprising if the variants performed identically. It does not say they do.

What the result is also compatible with. A real difference the study was too small to detect. Data consistent with 'no difference' are equally consistent with 'B is 0.5 points better' and with 'B is 0.4 points worse', if the sample cannot separate those. The test asks about compatibility with one hypothesis, and compatibility does not single out the null.

Why this matters commercially. If a 0.5-percentage-point gain would change the decision, and the data cannot rule it in or out, then the study has not answered the question the recommendation depends on.

What should be reported. The estimated difference of 0.3 percentage points with a confidence interval, which shows the range of differences the data are compatible with. If that interval runs from about $-0.3$ to $+0.9$ points, it should be stated plainly that a commercially meaningful difference remains possible in either direction.

If equivalence is the actual claim. It requires an equivalence test against a pre-specified margin of practical indifference. A different procedure that must be planned in advance, not inferred from a non-significant result.

A further note on the standard error. Nothing in the report says how the two proportions' variability was combined. With unequal group sizes or unequal variances, a pooled standard error can understate uncertainty; the default should be the unpooled form, chosen in advance rather than after inspecting the variances.

The structure. Each depot is measured under both algorithms, so these are paired measurements with the depot as the unit. This is a crossover design, and the vocabulary is the only thing unfamiliar about it.

Why the proposed test is wrong. A two-sample test computes its standard error from the spread across all 36 depot-fortnights, which includes the large differences between depots, size, geography, traffic, staffing. Those differences are identical for a depot's two fortnights and cancel completely in a within-depot comparison. Including them inflates the standard error and will usually understate what the study established.

The correct analysis. Form $D_i={\overline{Y}}_{i,X}-{\overline{Y}}_{i,Y}$ for each of the 18 depots, then a one-sample $t$-test on those differences: $T=\overline{D}/(s_D/\sqrt{18})$ with $df=17$, not 34. The degrees of freedom follow from 18 independent units, not 36 measurements.

The design concern no test can fix. Every depot ran X first and Y second, so the algorithm is completely confounded with the time period. Any seasonal change in demand, any learning by depot staff, any unrelated operational change between the fortnights enters the estimate as though it were an algorithm effect. The remedy is design, not analysis: randomize the order so that half the depots receive Y first, which turns period into something that can be estimated and separated rather than something inseparable from the treatment.

Why the variance question does not arise here. Once the analysis is paired, each depot contributes one difference and there is a single sample of 18 differences with one variance. The Welch-versus-pooled question belongs to two independent samples; pairing removes it entirely, which is another reason the unit of analysis has to be settled before any variance comparison is contemplated.

(a) Paired. The unit of analysis is the PATIENT, and each supplies one difference. A paired $t$ test on the sixty differences, with $59$ degrees of freedom. Treating the two sets of sixty readings as independent samples would ignore that each pair shares a person, and would usually overstate the standard error.

(b) Two independent samples. The unit is the patient, and each patient appears in one arm only. Welch's two-sample $t$ test. The superficial resemblance to (a), sixty patients, two sets of readings, is exactly what misleads: what differs is whether a reading in one group is tied to a particular reading in the other.

(c) Clustered. The treatment was assigned to CLINICS, so the unit of analysis is the clinic and the effective sample size is twelve, not four hundred and eighty. A $t$ test on the twelve clinic means, or a model with cluster-robust standard errors at the clinic level. Running a two-sample test on 480 patients would treat pupils within a clinic as independent evidence about a treatment that never varied within a clinic.

(d) One sample against a fixed value. The unit is the patient; the comparison value is a constant, not an estimate. A one-sample $t$ test with $59$ degrees of freedom.

The through-line. In every case the question is the same: what could have come out differently under the design? That is the unit of analysis, and it fixes the standard error before any arithmetic begins.