Practice: ANOVA for Experimental Research

Degrees of freedom. Between $=k-1=2$; within $=N-k=30-3=27$; total $=N-1=29$.

Residual sum of squares. $SSE=SST-SSB=984.0-246.0=738.0$, by the exact decomposition.

Mean squares. $MSB=246.0/2=123.0$; $MSE=738.0/27=27.33$.

Statistic. $F=123.0/27.33\approx 4.50$ on 2 and 27 degrees of freedom.

Conclusion. 4.50 exceeds the critical value of 3.35, so the null of equal means is rejected; $p\approx 0.02$. The three formulation means are not all equal.

Checks. The degrees of freedom add: $2+27=29$. And $MSE=27.33$ implies a within-group standard deviation of about 5.2, a plausible scale for plot yields.

What this does not license. Any statement about which formulation is best. That requires planned contrasts, or post-hoc comparisons with a multiplicity adjustment, each reporting an estimated difference with an interval, none of which $F$ supplies.

With two groups. The equal-variance one-way ANOVA is exactly the pooled two-sample $t$ test, and the statistics satisfy $F=t^2$. The $F$ reference distribution with $1$ and $N-2$ degrees of freedom is the square of the $t$ with $N-2$. Nothing is gained by running ANOVA on two groups: it is the same test written differently, and it offers no extra protection because there is only one comparison to make.

The error. The omnibus test established only that the four flows do not all perform alike. It never tested C against A. The decision rests on a comparison the analysis did not make.

Why the selection matters. Flow C was singled out for having the highest observed rate. With four groups there are six pairwise comparisons, and the largest observed gap is the one most inflated by chance. Testing it as though it had been chosen in advance understates how often such a gap arises when the flows are equivalent.

What a significant $F$ does not provide. Protection for subsequent unadjusted comparisons. The protection comes from making the adjustment, not from having run the omnibus test first.

What should be done. Test C against A explicitly, with a multiplicity adjustment across the six possible pairs. Report the estimated difference with an interval rather than two raw percentages: a 7-point gap with an interval of 1 to 13 points supports a different decision from one spanning $-1$ to 15.

Two further questions. Was any comparison planned in advance? A pre-specified contrast needs no post-hoc penalty, and if C-versus-A was the motivating question, that changes the analysis. And is 7 points large enough to justify the switching cost? The test cannot address that at all, which is another reason the interval matters more than the p-value.

Where the protection actually comes from. The omnibus test guards against the multiplicity created by having four flows to compare. For two flows there is no such multiplicity and ANOVA reduces to the pooled two-sample test with $F=t^2$, so the protection is a property of how many comparisons are in play, not of choosing ANOVA. Picking the best of four after a significant omnibus result reintroduces exactly the multiplicity the test was guarding.

First claim — variation by site. The $F$ test shows the six site means are not all equal. It does not show that the protocol's effectiveness varies by site: with no comparison group, site differences in readmission reflect case mix, catchment population, staffing and baseline quality as much as anything about the protocol. Establishing that effectiveness varies would require a treatment effect estimated per site, which needs pre-protocol data or unexposed comparison units.

Second claim — site 4 as the model. Site 4 was selected for having the most extreme observed value among six, which is the comparison most inflated by chance and also the one most exposed to regression to the mean: a site that looks best this period is likely to look less exceptional next period even if nothing about it is special. The omnibus test did not test site 4 against any other site.

Assumption concerns specific to this setting. Hospitals differ greatly in size, so the groups are unbalanced and within-site variances almost certainly differ, exactly the combination under which the classical $F$ reference is least reliable. Readmissions within a hospital are also unlikely to be independent in the way the model assumes.

What I would report. Site-level readmission rates with intervals, so the reader sees which differences are resolved and which are not; a comparison against each site's own pre-protocol baseline rather than against other sites, which is the contrast that bears on the protocol; adjustment for case mix; and, if site 4 is to be investigated, a statement that it is a hypothesis-generating choice rather than a demonstrated exemplar.

A note on scope. Had the system compared two hospitals rather than six, this analysis would be the pooled two-sample test with $F=t^2$, and the same objection would stand: a difference between sites is not an effect of the protocol. The number of groups changes the multiplicity, not what the comparison is capable of establishing.