ANOVA for Experimental Research

One-way analysis of variance asks whether several group means can be treated as equal, by comparing the variation between groups with the variation within them. The decomposition is exact and the test is a ratio of mean squares. What rejection establishes is narrow, that the means are not all equal, and answering which ones differ is a separate question requiring planned contrasts or multiplicity-adjusted comparisons.

Definition

Let $Y_{i j}$ be observation $j$ in group $i$ , with group mean ${\bar{Y}}_{i}$ , grand mean $\bar{Y}$ , group size $n_{i}$ , $k$ groups and $N = \sum_{i} n_{i}$ . The sums of squares are

\begin{aligned} S S T & = \sum_{i} \sum_{j} (Y_{i j} - \bar{Y})^{2}, \\ S S B & = \sum_{i} n_{i} ({\bar{Y}}_{i} - \bar{Y})^{2}, \\ S S E & = \sum_{i} \sum_{j} (Y_{i j} - {\bar{Y}}_{i})^{2}, \end{aligned}

and they satisfy the exact decomposition $S S T = S S B + S S E$ . The mean squares divide each component by its degrees of freedom, $M S B = \frac{S S B}{k - 1}$ and $M S E = \frac{S S E}{N - k}$ , and the omnibus statistic is

F = \frac{M S B}{M S E} .

Under $H_{0} : μ_{1} = μ_{2} = \dots = μ_{k}$ and the independent, normal, equal-variance model, $F \sim F_{k - 1, N - k}$ . For two groups the equal-variance one-way ANOVA is equivalent to the pooled two-sample test, with $F = t^{2}$ .

Formal statement

$S S T = S S B + S S E$ ; $M S B = \frac{S S B}{k - 1}$ ; $M S E = \frac{S S E}{N - k}$ ; $F = \frac{M S B}{M S E} \sim F_{k - 1, N - k}$ under $H_{0}$ ; for $k = 2$ , $F = t^{2}$ .

Assumptions and scope

Rejecting the omnibus null establishes that at least one mean differs. It does not identify which, and reading it as evidence about a particular pair is unsupported.
Identifying specific differences requires planned contrasts specified in advance, or post-hoc comparisons with a multiplicity adjustment.
The classical $F$ reference assumes independent observations, normality within groups and equal variances across them. Unequal variances, particularly with unequal group sizes, distort the test.
The decomposition $S S T = S S B + S S E$ is algebraic and holds regardless of the model; it is the reference distribution that depends on the assumptions.
For two groups this is the pooled two-sample test, not an alternative to it: $F = t^{2}$ exactly. It offers no protection that the $t$ -test lacks.
A significant $F$ says nothing about the size of any difference. Group means with intervals report what the test omits.
ANOVA can be written as a regression on indicator variables, which is the route to factorial designs and covariate adjustment.

Worked material

Example

A significant F, and what it leaves open

Four teaching methods are compared, 25 students each, $N = 100$ . The ANOVA gives $F = 4.62$ on $3$ and $96$ degrees of freedom, $p = 0.005$ .

Group means: A 68.2, B 71.4, C 74.9, D 69.1.

What the test establishes. The four means are not all equal. That is the entire content of $p = 0.005$ .

What it does not establish. That C beats A, which is the comparison the eye goes to. The omnibus test never examined that pair; it asked a single question about all four simultaneously.

Why the distinction has teeth. With four groups there are six pairwise comparisons. Testing each at the 5% level gives roughly a 26% chance of at least one spurious finding when all means are equal. The omnibus test protects against that inflation precisely by not making the individual comparisons, and reading a particular difference out of it forfeits the protection while keeping the appearance of it.

What to do instead. If C-versus-A was specified before seeing the data, test it as a planned contrast. If the interest arose from looking at the means, use a post-hoc procedure with a multiplicity adjustment. Either way, report the estimated difference with an interval, since $F$ says nothing about magnitude.

What is genuinely licensed now. That the four methods do not all produce the same mean score ( $F_{3, 96} = 4.62$ , $p = 0.005$ ), with pairwise comparisons still to follow.

Non-example

Conclusions an omnibus test does not support

"The ANOVA was significant, so group C differs from group A." The test examined all groups at once. No pairwise comparison was performed.

"The ANOVA was not significant, so the groups are equivalent." Failing to reject is inconclusive here as everywhere; the study may simply lack resolution.

Picking the largest gap after seeing the data and testing it unadjusted. The comparison was selected because it was largest, so its nominal p-value understates how often such a gap arises by chance.

Using ANOVA to protect a subsequent unadjusted comparison. A significant omnibus result does not license unadjusted pairwise tests afterwards. The protection comes from making the adjustment, not from having run $F$ first.

Applying the classical $F$ with badly unequal variances and unequal group sizes. The reference distribution assumes equal variances; when they differ and the groups are unbalanced, the actual error rate departs from the nominal one.

Treating ANOVA as different in kind from a $t$ -test for two groups. For $k = 2$ it is the pooled two-sample test, with $F = t^{2}$ . It offers no additional protection.

Contrast

What the omnibus test answers

	The omnibus $F$ test	A specific comparison
Question	Are all $k$ means equal?	Does group $i$ differ from group $j$ , and by how much?
Null	$μ_{1} = μ_{2} = \dots = μ_{k}$	$μ_{i} - μ_{j} = 0$
Rejection establishes	At least one mean differs	That particular difference
Reports magnitude	No	Yes, with an interval
Multiplicity	Handled by asking one question	Must be adjusted, or specified in advance

Why the confusion is so common. The table of group means sits directly beneath the $F$ statistic, and the eye compares them immediately. The arithmetic invites the reading the test does not support.

What the omnibus test supplies. A single question with a single error rate. Six pairwise tests at 5% carry roughly a 26% chance of at least one false positive when all means are equal; one omnibus test carries 5%.

What it costs. All the specificity. The result is uninformative about direction and magnitude, which is usually what a decision requires.

The legitimate routes to specificity. Planned contrasts, specified before the data are seen, testing the comparisons that motivated the study. Or post-hoc comparisons with a multiplicity adjustment, which pay for the specificity by widening the intervals. Both report estimated differences, which $F$ does not.

Common errors

Common misconception

A significant omnibus F test shows which groups differ, so the largest observed gap between group means can be reported as a established difference.

Related units

Requires

Hypothesis Tests for Experimental Research

Learn this topic

ANOVA for Experimental Research

Used in

Inferential Foundations for Experimental Research

Sources

All of Statistics: A Concise Course in Statistical Inference (2004)

ANOVA for Experimental Research

Assumptions and scope

Worked material

A significant F, and what it leaves open

Conclusions an omnibus test does not support

What the omnibus test answers

Common errors

Related units

Requires

Learn this topic

Used in

Sources

Settings

Focus timer

Today

Last seven days

Where the time went

Recent sessions

Settings

Notifications

Your history