Hypothesis Tests for Experimental Research

Every test in this family is one template: an estimate, a null value, and a standard error, assembled into a standardized distance and referred to a distribution that says how unusual such a distance would be if the null were true. Choosing the right test is choosing the right standard error and reference distribution for the design that produced the data, and what a large p-value licenses is far less than it is usually asked to carry.

Definition

Every test in this family computes $statistic = \frac{estimate - null value}{standard error}$ and refers it to a distribution derived under $H_{0}$ . One mean: $Z = \frac{\bar{X} - μ_{0}}{σ / \sqrt{n}}$ with $σ$ known, or $T = \frac{\bar{X} - μ_{0}}{s / \sqrt{n}} \sim t_{n - 1}$ under a normal population. One proportion: $Z = \frac{\hat{p} - p_{0}}{\sqrt{p_{0} (1 - p_{0}) / n}}$ , the null standard error using $p_{0}$ . Two independent means (Welch): $T = \frac{({\bar{X}}_{1} - {\bar{X}}_{2}) - Δ_{0}}{\sqrt{s_{1}^{2} / n_{1} + s_{2}^{2} / n_{2}}}$ , degrees of freedom by Welch–Satterthwaite. Pooled variant: $s_{p}^{2} = \frac{(n_{1} - 1) s_{1}^{2} + (n_{2} - 1) s_{2}^{2}}{n_{1} + n_{2} - 2}$ with $d f = n_{1} + n_{2} - 2$ . Paired: $D_{i} = X_{i} - Y_{i}$ , then $T = \frac{\bar{D} - μ_{D, 0}}{s_{D} / \sqrt{n}} \sim t_{n - 1}$ . Two proportions: with $\hat{p} = \frac{x_{1} + x_{2}}{n_{1} + n_{2}}$ , $Z = \frac{{\hat{p}}_{1} - {\hat{p}}_{2}}{\sqrt{\hat{p} (1 - \hat{p}) (1 / n_{1} + 1 / n_{2})}}$ .

Formal statement

$statistic = \frac{estimate - null value}{S E}$ ; $T = \frac{\bar{X} - μ_{0}}{s / \sqrt{n}} \sim t_{n - 1}$ ; Welch $T = \frac{({\bar{X}}_{1} - {\bar{X}}_{2}) - Δ_{0}}{\sqrt{s_{1}^{2} / n_{1} + s_{2}^{2} / n_{2}}}$ ; paired $T = \frac{\bar{D} - μ_{D, 0}}{s_{D} / \sqrt{n}}$ .

Assumptions and scope

Failing to reject is not evidence that the null is true. It is consistent with a real effect the study lacked the resolution to detect, so a null result should be reported with the effect sizes the design could have found.
Welch's test is the safer default for two independent groups. Pooling requires a substantive reason to believe the variances are equal.
Do not pool merely because a preliminary test of equal variances failed to reject. Selecting the main test using the same data distorts the reported error rate.
The paired test applies when the design paired the observations. Applying it to independent groups, or a two-sample test to paired data, uses a standard error the design does not support.
For one proportion the null standard error uses $p_{0}$ , not $\hat{p}$ ; the Wald interval uses $\hat{p}$ . The two differ deliberately, and the test's version is the one consistent with computing under $H_{0}$ .
The $t$ reference assumes approximate normality of the sampling distribution of the mean, which the central limit theorem supplies at moderate $n$ for most populations but not for very small samples from skewed ones.
These tests draw their reference distribution from a sampling model. A randomization test derives it from the assignment mechanism instead, and the two answer different questions even when the arithmetic coincides.

Worked material

Example

The same numbers, two designs

Twenty people are measured under condition A and condition B. Mean A is 52.0, mean B is 48.4, so the observed gap is 3.6.

If the design used two independent groups of 20 people each, with $s_{1} = 9.0$ and $s_{2} = 8.5$ :

S E = \sqrt{\frac{81}{20} + \frac{72.25}{20}} = \sqrt{4.05 + 3.61} = \sqrt{7.66} \approx 2.77,

T = \frac{3.6}{2.77} \approx 1.30,

which is unremarkable against a $t$ distribution with roughly 38 degrees of freedom.

If the design measured the same 20 people under both conditions, the analysis works on the 20 within-person differences. Suppose $s_{D} = 4.2$ :

\begin{aligned} S E & = \frac{4.2}{\sqrt{20}} \approx 0.94, \\ T & = \frac{3.6}{0.94} \approx 3.83 on 19 degrees of freedom, \end{aligned}

which is decisive.

Same gap, opposite conclusions. Nothing about the estimate changed. What changed is the standard error, because the designs differ in what varies. Between people, scores differ a lot; within a person across two conditions, much less. The paired design removes the between-person variation, and the test must be the one that reflects it.

You cannot pick the test from the numbers alone. The design tells you which standard error is the right one, and getting that wrong changes the answer rather than refining it.

Non-example

Tests applied to the wrong structure

A two-sample test on paired data. Comparing all before-measurements with all after-measurements ignores that each pair shares a unit. The standard error comes from the wrong variation, usually overstating it and obscuring a real effect.

A paired test on independent groups. Pairing observations that were not paired by the design, arbitrarily matching the first treated unit with the first control, invents a structure the data do not have.

Pooling because a variance test did not reject. Running a preliminary test of equal variances and pooling when it fails to reject uses the same data twice and distorts the error rate of the main test. Welch's test needs no such decision.

Using $\hat{p}$ in the null standard error. For $H_{0} : p = p_{0}$ , the standard error should be computed under the null, using $p_{0}$ . Substituting $\hat{p}$ answers a slightly different question and is the interval's formula, not the test's.

Concluding equivalence from a large p-value. "Not significant" is not "no difference". Showing two things are similar requires an equivalence test with a pre-specified margin, which is a different procedure.

Treating a randomization test as the same thing. The arithmetic may match a $t$ -test, but the reference distribution comes from the assignment mechanism rather than a sampling model, and the null is a per-unit statement rather than one about means.

Contrast

What rejecting and not rejecting establish

	Small p-value	Large p-value
What it says	The data would be surprising if $H_{0}$ were true	The data would not be surprising if $H_{0}$ were true
Licenses	Rejecting $H_{0}$ at the stated level	Nothing about $H_{0}$ being true
Compatible with	A real effect	A real effect too small for this study to detect
Correct wording	"We reject the null"	"We fail to reject the null"
Needed to claim no effect	—	An equivalence test with a stated margin

Why the asymmetry is built in. A test asks how compatible the data are with one hypothesis. Data compatible with "no effect" are also compatible with "a small effect", and with "a large effect the study was too small to see". Compatibility does not single out the null.

The practical consequence. A null result should be reported with what the design could have detected. "We found no significant difference" is nearly uninformative on its own; "no significant difference, and the study could have detected a difference of 5 points with 80% power" tells a reader what was actually learned.

The wording matters. "Accept the null" invites the error. "Fail to reject" is clumsier and describes what happened.

Common errors

Common misconception

A p-value above the threshold shows the null hypothesis is true, so a non-significant result establishes that there is no effect or no difference.

Common misconception

Before comparing two independent means, test whether the variances are equal; if that test does not reject, pool the variances and use the pooled t-test, because the data have shown the assumption is satisfied.

Related units

Requires

Sampling Distributions and Standard Error

Connected

Fisher Randomization Inference (contrasts with)
ANOVA for Experimental Research (suggested next)

Learn this topic

Hypothesis Tests for Experimental Research

Used in

Inferential Foundations for Experimental Research

Sources

All of Statistics: A Concise Course in Statistical Inference (2004)

Hypothesis Tests for Experimental Research

Assumptions and scope

Worked material

The same numbers, two designs

Tests applied to the wrong structure

What rejecting and not rejecting establish

Common errors

Related units

Requires

Connected

Learn this topic

Used in

Sources

Settings

Focus timer

Today

Last seven days

Where the time went

Recent sessions

Settings

Notifications

Your history