Confidence Intervals for Experimental Research

An interval estimate is the same three ingredients as a test, rearranged: an estimate, a standard error, and a critical value that sets the coverage. What the confidence statement describes is the procedure's long-run behaviour, not the probability that one computed interval contains the parameter, and keeping that straight is what separates a reportable interval from a misreported one.

Definition

Every interval here has the form $estimate \pm critical value \times standard error$ , where $z_{α / 2}$ and $t_{α / 2, d f}$ denote positive upper-tail critical values. One mean: $\bar{X} \pm z_{α / 2} \frac{σ}{\sqrt{n}}$ with $σ$ known, or $\bar{X} \pm t_{α / 2, n - 1} \frac{s}{\sqrt{n}}$ when it is estimated. Difference in means: $({\bar{X}}_{1} - {\bar{X}}_{2}) \pm t^{*} S E$ , with the Welch standard error $S E = \sqrt{s_{1}^{2} / n_{1} + s_{2}^{2} / n_{2}}$ ; for paired data, $\bar{D} \pm t_{α / 2, n - 1} \frac{s_{D}}{\sqrt{n}}$ . One proportion (Wald): $\hat{p} \pm z_{α / 2} \sqrt{\hat{p} (1 - \hat{p}) / n}$ . Experimental effect: $\hat{τ} \pm z_{α / 2} S E (\hat{τ})$ using the design-based standard error. A $100 (1 - α) %$ confidence procedure is one whose intervals capture the fixed parameter in at least that proportion of repeated samples or assignments when its assumptions hold; where they hold only approximately, the realised coverage is approximate too.

Formal statement

$estimate \pm critical value \times S E$ ; $\bar{X} \pm t_{α / 2, n - 1} \frac{s}{\sqrt{n}}$ ; $({\bar{X}}_{1} - {\bar{X}}_{2}) \pm t^{*} \sqrt{s_{1}^{2} / n_{1} + s_{2}^{2} / n_{2}}$ ; $\hat{τ} \pm z_{α / 2} S E (\hat{τ})$ .

Assumptions and scope

The confidence level describes the procedure across repetitions. Once computed, the endpoints are fixed and the parameter either lies within them or does not; no probability attaches to the particular interval.
A $100 (1 - α) %$ interval and a two-sided level- $α$ test using the same assumptions and standard error agree: a null value outside the interval is rejected.
The Wald interval for a proportion behaves poorly when counts are small or $\hat{p}$ is near 0 or 1: its coverage falls well below the nominal level, and because it is centred on $\hat{p}$ with a symmetric width it can place an endpoint outside $[0, 1]$ . At $\hat{p} = 0$ it returns zero width, asserting certainty from no positive observations. Wilson (score) and Clopper-Pearson (exact) intervals are the usual repairs, and neither leaves $[0, 1]$ .
The standard error must match the design. Paired data use $s_{D} / \sqrt{n}$ ; independent groups use the Welch form; a randomized experiment uses the design-based standard error.
Coverage rests on the sampling distribution being approximately as assumed, which for a mean comes from the central limit theorem and can fail for very small samples from skewed populations.
An interval reports magnitude and precision together, which a p-value does not. Reporting only whether an interval excludes the null discards most of what it was computed to convey.
A stated confidence level is NOMINAL. Coverage is exact only for the procedures whose distributional assumptions hold exactly: the normal-model mean interval with $σ$ known, the $t$ interval under normality, and Clopper-Pearson by construction (which is conservative, covering at least the nominal level). The Welch and large-sample proportion intervals are approximate, their coverage approaching the nominal level as $n$ grows and departing from it in small or skewed samples.

Worked material

Example

What an interval reports that a p-value does not

Two studies test the same intervention against a null of no effect. Both return $p = 0.04$ .

Study A. Estimated effect 12.0 points, 95% interval $[0.5, 23.5]$ .

Study B. Estimated effect 1.2 points, 95% interval $[0.05, 2.35]$ .

Identical p-values, and the two results say entirely different things.

Study A is compatible with an effect anywhere from negligible to very large. It establishes that something is probably happening and leaves the magnitude wide open. Study B pins the effect down tightly, and to a value that may be too small to matter.

What the p-value discarded. It reported only that both intervals exclude zero. Everything about magnitude and precision, which is what a decision needs, is in the interval and absent from the p-value.

Reading a null result the same way. Suppose a third study gives an estimate of 0.4 with interval $[- 4.8, 5.6]$ and $p = 0.88$ . Reporting "no significant effect" suggests the intervention does nothing. The interval says the study is compatible with a harm of nearly 5 points and a benefit of over 5. It did not resolve the question at all. That is a different finding from a tight interval around zero, and only the interval distinguishes them.

Non-example

Statements an interval does not support

"There is a 95% probability the true mean lies between 4.1 and 5.3." After computation, both endpoints are fixed numbers and the parameter is a fixed constant. Nothing random remains for the probability to describe.

"95% of the data fall in this interval." The interval describes a parameter, not the observations. The spread of the data is governed by $s$ , not by $s / \sqrt{n}$ , and at $n = 400$ these differ twentyfold.

"95% of future samples will produce a mean inside this interval." That is a prediction interval for a future statistic, which is a different and wider construction.

"The intervals overlap, so the groups do not differ." Overlapping intervals for two group means do not imply a non-significant difference. The comparison needs an interval for the difference, built from the standard error of the difference.

"The effect is not significant, so there is no effect." A large p-value and an interval straddling zero are compatible with substantial effects in either direction, as the width shows directly.

A Wald interval for a proportion with few events. With 2 events in 40 trials, $\hat{p} \pm z \sqrt{\hat{p} (1 - \hat{p}) / n}$ performs badly and can produce a lower endpoint below zero. Score or exact intervals are appropriate there.

Contrast

What the confidence level describes

	The procedure, before data	The computed interval, after data
What is random	Which sample or assignment occurs	Nothing — the endpoints are fixed numbers
Correct statement	"This procedure covers the parameter 95% of the time"	"Either this interval contains the parameter or it does not"
Probability applies	Yes, to the procedure	No, not to this interval
What 95% refers to	Long-run success rate of the method	—

Why the error is so natural. The interval is right there and the parameter is not, so it feels like a statement about where the parameter probably sits. The frequentist framework simply does not supply that: it assigns probabilities to procedures, never to fixed unknown constants.

A picture that helps. Imagine running the study 100 times, each time computing an interval. Roughly 95 of those intervals would cover the parameter and 5 would miss. You have one of the hundred and cannot know which kind it is. The 95% describes the collection, not your draw.

What you may legitimately say. That the interval contains the values not rejected by the corresponding two-sided test. The values compatible with your data at that level. This is the duality, and it is often the most useful reading in practice.

If you want a probability about the parameter, you need a Bayesian credible interval, which requires a prior and answers a different question.

Common errors

Common misconception

A 95% confidence interval has a 95% probability of containing the true parameter, so one can say the parameter is 95% likely to lie between the computed endpoints.

Related units

Requires

Sampling Distributions and Standard Error

Connected

Hypothesis Tests for Experimental Research (contrasts with)

Learn this topic

Confidence Intervals for Experimental Research

Used in

Inferential Foundations for Experimental Research

Sources

All of Statistics: A Concise Course in Statistical Inference (2004)

Confidence Intervals for Experimental Research

Assumptions and scope

Worked material

What an interval reports that a p-value does not

Statements an interval does not support

What the confidence level describes

Common errors

Related units

Requires

Connected

Learn this topic

Used in

Sources

Settings

Focus timer

Today

Last seven days

Where the time went

Recent sessions

Settings

Notifications

Your history