Reliability and Measurement Error

An observed score as a true score plus error, reliability as the share of observed variance that is not error, the standard error of measurement that puts the same information on the score scale, and why both are properties of scores from a population rather than of the instrument.

Definition

Classical test theory writes an observed score as $X = T + E$ , where $T$ is the true score, the expected observed score over hypothetical independent administrations, and $E$ is measurement error with $E [E] = 0$ and $E$ uncorrelated with $T$ . Consequently $σ_{X}^{2} = σ_{T}^{2} + σ_{E}^{2}$ .

Reliability is the share of observed variance that is true-score variance,

ρ_{X X^{'}} = \frac{σ_{T}^{2}}{σ_{X}^{2}} = 1 - \frac{σ_{E}^{2}}{σ_{X}^{2}},

a number between 0 and 1. The standard error of measurement converts a reliability coefficient back to the score scale:

SEM = σ_{X} \sqrt{1 - ρ_{X X^{'}}} .

Assumptions and scope

The decomposition $X = T + E$ defines the true score as an expectation over hypothetical repeated administrations under identical conditions. It is not the respondent's real standing on the construct, and nothing in classical test theory claims otherwise.
$SEM = σ_{X} \sqrt{1 - ρ_{X X^{'}}}$ requires a reliability coefficient. Substituting $α$ uses alpha as the reliability estimate under the assumed measurement model, which is exact only under essential tau-equivalence with uncorrelated errors; where those fail, the resulting SEM inherits the error in alpha.
A $\pm 2 SEM$ band is a 95% interval only under an assumed error distribution, conventionally normal, and it describes the spread of observed scores around a fixed true score over hypothetical repeated administrations. It is not a 95% interval for the respondent's true score without further assumptions, and reporting it as one overstates what the measurement model supplies.
Reliability coefficients are population-dependent. Restricting the range of the group being tested lowers the coefficient without changing the instrument, which is why a figure quoted from a manual cannot be assumed to hold for a new group.

Worked material

Example

Five instruments and what each coefficient settles

A 40-item anxiety questionnaire reporting $α = 0.91$ . Consistent with items sharing a great deal, and equally consistent with items sharing a fifth of their variation: at $\bar{r} = 0.20$ , forty items give $0.9091$ . The coefficient alone does not distinguish the two, and reporting $k$ alongside it does most of the work of telling them apart.

A 4-item scale reporting $α = 0.96$ . Far more informative, because four items cannot reach that by length. Inverting Spearman-Brown, the implied mean inter-item correlation is about $0.85$ . The items are nearly redundant with one another, which raises a different question: whether four items this similar are covering the construct or asking one question four ways.

A classroom test where $α$ drops from $0.82$ to $0.61$ between the full cohort and the top set. Nothing about the test changed. Restricting the range cut true-score variance while error variance stayed put, so the ratio fell. The second figure is the correct reliability for that group, and the instrument is neither better nor worse than it was.

---

In all three the coefficient is doing its job and is being read without the context that makes it interpretable: the number of items, and the population.