Practice: Reliability and Measurement Error

Error diagnosis · Classification

A test manual reports $ρ_{X X^{'}} = 0.91$ from a national standardisation sample. A school administers the same test to its sixth form alone and obtains $0.62$ .

The head of department concludes: "Either the manual is wrong or we administered it badly."

Which statement is correct?

2 hints available, least help first.

Hint 1: Retrieval cue

Write reliability as a ratio. Which of its two parts does a narrower group change?

Hint 2: Concept cue

Convert both coefficients to the score scale and compare.

Direct application

A scale has reliability $α = 0.9575$ and its total scores have standard deviation $5.2715$ . Compute the standard error of measurement to four decimal places.

Your answer

Enter the value. It is checked against the answer and the precision this task asks for.

2 hints available, least help first.

Hint 1: Retrieval cue

$SEM = σ_{X} \sqrt{1 - ρ_{X X^{'}}}$ .

Hint 2: Next step

$1 - 0.9575 = 0.0425$ , and $\sqrt{0.0425} \approx 0.2062$ .

Interpretation

A classroom test reports $α = 0.82$ across a full year group and $α = 0.61$ when computed on the top set alone. The same test, the same conditions, the same marking. What is the accurate reading?

Method selection

A licensing board sets a pass mark on a written examination and must decide which candidates near the boundary can be separated by the instrument. The technical report gives $α = 0.94$ , the number of items, the standard deviation of total scores, and a table of classification consistency near the cut. Which figure answers the board's question, and why?

Construction · Direct application · Explanation

A 40-item attainment test reports $ρ_{X X^{'}} = 0.88$ on a full year group whose total scores have standard deviation $14.0$ .

(a) Compute the standard error of measurement and state the band a single pupil's score carries at about 95% confidence, naming the assumptions that band rests on.

(b) Two pupils score 61 and 66. Say whether this instrument separates them, showing the comparison the question requires.

(c) The same test given to the top set alone, where scores have standard deviation $7.5$ , reports $0.56$ . Say what changed and what did not, and which figure is the correct reliability for that group.

Write your answer

Write your answer, then compare it with the worked solution.

3 hints available, least help first.

Hint 1: Retrieval cue

$SEM = σ_{X} \sqrt{1 - ρ_{X X^{'}}}$ puts the coefficient on the score scale.

Hint 2: Concept cue

In (b), comparing two measured scores needs the error of their difference.

Hint 3: Strategy cue

In (c), compute the SEM for both groups before deciding what changed.

Compare with the worked solution

Comparing does not record a result. Judging your own written answer cannot show that you can do this without help.

(a) The standard error of measurement.

SEM = σ_{X} \sqrt{1 - ρ_{X X^{'}}} = 14.0 \sqrt{0.12} = 4.850 .

A band of roughly $\pm 2 SEM \approx \pm 9.7$ points, so an observed 61 is consistent with true scores from about 51 to 71.

What the band assumes. Two things worth stating rather than inheriting. It assumes a distribution for measurement error, conventionally normal, since $\pm 2$ standard errors is a normal-theory interval. And it describes the spread of observed scores about a fixed true score over hypothetical repeated administrations; reading it as a 95% interval for the pupil's true score needs further assumptions the classical model does not supply.

(b) Comparing two pupils. The operative quantity is the standard error of the difference:

{SEM}_{diff} = SEM \sqrt{2} = 6.859 .

The gap is $66 - 61 = 5$ , which is $5 / 6.859 = 0.73$ of one such unit — well inside what measurement error alone produces. This instrument does not separate these two pupils, and reporting them as differently attaining would be reporting noise.

(c) The restricted group.

SEM = 7.5 \sqrt{1 - 0.56} = 4.975,

essentially unchanged from $4.850$ . That is the point. The error variance did not change — same instrument, same conditions — while the true-score variance fell, because the group is more homogeneous. Reliability is the ratio $σ_{T}^{2} / σ_{X}^{2}$ , so with the error term fixed the ratio drops.

Which figure is correct. Both, for their own group. $0.56$ is the correct reliability of these scores in the top set; $0.88$ is correct in the year group. Neither is a property of the test, which is why a coefficient quoted without its population and conditions describes an administration that is not the one being conducted.

A complete answer does each of these:

locates reliability in scores

Practice: Reliability and Measurement Error

Error diagnosis · Classification

Direct application

Interpretation

Method selection

Construction · Direct application · Explanation

Session complete

Settings

Focus timer

Today

Last seven days

Where the time went

Recent sessions

Settings

Notifications

Your history