Practice: Validity as an Argument for an Interpretation

(a) The interpretation. The proposal relies on: this score measures academic preparation well enough that a higher score identifies an applicant more likely to benefit from, and succeed with, the scholarship. That is a claim about individual-level prediction, for a selection decision with a fixed number of awards.

Naming the decision matters, because the same test might be defensible for advising applicants about preparation and indefensible for allocating money between them.

(b) Evidence that would bear on it. Three kinds, the third of which could count against:

• Criterion evidence at the individual level within the relevant range. The $0.28$ is computed across all applicants; the decision concerns the narrow band competing for the scholarship, where range restriction will attenuate the relation further. The relevant question is how scores relate to outcomes among applicants near the cut.
• Response-process evidence. Whether applicants answer from the preparation the test intends to measure, or from coaching, familiarity with the format, or test-taking speed. Evidence here would come from think-aloud studies or from comparing coached and uncoached groups.
• Consequential evidence, which could count against the interpretation. Once a scholarship depends on the score, preparation behaviour changes, and any group difference in access to coaching becomes a difference in awards. If the score's relation to outcomes differs across groups, or if the ranking systematically disadvantages a group without corresponding differences in outcomes, that is evidence against the proposed use even if the correlation holds overall.

(c) What each number establishes. $\alpha =0.91$ says the items covary substantially in last year's applicant pool, under a measurement model with errors uncorrelated across items. It establishes that the score is stable enough to be worth interpreting, and nothing at all about what is being measured: a highly consistent test of coaching familiarity would report the same figure. "A valid measure" is not a property the coefficient can confer.

The $0.28$ is the only figure here bearing on the interpretation, and it is weak: it accounts for under 8% of variance in first-year grades, it is computed on the full range rather than near the decision boundary, and it is an association rather than evidence that the scholarship will produce better outcomes for higher scorers. Note also that reliability bounds it. With ${\rho }_{X{X}^{\prime }}=0.91$ and a perfectly reliable criterion the observed correlation could reach about $0.95$, so unreliability is not what limits this one. The construct relation is simply weak, and the proposal needs evidence it does not yet have.

(a) The two interpretations.

Original: this patient's score indicates their current wellbeing well enough that a low score identifies someone who would benefit from a follow-up appointment.

Proposed: the mean score of a ward's patients indicates the quality of care that ward provides, well enough to rank wards against one another.

Writing both as sentences is what makes the difference visible. They share an instrument and share nothing else.

(b) Why the evidence does not transfer. Three things changed at once.

The unit changed, from a patient to a ward. A ward mean is not a larger version of a patient score; it aggregates over patients who were not sampled to represent the ward.

The construct changed, from an individual's wellbeing to an organisation's quality of care. A ward mean encodes case mix as much as care: a ward admitting sicker patients scores lower with no difference in what it does.

The decision changed, from offering one person an appointment to ranking units with consequences attached. Validity attaches to an interpretation for a use, so a change in any of the three requires its own argument. Nothing about the scores changed, which is exactly why the instrument feels transferable and is not.

(c) Evidence that would bear on it.

• Case-mix adjustment. Whether ward differences survive adjustment for admission severity, age and diagnosis. Without it the ranking may be measuring intake.
• Stability at the ward level. Whether a ward's mean is stable across months and across patient samples. Individual-level reliability says nothing about this: a highly reliable individual score can still produce an unstable mean from few respondents.
• Consequential evidence, which could count against it. What happens once rankings carry consequences: whether response rates fall on low-ranked wards, whether favourable respondents are encouraged, and whether measured improvement is matched by an independent indicator. Improvement on the ranking alone is evidence against the interpretation.

(d) What the coefficient establishes. $\alpha =0.92$ says the 20 items covary substantially in the development sample, under the classical model with errors uncorrelated across items. It establishes that individual scores are consistent enough to be worth interpreting.

It cannot establish what is measured — a consistent questionnaire about catering would report a similar figure — and says nothing about the ward level, having been computed on individual responses. 'Validated' is not a property a coefficient confers, and names nothing until the interpretation and the use accompany it.