Practice: Internal Consistency and Coefficient Alpha

1. The coefficient. $\alpha =0.93$ establishes that the 30 items covary substantially in this sample of 1,200 employees across all grades. It is a lower bound on reliability, not an estimate of it, and it describes these scores in this population rather than the instrument. The length is doing much of the work. Inverting Spearman-Brown, $\overline{r}=\alpha /(k-(k-1)\alpha )$ with $k=30$ and $\alpha =0.93$ gives $\overline{r}\approx 0.31$. So items sharing under a third of their variation reach $0.93$ because there are thirty of them. For comparison, four items at $\overline{r}=0.85$ give $\alpha =0.9575$. A similar coefficient from an entirely different instrument. The implied $\overline{r}\approx 0.307$ can be checked against the matrix described. With 20 items at $0.45$ within, 10 at $0.50$ within and $0.08$ across, the pair counts are $190$, $45$ and $200$ out of $435$, giving a mean pairwise correlation of $0.285$, which would produce $\alpha =0.923$ rather than the reported $0.93$. Close enough to be consistent with rounding in the description, and not identical; running that check is what would catch a reported coefficient that the matrix could not produce. 2. The score scale.

taking $\alpha$ as the reliability estimate, which presumes the items are at least essentially tau-equivalent. A 95% band is roughly $\pm 2\mathrm{SEM}\approx \pm 6.3$ points on a scale running 30 to 150, about 5% of the full range. Comparing two scores needs the standard error of their difference, $\mathrm{SEM}\sqrt{2}\approx 4.48$ when the two measurements are independent. A 6-point gap is about $1.34$ of those units, so the difference is small relative to measurement error and does not by itself separate the two employees. Since the proposal is to allocate bonuses to individuals, this is the operative figure and the one the note leads with least. 3. Dimensionality. The matrix describes two blocks: items 1 to 20 cohering at about $0.45$, items 21 to 30 at about $0.50$, and almost nothing between them at $0.08$. That pattern is strong evidence against a single homogeneous scale. Establishing how many dimensions there are, and what they are, requires a dimensional analysis such as factor analysis or parallel analysis rather than inspection of the correlation blocks alone. The near-zero cross-block correlation is the signature. $\alpha =0.93$ is entirely compatible with this, which is the point: the coefficient does not report the number of dimensions. Summing across both blocks produces a total whose meaning depends on the weighting between two unrelated things, and two employees with identical totals may differ completely in composition. I would report two subscale scores with their own coefficients and standard errors, and not a single total. If a single index is wanted, that requires an argument about why the two facets should be combined and in what proportion. 4. The proposed use. The interpretation management relies on is: this total score measures an employee's engagement well enough that a low score identifies someone worth paying to retain. That is a claim about individual-level prediction and about a decision. Evidence bearing on it would include the relation between scores and subsequent departure at the individual level, whether the relation holds within grades rather than only across them, whether the items behave equivalently across groups receiving different treatment, response-process evidence that employees answer about engagement rather than about their manager, and consequential evidence about what happens once bonuses are known to depend on the score. The correlation of $0.31$ with voluntary departure is weak support and does not establish the proposal. It accounts for under 10% of variance in departure, it is reported across all grades so it may reflect grade differences rather than individual engagement, and it is an individual-level association pooled across grades, so it may be confounded by grade or differ between grades, and it is being used to justify individual decisions. Reliability limits it: under the classical model the observed correlation is attenuated by $\sqrt{{\rho }_{X{X}^{\prime }}{\rho }_{Y{Y}^{\prime }}}$, so with ${\rho }_{X{X}^{\prime }}=0.93$ and a perfectly reliable criterion it could not exceed about $0.96$, so unreliability is not what limits it here. The construct relation is simply weak. A further consideration: attaching money to the score changes the response process. Once employees know a low score triggers a bonus, the scores stop measuring what the validity evidence was gathered on, and the instrument degrades in use. 5. What I would report instead. - Two subscale scores, each with its own $\alpha$, its standard deviation and its SEM, rather than one total.
- The SEM in points and the resulting band, stated before the coefficient.
- The number of items alongside every coefficient, and the population and conditions each was computed on.
- The implied mean inter-item correlation, so a reader can see whether the coefficient came from item quality or from test length.
- A statement of the interpretation the scores are offered for, with the evidence supporting it and the evidence still missing.
- An explicit note that the instrument has not been validated for individual allocation decisions, and that the correlation of $0.31$ does not support that use.