Practice: When a Coefficient Is Not an Effect

1. The control. The bias formula predicts ${\hat{\beta }}_{\text{short}}-{\hat{\beta }}_{\text{long}}={\beta }_2\delta$, where ${\beta }_2=640$ is tenure's coefficient and $\delta =0.198$ is the slope of tenure on training hours:

The observed movement is $312-185=127$. These agree to the precision the figures are reported at, so the arithmetic is consistent. What this establishes: tenure was a confounder, it was positively related to both training and output, and omitting it inflated the coefficient by about $127$ units per hour. What it does not establish: that $185$ is the effect. The relation is an algebraic identity between two fits on the same sample and holds exactly whatever generated the data. It would hold just as neatly with three further confounders still missing. If anything, a movement of $41\mathrm{\%}$ when one control is added is evidence that controls matter a great deal here, which is a reason to expect the unmeasured ones to matter too. 2. The remaining threat. Training was requested by employees and approved by supervisors. Neither the request nor the approval is random, and both plausibly depend on the employee's motivation, ability, and existing performance, none of which is in the data. Direction: an employee who is more capable or more motivated produces more output, so for that unmeasured factor, and such employees are more likely to request training and be approved, so . The product is positive, and the coefficient of $185$ is therefore still biased upward. The true effect is smaller than $185$, possibly much smaller. There is a second, subtler channel. Supervisors approving requests may favour employees they expect to improve, which makes approval depend on an expectation of the outcome, closer to simultaneity than to a simple omitted variable, and equally not repaired by controls. 3. The fit statistics. They contribute nothing to the question. $R^2=0.94$ says the equation as specified tracks output well using the regressors included. It is computed from those regressors and cannot register a determinant that is absent, motivation is by construction not in the model. A regression can reach $R^2=0.99$ with a coefficient wrong by more than a factor of two. The standard error of $11$ describes how much the coefficient would vary across samples under this specification. It makes the estimate precise, not correct: a precisely estimated biased coefficient is precisely wrong, and the narrow interval makes the number look more authoritative than it is. A sample of 4,000 sharpens both figures and moves the bias not at all. 4. The instrument. Relevance is satisfied. A first-stage $R^2$ of $0.71$ means distance explains much of the variation in training hours, which is unsurprising, travel cost affects attendance. Exclusion is doubtful, and this is the condition that matters. Distance from home to the training centre is a function of where the employee lives, which is related to housing cost, commuting time, seniority, and possibly which site they work at. Each of those plausibly affects output directly: a long commute is tiring, site differs in equipment, and residential location correlates with tenure and pay. If distance reaches output by any route other than through training hours, the estimate is not repaired. The first-stage $R^2$ bears only on relevance and is silent on exclusion. The two are different conditions, and only one has a sample analogue. An instrument can have an excellent first stage and return an estimate no better than the biased least-squares one it replaced, on the unit's worked data, a first-stage $R^2$ of $0.986014$ yields $2.451064$ where the structural coefficient is $1$, essentially identical to the biased $2.449301$. Notice also what the instrumented estimate does here: $205$ is higher than the controlled $185$. If the instrument were valid and the remaining bias were upward, one would expect the instrumented figure to fall. That it rises is a signal worth explaining rather than reporting. 5. What I would report. What the analysis supports: employees who trained more produced more, by about $185$ units per hour after adjusting for tenure; the association is precisely estimated and the adjustment for tenure moved it substantially. What it does not support: that training caused this, or any specific figure for the causal effect. The direction of the remaining bias is upward, so $185$ and $205$ should both be read as upper bounds rather than estimates. To obtain the number the firm wants, the assignment has to change. The clean option is to randomise: offer training to a random subset of requesters, or randomise the order in which a waiting list is served, which is often acceptable operationally because capacity is limited anyway. Failing that, an administrative rule creating variation unrelated to employee characteristics, a capacity cap at a threshold, a scheduling quirk, would supply a credible instrument, with the exclusion argument stated explicitly rather than assumed from a first-stage statistic.