Practice: Linear Regression for Experimental Research

The coefficient. Among stores in the same size category, one spending £1,000 more per month is associated with £1,850 more monthly revenue on average. The comparison is conditional on store size, and the units follow those of the variables as entered.

The test. $t=1.85/0.42\approx 4.40$ on $60-3=57$ residual degrees of freedom, far beyond any conventional critical value. The association is precisely estimated.

What it does not support. A causal claim. Stores chose their own budgets, so a store expecting a strong month plausibly advertises more. The coefficient mixes any effect of advertising with whatever drove the spending decision, and least squares cannot separate them.

What $R^2=0.71$ does not add. It says the two predictors track 71% of revenue variation in these 60 observations. It is not evidence that the model is correct, that no confounder is missing, or that the coefficient is unbiased. A confounded model can track an outcome closely.

A further caution. Revenue variance plausibly grows with store size, so homoskedasticity is doubtful and robust standard errors are the safer choice.

What to report. The conditional association with robust standard errors, an explicit statement that spending was not assigned, and, if the causal question is the one that matters, a proposal to randomize advertising budgets across stores.

The central error. The coefficient is almost certainly reverse causation. Patrols are allocated to neighbourhoods with high crime, so patrol hours are a consequence of crime at least as much as a potential cause of it. The positive sign is what that allocation process produces, and acting on it by reducing patrols could be actively harmful.

The $R^2$ error. An $R^2$ of 0.44 is offered as grounds for confidence. It reports in-sample tracking only. A model can track an outcome closely while reversing cause and effect, indeed that is exactly what happens here, because allocation makes patrols track crime well.

What the standard error shows. That the association is precisely estimated. Precision is not correctness of interpretation: a tightly estimated coefficient can be tightly estimating the wrong thing.

Whether the standard error can be trusted. Crime rates across neighbourhoods differ enormously in scale, so the error variance almost certainly grows with the neighbourhood's baseline level. Classical standard errors assume homoskedastic uncorrelated errors and would understate the uncertainty here; robust standard errors, clustered if neighbourhoods are observed repeatedly, are the appropriate choice.

What should be reported. The association, with an explicit statement that patrol allocation is determined by crime and therefore endogenous. Whether any variation in patrol hours arose independently of local crime, a staffing shock, a policy change, a boundary redrawing, a randomized pilot, since such variation could support a credible design. And absent that, no recommendation about patrol levels from this analysis at all.

Why no amount of fit would help. A specification with $R^2=0.95$ built the same way has the same problem. Only a change in where the variation comes from addresses it.

The badge is $R^2$ renamed. Calling it "model quality" invites exactly the upgrade the statistic does not support. A score of 0.91 says the model tracks 91% of the variation in churn within this sample. It is not a measure of correctness, and it says nothing about whether any coefficient can be read causally.

The likely reverse causation. Customers contact support, or get contacted, when something is going wrong. A billing problem, a failed integration, declining usage. Those same troubles drive churn. So support contact is a marker of a customer already in difficulty, and the association runs from trouble to both contact and churn.

Why the high score makes it worse, not better. A model tracking churn well has found strong correlates of churn, and support contact is a strong correlate precisely because it flags struggling customers. The badge rewards the feature that makes the causal reading wrong.

The consequence of acting on it. Reducing outreach would remove help from the customers most at risk, and would plausibly increase churn, with the model then showing the association weakening, which could be misread as vindication.

What I would report. The association stated as such, with the reverse-causation mechanism named explicitly. The timing of contact relative to churn signals, since a contact following a usage drop is clearly a consequence. And, if the question is whether outreach helps, a randomized trial: withhold proactive outreach from a random subset of at-risk customers and compare. That is a design question, and no fit statistic on observational data substitutes for it.

How the uncertainty was computed. A churn model fitted over customers of very different sizes and tenures will have error variance that grows with account size, and customers within the same account or cohort are not independent. Classical standard errors assume neither problem exists. Before quoting any coefficient as precise, ask whether robust, and where the data are grouped, clustered, standard errors were used; a badge derived from $R^2$ says nothing about this.

The income coefficient. Households differing by £1 in monthly income differ by £0.31 in monthly spending ON AVERAGE, among households of the SAME SIZE. Three parts carry the meaning and each is load-bearing:

• Units. Pounds of spending per pound of income, dimensionless here, but it must be stated, because the same number against income in thousands would mean something else entirely.
• On average. It describes a mean difference across households, not what any particular household does.
• Conditional on size. The coefficient compares households at the same household size. Without that clause it would be the unconditional association, which is a different quantity.

The $R^2$. The model accounts for 46% of the variation in spending IN THIS SAMPLE. That is all it reports. It is not the probability the model is right, not a measure of how well the model would predict a new household, and not evidence about whether income causes spending.

What neither number reports. Nothing here establishes that raising a household's income by £1 would raise its spending by £0.31. That is a claim about an intervention, and it rests on the design, not on the table.

What would have to be true. Conditional on store size, region and month, whether a store ran the campaign would have to be unrelated to everything else affecting its sales. Formally: within cells defined by those three controls, campaign assignment is as good as random with respect to potential sales. A plausible violation. Campaigns are not assigned at random. A regional manager chooses where to run them. If they target stores already trending upward, or stores in a catchment with a new housing development, then the campaign indicator is standing in for the growth that would have happened anyway. The coefficient then measures selection, and no amount of controlling for size and region repairs it, because the manager's reason is not in the data. Note the direction matters for the reading: targeting promising stores inflates the estimate, while targeting struggling stores would deflate it. "There is confounding" is a weaker statement than "the confounding runs this way." A more defensible analysis of the same data. Use the panel structure. Compare each store to ITSELF before and after its campaign, and difference that against stores that ran none in the same weeks. A difference-in-differences. It removes every fixed store characteristic, observed or not, and rests instead on a parallel-trends assumption, which is at least partly checkable: plot the pre-campaign trajectories of both groups and look for divergence before the campaign began. (c) The standard errors. No. There are 15,600 rows but only 300 stores, and sales within a store are correlated week to week. A store that sells well in one week sells well in the next, for reasons that have nothing to do with the campaign. Ordinary standard errors treat every row as an independent observation and will be far too small, so an effect can appear decisively estimated on the strength of repeated measurements of the same few stores. Cluster the standard errors at the STORE level, which is the level at which the campaign varied and at which the correlation lives. With 300 clusters the asymptotics are comfortable; with a dozen they would not be, and a wild bootstrap would be the alternative. The point. Neither improvement came from a better fit or more controls. One came from a design making a weaker, checkable assumption; the other from counting how many things could actually have come out differently.