Practice: Inverse Probability Weighting

Treated contributions. Unit 1: $30/0.50=60$. Unit 2: $20/0.25=80$. Sum $=140$. Control contributions. Unit 3: $12/(1-0.50)=12/0.50=24$. Unit 4: $18/(1-0.75)=18/0.25=72$. Sum $=96$. The estimate.

The weights. Treated: $2$ and $4$. Controls: $2$ and $4$. The largest is 4, and no score approaches 0 or 1, so nothing here indicates an overlap problem, though with four units the diagnostics are illustrative rather than informative. ** Unit 4 carries a weight of 4 because it looked 75% likely to be treated and was not; it is doing a quarter of the work in this small example. In a real sample the figures to report alongside the estimate are the maximum weight, the effective sample size, and weighted balance on the original covariates. What to report beside it. A weighted point estimate is not readable alone. Report weighted covariate balance, the maximum weight and the effective sample size, together with sensitivity to the propensity specification. With four units these are illustrative; in a real sample they are what decides whether the estimate means anything. The fifth unit. A control with $\hat{e}=0.98$ carries weight $1/(1-0.98)=50$. One respondent standing in for fifty. It is a control who looked almost certain to be treated, and there are few such controls precisely because the covariates nearly determined assignment there. What dropping it would change. Trimming that unit does not clean up the estimate; it changes what is being estimated. The remaining sample is everyone with $\hat{e}$ below the trimming threshold, so the answer becomes an average effect within that subpopulation, and the original ATE over all five units is no longer what the number describes. That is a legitimate move only if it is reported as one: state the rule, how many units it removed from each arm, and name the retained population as the estimand. Why this is not the same as capping it.** Setting the weight to, say, 10 keeps the unit in the sample and alters the estimator instead. The capped quantity has no tidy description as an effect for any stated population, which makes it harder to report honestly than trimming, not easier, and either way, the comparable units that were missing are still missing.

The error. A maximum weight of 640 is being read as a variance problem. It is an overlap failure: a weight that size implies an estimated propensity near 0.0016, meaning some units had essentially no chance of the treatment status they had, and the data contain almost nothing about what comparable units would have done otherwise.

What winsorising does. It replaces the largest weights with smaller ones, which stabilises the variance and silently changes the estimand. The affected units are no longer represented as themselves. The comparison for those units still has no empirical basis; only the symptom has been treated.

Why the balance figure does not rescue it. Standardized differences below 0.03 are computed over the altered weighted sample. Balance cannot distinguish genuine comparison from model-based extrapolation, and it speaks only to the measured covariates in any case.

What should be reported. The propensity-score distributions by arm and the proportion of each arm beyond 0.05 and 0.95. The effective sample size, which with weights of this magnitude is likely a small fraction of the cohort. If units are to be removed, explicit trimming to a region of overlap, with the estimand restated as the effect within that subpopulation rather than the original ATE. Uncertainty estimates that account for the weights having been estimated. And a statement that unconfoundedness remains assumed throughout, with a sensitivity analysis, health-seeking behaviour plausibly drives both screening and mortality and is rarely measured.

What the report should carry. Weighted covariate balance, the maximum and tail weights, and the effective sample size before and after winsorising. A weight of 640 with an unreported effective sample size hides how few observations carry the estimate, and stabilising the variance does not restore the comparable units that were never there.

The structure is the same. Weighting by the inverse of a sampling probability to build a pseudo-population that matches a target composition is the mechanism from this unit, wearing survey vocabulary. What weights above 400 indicate. A handful of rural respondents are standing in for hundreds of people each. The estimate for that segment rests on a few individuals' answers, so it will move substantially if any one of them differs. This is the same limitation as extreme propensity weights, and it is about the information available, not the arithmetic. Why matching the census exactly is not reassurance. Composition matching is the analogue of covariate balance: it shows the weighting achieved what it was asked to achieve on region. It says nothing about whether the few rural respondents resemble rural non-respondents in their product preferences. The analogue of unconfoundedness, and equally unverifiable from these data. What I would report. The effective sample size overall and within region, which is the honest measure of how much information the weighted estimate carries; the number of distinct rural respondents behind the rural estimate; the maximum weight; and an interval that accounts for the weighting rather than treating the weighted count as the sample size. If rural coverage is too thin to support a segment estimate, that should be stated rather than concealed by a composition that matches the census. The remedy is design. Recruiting more rural respondents addresses the problem; no reweighting of the existing panel does. Diagnostics for a survey weight too. Report the maximum weight, the effective sample size and balance on the weighting variables. A design weight and a propensity weight raise the same question, how much of the estimate rests on how few respondents, even though only one of them is estimated. Capping at 100. The high-weight respondents stay in the sample and their influence is reduced by fiat. That changes the estimator, buying stability at the price of bias, and what it now estimates has no clean description as the preference of any stated population. The census composition it was built to match is no longer reproduced either. It is the harder of the two to report honestly, because nothing in the output announces it. Dropping them. This changes the population rather than the estimator. The estimate becomes product preference among respondents whose weights fall below the threshold, which, given how the weights arose, means substantially excluding the rural segment the weighting existed to represent. That is defensible only when stated: the rule, the number dropped, and the retained population named as what the 38% now describes. What I would report. Neither as a repair. Both remedies address the arithmetic and neither creates the rural respondents the panel lacks. Report the unmodified weighted estimate with its effective sample size, the number of distinct rural respondents behind the rural figure, and the maximum weight, and if a trimmed figure is reported alongside it, report it as an estimate for the retained population rather than as a corrected national one.