Subject
Experimental Research Design
Whether a treatment caused a change, and what evidence licenses saying so. Starts from the fact that each unit reveals only one of its two potential outcomes, builds the designs that make a comparison credible, and ends with what can and cannot be claimed when treatment was never assigned by anyone.
Learning paths
Inferential Foundations for Experimental Research
Sampling distributions, standard errors, hypothesis tests, confidence intervals, ANOVA and regression for experimental research.
Causal Inference from Experiments and Observational Data
When a comparison supports a causal claim. Defines the causal estimand, covers the designs that support a credible comparison and the inference each supports, and treats what can be claimed under stated assumptions when treatment was not assigned.
What this subject develops
Reason about an estimate's variability from its procedure
Reason about how much an estimate would have moved under a different sample. Standard errors, reference distributions, intervals and comparisons of several groups all rest on that one idea. Knowing which distribution a claim refers to is what keeps the rest straight. The probability objective sits first because that reasoning is conducted entirely in its notation: a standard error is the square root of a variance, the rule that produces it holds only for uncorrelated observations, and the commonest misreadings of a p-value or a diagnostic result are inversions of a conditional probability.
State what causal quantity is being estimated
Say exactly which causal quantity you are estimating before you estimate it. Separate an individual effect from an average one, and the quantity itself from the method that approximates it. A fluent calculation of an unnamed quantity answers nothing.
Read and choose an assignment mechanism
Identify how treatment was assigned and choose a design that fits the units and the question. Design is where a credible comparison is won or lost. No later analysis recovers what the design did not provide.
Quantify uncertainty under the data-generating mechanism
Produce an interval or a p-value whose reference matches the process that actually generated the data. The arithmetic is often identical across methods; what differs is what the number claims.
Judge whether an observational comparison identifies an effect
State what an observational comparison has to assume, check what the data can check, and say plainly what it cannot. A balanced table is evidence about the adjustment, never proof that the assumption holds.
See detailed outcomes
Reason about an estimate's variability from its procedure
- The learner can compute the expectation and variance of a discrete random variable by either route, and apply the linearity and scaling rules to a linear transformation of it.
- The learner can determine whether the variance of a sum decomposes, justifying the decision by the covariance term, and can distinguish independence from zero covariance by exhibiting or interpreting a pair that is uncorrelated yet dependent.
- The learner can compute a conditional probability from a joint table or mass function, apply the law of total probability, and obtain a posterior by Bayes' rule without inverting the conditioning or discarding the base rate.
- Given a described sampling situation, the learner can obtain the standard error of a mean, say what the central limit theorem does and does not claim, and identify when the independence the formula assumes fails.
- Given a described study, the learner can identify the unit of analysis and the data structure, paired or independent, one sample or two, and select a test whose standard error matches that design.
- Given an estimate, a null value and the design's standard error, the learner can compute the standardized distance, name the reference distribution with its degrees of freedom, and state what the result does and does not establish, treating failure to reject as inconclusive rather than as evidence for the null.
- Given an estimate and its standard error, the learner can construct the appropriate interval and state what the confidence level does and does not claim.
- Given an ANOVA table or a description of a multi-group comparison, the learner can read the decomposition, compute or check the statistic, and state precisely what rejection establishes.
- Given regression output, the learner can state what a coefficient means as an average change per unit of the predictor, conditional on the others in the model and with its units, and treat
as in-sample explained variation rather than evidence of correctness. - Given regression output and a described study, the learner can identify the design or identification argument, not the fit, as the source of any causal reading, and say when the classical standard errors are inappropriate and what should replace them.
- Given a model containing indicators and interactions, the learner can state what each coefficient means, identify the reference point it depends on, and recover a quantity of interest from the parameterisation.
- Given a binary-outcome analysis, the learner can say what each model reports, convert or refuse to convert between scales, and identify a result stated on the wrong one.
State what causal quantity is being estimated
- Given a description of units, an assignment, and observed data, the learner can write each unit's potential outcomes and observed outcome, state which quantities are known and which are missing, and explain why no individual treatment effect can be computed from the data.
Read and choose an assignment mechanism
- Given a description of how treatment was assigned, the learner can identify the mechanism, state whether it is known and independent of the potential outcomes, say what causal claim it supports, and explain why observed imbalance in a single realisation is not evidence against it.
- Given a description of units and a design, the learner can decide whether blocking or pairing is warranted, compute the estimator the design requires, and identify analyses that ignore the design structure.
- Given an experiment and a proposed regression, the learner can say what adjustment contributes, judge whether the covariates are admissible, and state what the specification cannot repair.
Quantify uncertainty under the data-generating mechanism
- Given summary data from a completely randomized experiment, the learner can compute the difference in means, compute its conservative variance and standard error, form a confidence interval, and explain which term of the exact variance has been omitted and why that makes the result conservative rather than incorrect.
- Given a small experiment and its assignment mechanism, the learner can state the sharp null, enumerate or sample the permitted allocations, compute the randomization p-value, and explain what rejecting or failing to reject establishes, distinguishing it from a claim about the average effect.
Judge whether an observational comparison identifies an effect
- Given an observational study, the learner can state the assumptions adjustment requires, judge which of them the data can speak to, and say what the comparison would estimate if an assumption failed.
- Given an observational study, the learner can say what the propensity score is for, judge a fitted score by balance and overlap rather than by predictive accuracy, and state what it does not repair.
- Given estimated propensity scores, the learner can form inverse-probability weights, compute the weighted estimate, and diagnose what extreme weights indicate about the comparison.
- Given a described matching procedure, the learner can say which estimand it targets, how its design choices change the population described, and what it does not establish.
Browse Experimental Research Design reference · Practice Experimental Research Design