Covariance, Independence and the Variance of a Sum

Covariance as the cross term that decides whether the variance of a sum is the sum of the variances, independence as the stronger condition that the joint distribution factors, and the one-way implication between them that makes a zero correlation weaker evidence than it appears.

Definition

Covariance. For random variables $X$ and $Y$ ,

Cov (X, Y) = E [(X - E [X]) (Y - E [Y])] = E [X Y] - E [X] E [Y],

positive when the two tend to lie on the same side of their means together, negative when on opposite sides. Variables with $Cov (X, Y) = 0$ are called uncorrelated, equivalently $E [X Y] = E [X] E [Y]$ .

The variance of a sum. Expanding $(X + Y)^{2}$ and taking expectations gives

Var (X + Y) = Var (X) + Var (Y) + 2 Cov (X, Y),

so the familiar addition rule

Rule	Holds
$E [X + Y] = E [X] + E [Y]$	always, even if dependent
$Var (X + Y) = Var (X) + Var (Y)$	only if uncorrelated
$E [X Y] = E [X] E [Y]$	only if uncorrelated

is the special case in which the cross term vanishes. The asymmetry is the point: expectation adds unconditionally, variance does not.

Independence. Events $A$ and $B$ are independent when $P (A \cap B) = P (A) P (B)$ . Random variables $X$ and $Y$ are independent when their joint distribution factors, $P (X = x, Y = y) = P (X = x) P (Y = y)$ for every pair of values.

The one-way implication.

independent ⟹ Cov = 0 ⟹ Var (X + Y) = Var (X) + Var (Y),

with neither implication reversible. Covariance detects linear association only, so a symmetric non-linear relationship can be perfectly deterministic and still yield $Cov = 0$ .

Assumptions and scope

$Var (X + Y) = Var (X) + Var (Y)$ requires the variables to be uncorrelated. Without that, the covariance term is present and the identity fails.
Independence implies uncorrelatedness, but uncorrelated variables can be strongly dependent: $Cov = 0$ detects only linear association.
Uncorrelatedness is exactly what variance addition requires; independence is more than required. An argument resting on independence where uncorrelatedness would serve is claiming more than it needs.
Covariance is not scale-free: multiplying $X$ by a constant multiplies the covariance by it too, which is why correlation rather than covariance is reported when the magnitude is to be compared across variable pairs.

Worked material

Example

A pair with a covariance that does not vanish

Every pair examined so far had zero covariance. Here is one that does not, worked end to end, so the cross term can be seen doing its work.

---

The setup. Draw one card from four, labelled $1, 2, 3, 4$ , each equally likely. Let

X = the number on the card, Z = {\begin{cases} 1 & if the number is 3 or 4 \\ 0 & otherwise. \end{cases}

$Z$ is an indicator for a high card. It is a function of $X$ , so knowing the card fixes it.

The marginals.

E [X] = \frac{1}{4} (1 + 2 + 3 + 4) = 2.5, Var (X) = \frac{1}{4} (1 + 4 + 9 + 16) - {2.5}^{2} = 7.5 - 6.25 = 1.25 .

$Z$ is 1 with probability $\frac{1}{2}$ , so $E [Z] = 0.5$ and $Var (Z) = E [Z^{2}] - E [Z]^{2} = 0.5 - 0.25 = 0.25$ , using $Z^{2} = Z$ for an indicator.

The covariance. $X Z$ takes the value $X$ when the card is high and 0 otherwise, so

E [X Z] = \frac{1}{4} (0 + 0 + 3 + 4) = 1.75, Cov (X, Z) = 1.75 - (2.5) (0.5) = 1.75 - 1.25 = 0.5 .

Positive, as expected: high cards are exactly when $Z$ is large, so the two deviations share a sign more often than not.

What that does to the sum. Let $W = X + Z$ . Adding the variances alone would give $1.25 + 0.25 = 1.5$ . The true value includes the cross term:

Var (W) = 1.25 + 0.25 + 2 (0.5) = 2.5 .

Verified directly. $W$ takes $1, 2, 4, 5$ each with probability $\frac{1}{4}$ , so $E [W] = 3$ and $E [W^{2}] = \frac{1}{4} (1 + 4 + 16 + 25) = 11.5$ , giving $Var (W) = 11.5 - 9 = 2.5$ .

Adding the variances would have understated the spread by 40%. The expectation, meanwhile, adds correctly without any condition: $E [W] = 2.5 + 0.5 = 3$ .

---

The same calculation with the sign flipped. Let $Z^{'} = 1 - Z$ , an indicator for a low card. Then $E [Z^{'}] = 0.5$ and $Var (Z^{'}) = 0.25$ , unchanged, but

Cov (X, Z^{'}) = - Cov (X, Z) = - 0.5,

so $Var (X + Z^{'}) = 1.25 + 0.25 - 1 = 0.5$ , less than either variance added naively would suggest. Checking: $X + Z^{'}$ takes $2, 3, 3, 4$ , with mean 3 and variance $\frac{1}{4} (1 + 0 + 0 + 1) = 0.5$ .

What the two cases show. The cross term is not a correction for sloppiness; it carries real information about the pair. When the variables move together the sum ranges further than the parts; when they move oppositely, the sum is tamer than either. Negative covariance is why a hedged portfolio can be steadier than either holding alone, and why a difference of two positively correlated measurements is less variable than a difference of independent ones.

Scale dependence, seen once. Measure the card in tens, $X^{*} = 10 X$ . Then $Cov (X^{*}, Z) = 5$ , ten times larger, though nothing about the relationship changed. The correlation is unmoved: $ρ = 0.5 / \sqrt{1.25 \times 0.25} \approx 0.894$ before and after. That invariance is why correlation is what gets reported when magnitudes are to be compared.

Contrast

Uncorrelated does not mean independent

Zero covariance is often treated as a synonym for independence. It is strictly weaker, and the gap matters wherever a claim of "no relationship" is made.

---

The pair. Let $X$ take the values $- 1, 0, 1$ each with probability $\frac{1}{3}$ , and let $Y = X^{2}$ .

$Y$ is a deterministic function of $X$ , knowing $X$ determines $Y$ exactly. No two variables could be more dependent.

Yet the covariance is zero.

E [X] = \frac{1}{3} (- 1 + 0 + 1) = 0, E [Y] = \frac{1}{3} (1 + 0 + 1) = \frac{2}{3},

E [X Y] = E [X^{3}] = \frac{1}{3} ((- 1)^{3} + 0 + 1^{3}) = 0,

Cov (X, Y) = E [X Y] - E [X] E [Y] = 0 - 0 \cdot \frac{2}{3} = 0 .

All four values verified exactly in rational arithmetic.

But independence fails plainly. Independence would require $P (X = 0, Y = 0) = P (X = 0) P (Y = 0)$ . The left side is $\frac{1}{3}$ , since $X = 0$ forces $Y = 0$ . The right side is $\frac{1}{3} \times \frac{1}{3} = \frac{1}{9}$ . Not equal, so $X$ and $Y$ are dependent.

Equivalently: $P (Y = 0) = \frac{1}{3}$ unconditionally, but $P (Y = 0 ∣ X = 0) = 1$ . Learning $X$ changes everything about $Y$ .

---

Why covariance missed it. Covariance measures linear association, whether $Y$ tends to rise as $X$ rises. Here $Y$ falls as $X$ goes from $- 1$ to $0$ and rises as $X$ goes from $0$ to $1$ . The relationship is perfect and symmetric, so the rising and falling halves cancel exactly, leaving zero.

Covariance does not report "no relationship". It reports no net linear tendency, which a symmetric non-linear relationship produces just as readily as no relationship at all.

---

The comparison.

	$X$ and $Y = X^{2}$	Two independent dice
$Cov$	$0$	$0$
$E [X Y] = E [X] E [Y]$	yes ( $0 = 0$ )	yes ( $\frac{49}{4}$ both)
$Var (X + Y) = Var X + Var Y$	yes	yes
$P (joint) = P \cdot P$	no ( $\frac{1}{3}$ vs $\frac{1}{9}$ )	yes
Independent	no	yes

The variance-addition rule holds in both columns, which is the practically important point. Uncorrelatedness is exactly what that rule needs; independence is more than required.

So the logical structure is:

independent ⟹ Cov = 0 ⟹ Var (X + Y) = Var (X) + Var (Y),

with neither implication reversible.

---

Where this bites. A reported correlation of zero between two variables licenses "no linear relationship", not "no relationship" and certainly not "independent". A U-shaped dose–response, harmful at low and high doses, beneficial in the middle, can show a correlation near zero while the dose determines the outcome entirely.

The converse error is rarer but worse: assuming independence from an observed zero correlation, then using it to justify something genuinely stronger, such as treating observations as independent draws when computing a standard error. Variance addition survives; most other consequences of independence do not.