Why Econometrics is Confusing Part II: The Independence Zoo

Example #1 - Discrete RVs \((X,Y)\)

My first example involves two discrete random variables \(X\) and \(Y\) with joint probability mass function \(p_{XY}(x,y)\) given by

Even without doing any math, we see that knowing \(X\) conveys information about \(Y\), and vice-versa. For example, if \(X = -1\) then we know that \(Y\) must equal zero. Similarly, if \(Y=1\) then \(X\) must equal zero. Spend a bit of time thinking about this joint distribution before reading further. We’ll have plenty of time for mathematics below, but it’s always worth seeing where our intuition takes us before calculating everything.

To streamline our discussion below, it will be helpful to work out a few basic results about \(X\) and \(Y\). A quick calculation with \(p_{XY}\) shows that \[ \mathbb{E}(XY) \equiv \sum_{\text{all } x} \sum_{\text{all } y}= x y \cdot p_{XY}(x,y) = 0. \] Calculating the marginal pmfs for \(X\) we see that \[ p_X(-1) = p_X(0) = p_X(1) = 1/3 \implies \mathbb{E}(X) \equiv \sum_{\text{all } x} x \cdot p_X(x) = 0. \] Similarly, calculating the marginal pmf of \(Y\), we obtain \[ p_Y(0) = 2/3,\, p_Y(1) = 1/3 \implies \mathbb{E}(Y) \equiv \sum_{\text{all } y} p_Y(y) = 1/3. \] We’ll use these results as ingredients below as we explain and relate three key notions of dependence: correlation, conditional mean independence, and statistical independence.

Example #2 - Continuous RVs \((W,Z)\)

My second example concerns two continuous random variables \(W\) and \(Z\), where \(W \sim \text{Uniform}(-1, 1)\) and \(Z = W^2\). In this example, \(W\) and \(Z\) are very strongly related: if I tell you that the realization of \(W\) is \(w\), then you know for sure that the realization of \(Z\) must be \(w^2\). Again, keep this intuition in mind as we work through the mathematics below.

In the remainder of the post, we’ll find it helpful to refer to a few properties of \(W\) and \(Z\), namely \[ \begin{aligned} \mathbb{E}[W] &\equiv \int_{-\infty}^\infty w\cdot f_W(w)\, dw = \int_{-1}^1 w\cdot \frac{1}{2}\,dw = \left. \frac{w^2}{4}\right|_{-1}^1 = 0\\ \mathbb{E}[Z] &\equiv \mathbb{E}[W^2] = \int_{-\infty}^{\infty} w^2 \cdot f_W(w)\, dw = \int_{-1}^1 w^2 \cdot \frac{1}{2} \, dw = \left. \frac{w^3}{6}\right|_{-1}^1 = \frac{1}{3}\\ \mathbb{E}[WZ] &= \mathbb{E}[W^3] \equiv \int_{-\infty}^\infty w^3 \cdot f_W(w)\, dw =\int_{-1}^1 w^3 \cdot \frac{1}{2}\, dw = \left. \frac{w^4}{8} \right|_{-1}^1 = 0. \end{aligned} \] Since \(W\) is uniform on the interval \([-1,1]\), its pdf is simply \(1/2\) on this interval, and zero otherwise. All else equal, I prefer easy integration problems!

Example #1: \(X\) and \(Y\) are uncorrelated.

In Example #1 from above, \(\mathbb{E}[XY]=0\) and \(\mathbb{E}(X)\mathbb{E}(Y) = 0 \times 1/3 = 0\) so \(X\) and \(Y\) are uncorrelated. Lack of correlation is one possible way in which two random variables can be thought of as “unrelated.” But it is a relatively weak property. Indeed, \(X\) and \(Y\) are in fact highly dependent in Example #1. For example, if \(X=-1\) then we know for sure that \(Y=0\). I simply cooked up the numbers to ensure that \(\mathbb{E}[XY]=\mathbb{E}[X]\mathbb{E}[Y]\) in spite of this.

Example #2: \(W\) and \(Z\) are uncorrelated.

Because Example #1 is discrete, it can be a bit tricky to think about what it would mean for a dependence relationship to be nonlinear. Here Example #2 can help. As mentioned above, there is clearly a relationship between \(Z\) and \(W\). But this relationship is nonlinear in that \(Z\) is a quadratic function of \(X\). Since \(\mathbb{E}(WZ) = 0\) and \(\mathbb{E}(W) \times \mathbb{E}(Z) = 0 \times \mathbb{E}(Z) = 0\), we see that \(W\) and \(Z\) are uncorrelated. Another way to see this is by simulating some data with the same properties as Example #2

set.seed(1983)
n_sims <- 250
w <- runif(n_sims, -1, 1)
z <- w^2
cor(w, z)

## [1] 0.008379101

plot(w, z)
abline(lm(z ~ w))

The regression line is flat despite there being an obvious relationship between \(W\) and \(Z\). When \(W\) is positive, there is a positive relationship between the two RVs; but when \(W\) is negative the picture is reverses. The line of best fit “averages out” the increasing and decreasing relationships on either side of zero to give an overall slope of zero.²

Example #1: \(X\) is mean independent of \(Y\).

Using the table of joint probabilities for Example #1 above, we found that \(\mathbb{E}(X) = 0\). To determine whether \(X\) is mean independent of \(Y\), we need to calculate \(\mathbb{E}(X|Y=y)\), which we can accomplish as follows: \[ \begin{aligned} \mathbb{E}(X|y=0) &= \sum_{\text{all } x} x \cdot \mathbb{P}(X=x|Y=0) = \sum_{\text{all } x} x \cdot \frac{\mathbb{P}(X=x,Y=0)}{\mathbb{P}(Y=0)}\\ \\ \mathbb{E}(X|y=1) &= \sum_{\text{all } x} x \cdot \mathbb{P}(X=x|Y=1) = \sum_{\text{all } x} x \cdot \frac{\mathbb{P}(X=x,Y=1)}{\mathbb{P}(Y=1)}. \end{aligned} \] Substituting the joint and marginal probabilities from the table above, we find that \[ \mathbb{E}(X|Y=0) = 0, \quad \mathbb{E}(X|Y=1) = 0. \] Thus \(\mathbb{E}(X|Y=y)\) simply equals zero, regardless of the realization \(y\) of \(Y\). Since \(\mathbb{E}(X) = 0\) we have shown that \(X\) is conditionally mean independent of \(X\).

Example #1: \(Y\) is NOT mean independent of \(X\).

To determine whether \(Y\) is mean independent of \(X\) we need to calculate \(\mathbb{E}(Y|X)\). But this is easy. From the table we see that \(Y\) is known with certainty after we observe \(X\): if \(X = -1\) then \(Y = 0\), if \(X = 0\) then \(Y = 1\), and if \(X = 1\) then \(Y = 0\). Thus, without doing any math at all we find that \[ \mathbb{E}(Y|X=-1) = 0, \quad \mathbb{E}(Y|X=0) = 1, \quad \mathbb{E}(Y|X=1) = 0. \] (If you don’t believe me, work through the arithmetic yourself!) This clearly depends on \(X\), so \(Y\) is not mean independent of \(X\).

Example #2: \(Z\) is NOT mean independent of \(W\).

Above we calculated that \(\mathbb{E}(Z) = \mathbb{E}(W^2) = 1/3\). But the conditional expectation is \[ \mathbb{E}(Z|W) = \mathbb{E}(W^2|W) = W^2 \] using the “taking out what is known” property: conditional on \(W\), we know \(W^2\) and can hence treat it as though it were a constant in an unconditional expectation, pulling it in front of the \(\mathbb{E}\) operator. We see that \(\mathbb{E}(Z|W)\) does not equal \(1/3\): its value depends on \(W\). Therefore \(Z\) is not mean independent of \(W\).

Example #2: \(W\) is mean independent of \(Z\).

This one is trickier. To keep this post at an elementary level, my explanation won’t be completely rigorous. For more details see here. We need to calculate \(\mathbb{E}(W|Z)\). Since \(Z \equiv W^2\) this is the same thing as \(\mathbb{E}(W|W^2)\). Let’s start with an example. Suppose we observe \(Z = 1\). This means that \(W^2 = 1\) so \(W\) either equals \(1\) or \(-1\). How likely is each of these possible realizations of \(W\) given that \(W^2 = 1\)? Because the density of \(W\) is symmetric about zero, \(f_W(-1) = f_W(1)\). So given that \(W^2 = 1\), it is just as likely that \(W = 1\) as it is that \(W = -1\). Therefore, \[ \mathbb{E}(W|W^2 = 1) = 0.5 \times 1 + 0.5 \times -1 = 0. \] Generalizing this idea, if we observe \(Z = z\) then \(W = \sqrt{z}\) or \(-\sqrt{z}\). But since \(f_W(\cdot)\) is symmetric about zero, these possibilities are equally likely. Therefore, \[ \mathbb{E}(W|Z=z) = 0.5 \times \sqrt{z} - 0.5 \times \sqrt{z} = 0. \] Above we calculated that \(\mathbb{E}(W) = 0\). Therefore, \(W\) is mean independent of \(Z\).

Example #1: \(X\) and \(Y\) are NOT independent.

If I tell you that \(X = 0\), then you know for sure that \(Y = 0\). Before I told you this, you did not know that \(Y\) would equal zero: it’s a random variable with support set \(\{0,1\}\). Since learning \(X\) has the potential to tell you something about \(Y\), \(X\) and \(Y\) are not independent. That was easy! For extra credit, \(p_{XY}(-1,0) = 1/3\) but \(p_X(-1)p_Y(0) = 1/3 \times 2/3 = 2/9\). Since these are not equal, \(p_{XY}\neq p_X p_Y\) so the marginal doesn’t equal the product of the joint. We didn’t need to check this, but it’s reassuring to see that everything works out as it should.

Example #2: \(W\) and \(Z\) are NOT independent.

Again, this one is easy: learning that \(W = w\) tells us that \(Z = w^2\). We didn’t know this before, so \(W\) and \(Z\) cannot be independent.

Uncorrelatedness and Statistical Independence are Symmetric

In the equality \(\mathbb{E}(XY) = \mathbb{E}(X) \mathbb{E}(Y)\), nothing changes if we swap the roles of \(X\) and \(Y\); this statement is equivalent to \(\mathbb{E}(YX) = \mathbb{E}(Y) \mathbb{E}(X)\). This shows that uncorrelatedness is symmetric. The same goes for statistical independence: we showed that \(p_{Y|X} = p_Y\) is equivalent to \(p_{X|Y} = p_X\) above. In contrast, mean independence is not symmetric: \(X\) can be mean independent of \(Y\) without \(Y\) being mean independent of \(X\).

Here’s an analogy: uncorrelatedness and independence are like the relation “being biological siblings.” If \(X\) is the sibling of \(Y\), then \(Y\) must be the sibling of \(X\) because “being siblings” is defined as “having the same parents.” In contrast, mean independence is like the relation “being in love.” Sadly, it’s possible for \(X\) to be in love with \(Y\) despite \(Y\) not being in love with \(X\).⁴

Statistical Independence Implies Conditional Mean Independence

Statistical independence is the “strongest” of the three properties: it implies both mean independence and uncorrelatedness. We’ll show this in two steps. In the first step, we’ll show that statistical independence implies mean independence. In the second step we’ll show that mean independence implies uncorrelatedness. Then we’ll bring this overly-long blog post to a close! Suppose that \(X\) and \(Y\) are discrete random variables. (For the continuous case, replace sums with integrals.) If \(X\) is statistically independent of \(Y\), then \(p_{Y|X} = p_Y\) and \(p_{X|Y} = p_X\). Hence, \[ \begin{aligned} \mathbb{E}(Y|X=x) &\equiv \sum_{\text{all } y} y \cdot p_{Y|X}(y|x) = \sum_{\text{all } y} y \cdot p_Y(y) \equiv \mathbb{E}(Y)\\ \mathbb{E}(X|Y=y) &\equiv \sum_{\text{all } x} x \cdot p_{X|Y}(x|y) = \sum_{\text{all } x} x \cdot p_X(x) \equiv \mathbb{E}(X) \end{aligned} \] so \(Y\) is mean independent of \(X\) and \(X\) is mean independent of \(Y\).

Conditional Mean Independence Implies Uncorrelatedness

If either \(\mathbb{E}(Y|X) = \mathbb{E}(Y)\) or \(\mathbb{E}(X|Y) = \mathbb{E}(X)\), then \(X\) and \(Y\) are uncorrelated. To show this, we use the Law of Iterated Expectations and the “taking out what is known” property, along with the fact that \(\mathbb{E}(X)\) and \(\mathbb{E}(Y)\) are constants. Suppose first that \(Y\) is mean independent of \(X\), i.e. \(\mathbb{E}(Y|X) = \mathbb{E}(Y)\). Then, taking iterated expectations over \(X\), \[ \mathbb{E}(XY) = \mathbb{E}[\mathbb{E}(XY|X)] = \mathbb{E}[X \mathbb{E}(Y|X)] = \mathbb{E}[X \mathbb{E}(Y)] = \mathbb{E}(X) \mathbb{E}(Y). \] Alternatively, suppose that \(X\) is mean independent of \(Y\), i.e. \(\mathbb{E}(X|Y) = \mathbb{E}(X)\). Then, taking iterated expectations over \(Y\), \[ \mathbb{E}(XY) = \mathbb{E}[\mathbb{E}(XY|Y)] = \mathbb{E}[Y\mathbb{E}(X|Y)] = \mathbb{E}[Y \mathbb{E}(X)] = \mathbb{E}(Y) \mathbb{E}(X). \] Therefore, if either \(X\) is mean independent of \(Y\), or \(Y\) is mean independent of \(X\), or both, then \(X\) and \(Y\) are uncorrelated. Since statistical independence implies mean independence, it follows that statistical independence implies uncorrelatedness. And we’re finally done!