The Model

We’ll start by being a bit more precise about the setup. Suppose that \(Y\) is related to \(X\) according to the following linear causal model \[ Y \leftarrow \alpha + \beta X + U \] where \(\beta\) is the causal effect of interest and \(U\) represents unobserved causes of \(Y\) that may be related to \(X\). Now, for any observed random variable \(Z\), we can define \[ V \equiv X - (\pi_0 + \pi_1 Z), \quad \pi_0 \equiv \mathbb{E}[X] - \pi_1 \mathbb{E}[Z], \quad \pi_1 \equiv \frac{\text{Cov}(X,Z)}{\text{Var}(Z)}. \] This is the population linear regression of \(X\) on \(Z\). By construction it satisfies \(\mathbb{E}[V] = \text{Cov}(Z,V) = 0\).² Thus we can write, \[ X = \pi_0 + \pi_1 Z + V, \quad \mathbb{E}[V] = \text{Cov}(Z,V) = 0 \] for any random variables \(X\) and \(Z\), simply by constructing \(V\) as described above. If \(\pi_1 \neq 0\), we say that \(Z\) is relevant. If \(\text{Cov}(Z,U) = 0\), we say that \(Z\) is exogenous. If \(Z\) is both relevant and exogenous, we say that it is a valid instrument for \(X\).

As we’ve defined it above, \(V\) is simply a regression residual. But if \(Z\) is a valid instrument, it turns out that we can think of \(V\) as the “endogenous part” of \(X\). To see why, expand \(\text{Cov}(X,U)\) as follows: \[ \text{Cov}(X,U) = \text{Cov}(\pi_0 + \pi_1 Z + V, \,U) = \pi_1 \text{Cov}(Z,U) + \text{Cov}(U,V) = \text{Cov}(U,V) \] since we have assumed that \(\text{Cov}(Z,U) = 0\). In words, the endogeneity of \(X\) is precisely the same thing as the covariance between \(U\) and \(V\).

Here’s a helpful way of thinking about this. If \(Z\) is exogenous then our regression of \(X\) on \(Z\) partitions the overall variation in \(X\) into two components: the “good” (exogenous) variation \(\pi_1 Z\) is uncorrelated with \(U\), while the “bad” (endogenous) variation \(V\) is correlated with \(U\). The logic of two-stage least squares is that regressing \(Y\) on the “good” variation, \(\pi_1 Z\) allows us to recover \(\beta\), the causal effect of interest.³

A Simulation Example

Using the model and derivations from above, let’s run a little simulation. To simulate a valid instrument \(Z\) and an endogenous regressor \(X\) we can proceed as follows. First generate independent standard normal draws \(\{Z_i\}_{i=1}^n\). Next independently generate pairs of correlated standard normal draws \(\{(U_i, V_i)\}_{i=1}^n\) with \(\text{Corr}(U_i, V_i) = \rho\). Finally, set \[ X_i = \pi_0 + \pi_1 Z_i + V_i \quad \text{and} \quad Y_i = \alpha + \beta X_i + U_i \] for each value of \(i\) between \(1\) and \(n\).⁴ The following chunk of R code runs this simulation with \(n = 5000\), \(\rho = 0.5\), \(\pi_0 = 0.5\), \(\pi_1 = 0.8\), \(\alpha = -0.3\) and \(\beta = 1\):

set.seed(1234)
n <- 5000
z <- rnorm(n)

library(mvtnorm)
Rho <- matrix(c(1, 0.5, 
                0.5, 1), 2, 2, byrow = TRUE)
errors <- rmvnorm(n, sigma = Rho)

u <- errors[, 1]
v <- errors[, 2]
x <- 0.5 + 0.8 * z + v
y <- -0.3 + x + u

In the simulation \(Z\) is a valid instrument, \(X\) is an endogenous regressor, and the true causal effect of interest equals one. Using our simulation data, let’s test out three possible estimators:

• \(\widehat{\beta}_\text{OLS}\equiv\) the slope coefficient from an OLS regression of \(Y\) on \(X\).
• \(\widehat{\beta}_\text{IV}\equiv\) slope coefficient from an IV regression of \(Y\) on \(X\) with \(Z\) as an instrument.
• \(\widehat{\beta}_{X.Z}\equiv\) the coefficient on \(X\) in an OLS regression of \(Y\) on \(X\) and \(Z\).

c(truth = 1,
  b_OLS = cov(x, y) / var(x), 
  b_IV = cov(z, y) / cov(z, x), 
  b_x.z = unname(coef(lm(y ~ x + z))[2])) |> # unname() makes the names prettier!
  round(2)

## truth b_OLS  b_IV b_x.z 
##  1.00  1.31  1.01  1.49

As expected, OLS is far from the truth while IV pretty much nails it. Interestingly, the regression of y on x and z gives the worst performance of all! Is this just a fluke? Perhaps it’s an artifact of the simulation parameters I chose, or just bad luck arising from some unusual simulation draws. To find out, we’ll need a bit more algebra. But stay with me: the payoff is worth it, and there’s not too much extra math required!

The General Result

Some Intuition

In our simulation, \(\widehat{\beta}_{X.Z}\) gave a worse estimate of \(\beta\) than \(\widehat{\beta}_{X.Z}\). The derivations from above show that this wasn’t a fluke: adding a valid instrument \(Z\) as an additional control regressor only makes the bias in our estimated causal effect worse than it was to begin with. This holds for any valid instrument and any endogenous regressor in a linear causal model. I hope you found the derivations from above convincing. But even so, you may be wondering if there’s an intuitive explanation for this phenomenon. I am please to inform you that the answer is yes!

In an earlier post I described the control function approach to instrumental variables regression. That post showed that the coefficient on \(X\) in a regression of \(Y\) on \(X\) and \(V\) gives the correct causal effect. We don’t know \(V\), but we can estimate it by regressing \(X\) on \(Z\) and saving the residuals. The logic of multiple regression shows that including \(V\) as a control regressor “soaks up” the portion of \(X\) that is explained by \(V\). Because \(V\) represents the “bad” (endogenous) variation in \(X\), this solves our endogeneity problem. In effect, \(V\) captures the unobserved “omitted variables” that play havoc with a naive regression of \(Y\) on \(X\).

Now, contrast this with a regression of \(Y\) on \(X\) and \(Z\). In this case, we soak up the variation in \(X\) that is explained by \(Z\). But \(Z\) represents the good (exogenous) variation in \(X\)! Soaking up this variation leaves only the bad variation behind, making our endogeneity problem worse than it was to begin with. In this example, \(Z\) is what is known as a bad control, a control regressor that makes things worse rather than better. A common piece of advice for avoiding bad controls is to only include control regressors that are correlated with \(X\) and \(Y\) but are not themselves caused by \(Z\). The example in this post shows that this advice wrong. Here \(Z\) is not caused by \(X\), and is correlated with both \(X\) and \(Y\). Nevertheless, it is a bad control In short, a valid instrument provides a powerful way to carry out causal inference from observational data, but only if you use it in the right way. A good instrument is a bad control!