(Mis)understanding Selection on Observables

Last updated on Aug 8, 2021 6 min read treatment effects

On a recent exam I asked students to extend the logic of propensity score weighting to handle a treatment that takes on three rather than two values: basically a stripped-down version of Imbens (2000). Nearly everyone figured this out without much trouble, which is good news! At the same time, I noticed some common misconceptions about the all-important selection-on-observables assumption: $E [Y_{0} | D, X] = E [Y_{0} | X] and E [Y_{1} | D, X] = E [Y_{1} | X]$ where $(Y_{0}, Y_{1})$ are the potential outcomes corresponding to a binary treatment $D$ and $X$ is a vector of observed covariates.¹ Since more than a handful of students made the same mistakes, it seemed like a good opportunity for a short post.

Two Misconceptions

The following two statements about selection on observables are false:

Under selection on observables, if I know the value of someone’s covariate vector $X$ , then learning her treatment status $D$ provides no additional information about the average value of her observed outcome $Y$ .
Selection on observables requires the treatment $D$ and potential outcomes $(Y_{0}, Y_{1})$ to be conditionally independent given covariates $X$ .

If you’ve studied treatment effects, pause for a moment and see if you can figure out what’s wrong with each of them before reading further.

The First Misconception

The first statement:

Under selection on observables, if I know the value of someone’s covariate vector $X$ , then learning her treatment status $D$ provides no additional information about the average value of her observed outcome $Y$ .

is a verbal description of the following conditional mean independence condition: $E [Y | X, D] = E [Y | X] .$ So what’s wrong with this equality? The potential outcomes $(Y_{0}, Y_{1})$ and the observed outcome $Y$ are related according to $Y = Y_{0} + D (Y_{1} - Y_{0}) .$ Taking conditional expectations of both sides and using the selection on observables assumption $\begin{aligned} E [Y | X, D] & = E [Y_{0} | X, D] + D E [Y_{1} - Y_{0} | D, X] \\ = E [Y_{0} | X] + D E [Y_{1} - Y_{0} | X] . \end{aligned}$ In contrast, conditioning on $X$ alone gives $\begin{aligned} E [Y | X] & = E [Y_{0} | X] + E [D (Y_{1} - Y_{0}) | X] \\ = E [Y_{0} | X] + E_{D | X} [D E (Y_{1} - Y_{0} | D, X)] \\ = E [Y_{0} | X] + E_{D | X} [D E (Y_{1} - Y_{0} | X)] \\ = E [Y_{0} | X] + E (D | X) \cdot E (Y_{1} - Y_{0} | X) \end{aligned}$ by iterated expectations and the selection on observables assumption, since $E (Y_{1} - Y_{0} | X)$ is a measurable function of $X$ . Subtracting these expressions, we find that $E (Y | X, D) - E (Y | X) = [D - E (D | X)] \cdot E (Y_{1} - Y_{0} | X)$ so that $E (Y | X, D) = E (Y | X)$ if and only if the RHS equals zero.

So how could the RHS equal zero? One way is if $D = E (D | X)$ . Since $D$ is a binary random variable, this would require $E (D | X)$ to be a binary random variable as well. But notice that $E (D | X) = P (D = 1 | X)$ is simply the propensity score $p (X)$ . Because $X$ is a random variable, so is $p (X)$ . But $p (X)$ cannot take on the values zero or one. If it did, this would violate the overlap assumption: $0 < p (X) < 1$ .

So we can’t have $D = E (D | X)$ , but what about $E (Y_{1} - Y_{0} | X) = 0$ ? Since $(Y_{1} - Y_{0})$ is the treatment effect of $D$ , it follows that $E (Y_{1} - Y_{0} | X)$ is the conditional average treatment effect $ATE (X)$ given $X$ . It’s not a contradiction for $ATE (X)$ to equal zero, but think about what it would mean: it would require that the average treatment effect for a person with covariates $(X = x)$ is exactly zero regardless of $x$ . Moreover, by iterated expectations it would imply that $ATE = E (Y_{1} - Y_{0}) = E_{X} [E (Y_{1} - Y_{0} | X)] = E [ATE (X)] = 0$ so the average treatment effect would also be zero. Again, this is not a contradiction but it would definitely be odd to assume that the treatment effect is zero before you even try to estimate it!

To summarize: the first statement above cannot be an implication of selection on observables because it would either require a violation of the overlap assumption, or imply that there is no treatment effect whatsoever. To correct the statement, we simply need to change the last three words:

Under selection on observables, if I know the value of someone’s covariate vector $X$ , then learning her treatment status $D$ provides no additional information about the average values of her potential outcomes $(Y_{0}, Y_{1})$ .

This is a correct verbal statement of the mean exclusion restriction $E (Y_{0} | D, X) = E (Y_{0} | X)$ and $E (Y_{1} | D, X) = E (Y_{1} | X)$ .

The Second Misconception

And this leads nicely to the second misconception:

Selection on observables requires the treatment $D$ and potential outcomes $(Y_{0}, Y_{1})$ to be conditionally independent given covariates $X$ .

To see why this is false, consider an example in which $\begin{aligned} Y & = (1 - D) \cdot (α_{0} + X^{'} β_{0} + U_{0}) + D \cdot (α_{1} + X^{'} β_{1} + U_{1}) \\ U_{0} | (D, X) & \sim Normal (0, 1 - D / 2) \\ U_{1} | (D, X) & \sim Normal (0, 1 + D) . \end{aligned}$ Notice that the distributions of $U_{0}$ and $U_{1}$ given $(D, X)$ depend on $D$ . Now, by iterated expectations, $\begin{aligned} E (U_{0} | X) & = E_{(D | X)} [E (U_{0} | D, X)] = 0 \\ E (U_{0}) & = E_{X} [E (U_{0} | X)] = 0 \end{aligned}$ and similarly $E (U_{1} | X) = E (U_{1}) = 0$ . Substituting $D = 0$ and $D = 1$ , we can calculate the potential outcomes and average treatment effect as follows $\begin{aligned} Y_{0} & = α_{0} + X^{'} β_{0} + U_{0} \\ Y_{1} & = α_{1} + X^{'} β_{1} + U_{1} \\ ATE & = E (Y_{1} - Y_{0}) = (α_{1} - α_{0}) + E [X^{'}] (β_{1} - β_{0}) . \end{aligned}$ It follows that $D$ is not conditionally independent of $(Y_{0}, Y_{1})$ given $X$ . In particular, the variance of the potential outcomes depends on $D$ even after conditioning on $X$ : $\begin{aligned} Var (Y_{0} | X, D) & = Var (U_{0} | X, D) = 1 - D / 2 \\ Var (Y_{1} | X, D) & = Var (U_{1} | X, D) = 1 + D . \end{aligned}$ In spite of this, the selection on observables assumption still holds: $\begin{aligned} E (Y_{0} | D, X) & = α_{0} + X^{'} β_{0} + E (U_{0} | D, X) = α_{0} + X^{'} β_{0} \\ E (Y_{0} | X) & = α_{0} + X^{'} β_{0} + E (U_{0} | X) = α_{0} + X^{'} β_{0} \end{aligned}$ and similarly $E (Y_{1} | D, X) = E (Y_{1} | X) = α_{1} + X^{'} β_{0}$ . While this example is admittedly a bit peculiar, the point is more general: because the average treatment effect is an expectation, identifying it only requires assumptions about conditional means.² The second statement is even easier to correct than the first: we need only add a single word:

Selection on observables requires the treatment $D$ and potential outcomes $(Y_{0}, Y_{1})$ to be conditionally mean independent given covariates $X$ .

Conditional independence implies conditional mean independence, but the converse is false.

Epilogue

So what’s the moral here? First, it’s crucial to distinguish between the observed outcome $Y$ and the potential outcomes $(Y_{0}, Y_{1})$ . Second, the various notions of “unrelatedness” between random variables—independence, conditional mean independence, and uncorrelatedness—can be confusing. Be sure to pay attention to exactly which condition is used and why. In a future post, I’ll have more to say about the relationships between these notions.

For more details see my lecture notes on treatment effects ↩︎
You might object that in the real world it is difficult to think of settings in which conditional mean independence is plausible but full independence does not. This is a fair point. Nevertheless, it’s important to be clear about which assumptions are actually used in a given derivation, and here we only rely on conditional mean independence.↩︎