Causal Inference with Linear Regression: Omitted variables and Irrelevant variables

In my previous article, Causal Inference: Econometric Models vs. A/B Testing, we discuss how to use an econometric model, namely, linear regression to investigate the causal relationship between the treatment variable and the response variable while controlling other covariates. In this article, we'll discuss some common issues when designing a linear regression - Omitting Important Variables and Including Irrelevant Variables.

Before we discuss these issues, we need to get ourselves familiar with the bias and variance of coefficient estimates.

• The bias measures the difference between the fitted value and the true value of estimates. If a treatment effect from linear regression is biased, it means we have an inaccurate causal effect.

• The variance measures the spread of the estimates (which are random variables) around their expected values. The higher the variance, the less the precision of estimates.

What happens if Omitting Important Variables?

We should all know that we shouldn't omit important variables in linear regression. The consequence of such action would make the model incapable to explain the response variable correctly (a.k.a., Under-fitting) and might make a wrong causal inference statement.

Let's dive deeper and investigate how badly it could damage our model.

The simplest linear regression is shown as follows. The response variable (i.e., Y) can be explained as a linear combination of explanatory variables (e.g., the intercept, X1, X2, X3, ...) and ε is the error term that represents the difference between the fitted response value and the actual response value. The normality assumption of the error term is optional for a linear regression model, but is recommended for the task of causal inference.

Depending on how you would like to set up the cost function, we can drive the closed-form solution through either Ordinary Least Squares (OLS) or Maximum Likelihood Estimation (MLE).

Next, let's rewrite the equation in Figure 1 by decomposing the explanatory variables into the treatment variable (i.e., T) and other explanatory variables (i.e., X) in the model, so that it is easier to investigate how badly omitting important variables would damage coefficient estimator (i.e., α) of the treatment variable.

From the second equation in Figure 3 and Figure 2, we can express the fitted value of "α_hat" as follows,

Let's say that we found out an important variable (i.e., Z) was omitted in the linear regression model in Figure 3. The correct model should be

where γ is the coefficient for the omitted variable, Z.

To investigate how badly we've messed up the coefficient estimate of the treatment variable in Figure 4, we will substitute Y in Figure 4 with the correct model from Figure 5. Now we have

The first term in Figure 6 can be simplified as α because

The third term in Figure 6 should equal to 0, because the error term should be independent of explanatory variables by assumption when we set up the linear model. Moreover, the expected value of the third term would also be 0 because the expected value of the error term is 0 by assumption as well.

Therefore, the equation in Figure 6 can be simplified as follows:

Scenario 1: The omitted variable Z is correlated with the treatment variable T. We call this kind of variable a Confounding Variable because they are correlated to both the response variable and the treatment variable.

When the omitted variable Z is correlated with the treatment variable T and can explain the response variable meaningfully, then the second term in Figure 8 is no longer 0. Therefore, the OLS estimator for the treatment effect is no longer unbiased and we would likely make a wrong causal inference statement if omitting variable Z.

If confounding variable Z is omitted in a linear regression model, then the treatment variable would become endogenous because the "unexplained" variable Z leaks into the error term, then the treatment variable will be correlated with the error term. In this case, the estimates of the treatment variable would become biased (i.e., endogeneity bias).

Scenario 2: The omitted variable Z is NOT correlated with the treatment variable T.

When the omitted variable is uncorrelated with the treatment variable in the regression model, the second term in Figure 8 will be 0. Therefore, the OLS estimator for the treatment effect continues to be unbiased.

However, there is still a price to pay if omitting such variables. Even though omitted variable Z is not correlated ** with the treatment variable T, variable Z still plays a role to explain the response variable, excluding variable Z would group this "unexplained" portion into the error term ε and make the variance of the errors larger. As a result, the variance** of all the estimators (including the treatment effect) would be larger (see Figure 12).

In other words, although the treatment effect is still unbiased if the omitted variable is uncorrelated with the treatment variable, the variance of the treatment effect would become larger, the t-value of hypothesis testing would be smaller, then the p-value would be larger, therefore, we might incorrectly conclude that the treatment effect is statistically insignificant (i.e., false negative).

What happens if Including Irrelevant Variables?

It seems that the issue of omitted variables can be easily addressed by including all the relevant variables in a linear regression model. But including irrelevant variables in the model could lead to other problems.

From the Jeffrey Wooldridge's textbook, Introductory Econometrics, under Gauss-Markov assumptions, conditional on the sample values of the independent variables, we can rewrite the variance formula (in Figure 12) as follows:

Where j represents a specific explanatory variable j. SST_j is the total sample variation of explanatory variable j,

R2_j is the coefficient of determination from a regression of predictor j on the remaining predictors, with predictor j on the left-hand side, and all other predictors on the right-hand side.

The following term is called the variance inflation factor (VIF). It is a useful tool to analyze the magnitude of multicollinearity in a linear regression model, in which predictors are correlated with each other.

• VIF_j equals 1 when the predictor j is NOT correlated to other predictors.

• VIF_j would become larger than 1 when predictor j can be explained by other predictors.

• A rule of thumb is that if VIF_j is greater than 10, then multicollinearity is high.

Now let's consider the following scenarios:

Scenario 1: From the previous section (Figure 8), we know that if a variable is highly correlated with the treatment variable, including such a variable in the linear regression model will very likely mask the true causal effect of the treatment variable (i.e., high bias). It's clear that we will need to exclude such a variable.

Scenario 2: If a variable is not correlated with the treatment variable, but highly correlated with the rest of the explanatory variables (i.e., multicollinearity). Then including this kind of variable will not increase the bias and variance of the treatment effect (see Figures 8 and 13). However, it would increase both the bias and variance of other explanatory variables, when the variable doesn't explain the variation of the response variable (i.e., the variable doesn't reduce the numerator, but reduce the denominator in Figure 13. This is multicollinearity at its worst). In this case, we will exclude such a variable.

Scenario 3: This scenario is similar to scenario 2, except this variable also explains the variation of the response variable. In this scenario, if we're more concerned about the issue of an omitted variable than multicollinearity, then we can keep it in the model and live with multicollinearity. We might have larger or smaller standard errors (See Figure 13, although the denominator becomes smaller with this added variable, the numerator could also become smaller if the added variable also helps to explain the response variable). In this case, we need to consider the tradeoff between the added explanatory power and multicollinearity.

Scenario 4: If variables are not correlated to both explanatory variables and the response variable, it should matter too much (in terms of bias and variance) whether you add them or omit them. However, if a lot of such variables are added to the model, it will start to decrease the degrees of freedom in the model, then increase the variance of estimates (See Figure 12).

Final Notes

In conclusion, if an omitted variable or an irrelevant variable is correlated with the treatment variable, then the treatment effect will become biased. Including irrelevant variables that are correlated with existing predictors will increase the variance of estimates and make estimates and predictions less precise.

Here are some related posts you can explore if you're interested in Linear Regression and Causal Inference.

• Causal Inference: Econometric Models vs. A/B Testing

• Linear Regression vs. Logistic Regression: OLS, Maximum Likelihood Estimation, Gradient Descent

• Linear Regression with OLS: Unbiased, Consistent, BLUE, Best (Efficient) Estimator

• Causal Inference with Linear Regression: Omitted variables and Irrelevant variables

• Causal Inference with Linear Regression: Endogeneity

• Linear Regression with OLS: Heteroskedasticity and Autocorrelation

Thank you for reading !!!

If you enjoy this article and would like to Buy Me a Coffee, please click here.

You can sign up for a membership to unlock full access to my articles, and have unlimited access to everything on Medium. Please subscribe if you'd like to get an email notification whenever I post a new article.