Linear Regression

In a “level-level” regression specification:
$$y = beta_0 + beta_1x_1 + epsilon$$
The marginal effect of (x_1) on (y) is found by differentiating with respect to (x_1). So, (frac{dy}{dx} = beta_1)

In a “log-level” regression specification:
$$ log(y) = beta_0 + beta_1x_1 + epsilon $$
To get the marginal effect (frac{dy}{dx_1}), we have to first exponentiate both sides:
$$ y = exp(beta_0 + beta_1x_1 + epsilon) $$
Then, differentiate with respect to (x_1):

frac{dy}{dx} = beta_1 exp(beta_0 + beta_1 x_1 + epsilon)
= beta_1 y

Logistic Regression

First, assume that there is a probability of success of an event (p) that we would like to predict. We could try to predict this probability (p) using explanatory variables in a normal linear regression, like we did before (in fact, economists call this the “linear probability model”), but there would probably be many combinations of our various predictor variables that would result in predicted values for (p) that are outside of the range 0 – 1, and this would not make much sense. To deal with this, we *transform* the form of the equation twice to extend the range of the dependent variable that we are estimating, so that a linear combination of predictor variables will produce that range.

The first transformation is from probability (p) to odds (frac{p}{1-p}). Conceptually, odds is closely related to probability, but its range extends from 0 to positive infinity, instead of the 0 to 1 range of (p).

The second transformation is from odds to log odds. Taking the log of odds (log(frac{p}{1-p})) extends the range of 0 to positive infinity to the all real numbers.

Interpretation of the odds ratio and the effect on odds

The estimated coefficients for the logistic regression are the effect on the log odds of a success. This is not very interpretable. Let’s take some steps back towards probability. The first step brings us to the odds ratio. Exponentiating the estimated coefficient gives us the odds ratio. This is not to be confused with odds, that I explained above. The odds ratio is the proportionate change in the odds after a unit change in the predictor varible:
$$Delta mathrm{odds} = frac{mathrm{odds , after, a, unit, change ,in ,the ,predictor}}{mathrm{original , odds}} $$

The odds ratio is more interpretable for a dummy variable than it is for continuous varialbes. The reason why is because a unit change for a dummy variable corresponds to a specific state, and we can say for example that the odds of success for being/having state (x_1=1) are (exp(beta_1)) times being/having state (x_1=0). For example, let’s say (x_1=1) is a dummy variable for Green Party membership, and the event of success that we are trying to predict is whether or the individual installed solar panels. The estimated coefficient is (beta_1) (say, 1.19). The odds ratio is (exp(beta_1)) (3.3). The odds ratio (OR) is equivalent to the proportionate change in odds of solar panel installation given that the person belongs to the Green Party divided by the odds of installation given that the person DOES NOT belong to the Green Party. We can say “The odds of Green Party members installing solar panels is 3.3 times the odds of non-Green Party members installing solar panels.”

For continuous variables, a very simple calculation provides a more interpretable number: (100(exp(beta_1 – 1))) is the percent change in the odds for a 1-unit increase in (x_1). Say (x_1) is number of years of education, for example, and (beta_1) is estimated to be 0.44. Then for each additional year of education, a person’s odds of adopting solar panels increases by 55%.

What we all wish we could interpret simply: partial effect on probability

All right, assuming now that you know how to interpret the estimated coefficients from this model and the most commonly used effect on oddsinterpretation, I will show you how to calculate the partial effect of an explanatory variable on the probability of success, which is a bit more complicated, but utlimately what everyone wishes they could easily interpret from their logistic regression results.

We estimated the effect of (x_1) from (log(frac{p}{1-p}) = beta_0 + beta_1x_1). Exponentiating both sides, we get:

frac{p}{1-p} = exp(beta_0 + beta_1x_1)

We want to solve for (p), since our goal is an expression for the partial effect of (x_1) on (p):(frac{dp}{dx_1}). So, first multiply both sides by (1 -p) to get ((1-p)) out of the denominator, multiply out, and move the (p) term back to the left hand side, factor out (p), and divide. This should leave you with the following expression:
p = frac{exp(beta_0 + beta_1x_1)}{1 + exp(beta_0 + beta_1x_1)}
Now, we can differentiate with respect to (x_1). You need some calculus for this.

frac{dp}{dz} = frac{exp(beta_0 + beta_1x_1)}{[1 + exp(beta_0 + beta_1x_1)]^2} beta_1

Note that unlike the partial effects for (x_1) in linear regression, the partial effect of (x_1) on probability from a logistic regression is dependent on the value of (x_1). This is because of the non-linearity of the logistic function, which is a sigmoidal cumulative distribution function. Typically, people evaluate the partial effect at the mean values of the predictor variable, in this case (hat{x_1}).


Leave a Reply

Your email address will not be published. Required fields are marked *