(Intercept) education
-16.45 2.90 4 Linear Regression
Section 2 introduced the conditional expectation function (CEF) and its exact residual.
Section 3 showed how population moments can be estimated by sample moments.
We now combine these ideas to specify and estimate linear regression models.
4.1 From the CEF to Simple Linear Regression
Let Y_i denote hourly wage and let Z_i denote years of education.
The population CEF is m_Z(z)=E[Y_i\mid Z_i=z].
Section 2 defined the CEF residual U_i=Y_i-m_Z(Z_i).
It gives the exact decomposition Y_i=m_Z(Z_i)+U_i, \qquad E[U_i\mid Z_i]=0.
Suppose the CEF is a straight line: m_Z(z)=\beta_1+\beta_2z. \tag{4.1}
NoteSimple linear regression model
When the CEF is linear, the exact decomposition is Y_i=\beta_1+\beta_2Z_i+U_i, \qquad E[U_i\mid Z_i]=0. \tag{4.2}
At each education level, positive and negative values of U_i average to zero.
Individual wages need not lie on the line. The line describes the conditional population mean.
The zero conditional mean condition is also called exogeneity. Here it follows because U_i is the CEF residual and the CEF in Equation 4.1 is linear.
4.2 Population Coefficients
The CT and LIE from Section 2 imply E[r(Z_i)U_i]=0 for every function r(Z_i) for which the expectation exists.
Setting r(Z_i)=1 and r(Z_i)=Z_i gives E[U_i]=0, \qquad E[Z_iU_i]=0. \tag{4.3}
Taking expectations in Equation 4.2 gives E[Y_i]=\beta_1+\beta_2E[Z_i].
Moreover, Equation 4.3 implies \mathrm{Cov}(Z_i,U_i)=0. Therefore, \mathrm{Cov}(Y_i,Z_i) =\beta_2\mathrm{Var}(Z_i).
If Y_i and Z_i have finite second moments and \mathrm{Var}(Z_i)>0, the population coefficients are \boxed{ \beta_2 =\frac{\mathrm{Cov}(Y_i,Z_i)}{\mathrm{Var}(Z_i)}, \qquad \beta_1=E[Y_i]-\beta_2E[Z_i] }. \tag{4.4}
Thus, \beta_1 and \beta_2 are functions of population moments.
4.3 Estimating the Coefficients
Suppose (Y_1,Z_1),\ldots,(Y_n,Z_n) are an i.i.d. sample.
The method of moments replaces Equation 4.3 by the sample conditions \begin{align*} 0&=\frac{1}{n}\sum_{i=1}^n(Y_i-b_1-b_2Z_i), \\ 0&=\frac{1}{n}\sum_{i=1}^nZ_i(Y_i-b_1-b_2Z_i). \end{align*}
If \widehat\sigma_Z^2>0, their unique solution is \boxed{ \widehat\beta_2 =\frac{\widehat\sigma_{YZ}}{\widehat\sigma_Z^2}, \qquad \widehat\beta_1=\overline Y-\widehat\beta_2\overline Z }. \tag{4.5}
The fitted value and residual for observation i are \begin{align*} \widehat Y_i &=\widehat\beta_1+\widehat\beta_2Z_i, \\ \widehat u_i &=Y_i-\widehat Y_i. \end{align*}
The population residual U_i is defined using the population CEF. The sample residual \widehat u_i is calculated using estimated coefficients.
Wage and Education
Let’s estimate the simple linear regression using the CPS sample:
Under the linear specification, the fitted regression line is \widehat m_Z(z)=-16.45+2.90z.
Thus, the fitted mean hourly wage increases by USD 2.90 for one additional year of education.
The points in Figure 4.1 are sample quantities, not values of the population CEF.
Their systematic deviations from the line may indicate that an exactly linear CEF is too restrictive. We return to this issue below.
4.4 Why It Is Called Ordinary Least Squares
For possible coefficients b_1 and b_2, consider the sum of squared residuals S_n(b_1,b_2) =\sum_{i=1}^n(Y_i-b_1-b_2Z_i)^2.
Under the condition \widehat\sigma_Z^2>0, the method-of-moments estimators in Equation 4.5 also satisfy (\widehat\beta_1,\widehat\beta_2) =\arg\min_{(b_1,b_2)\in\mathbb R^2}S_n(b_1,b_2).
The notation \arg\min returns the coefficient values that make S_n(b_1,b_2) as small as possible.
They are therefore called the ordinary least squares (OLS) estimators.
Differentiating S_n(b_1,b_2) with respect to b_1 and b_2, and setting the derivatives equal to zero, gives \sum_{i=1}^n\widehat u_i=0, \qquad \sum_{i=1}^nZ_i\widehat u_i=0.
These are precisely the two sample moment conditions used above.
4.5 Multiple Linear Regression
Let \boldsymbol X_i=(1,X_{i2},\ldots,X_{ik})' contain a constant and k-1 observed regressors.
The exact linear CEF is m(\boldsymbol x) =E[Y_i\mid\boldsymbol X_i=\boldsymbol x] =\boldsymbol x'\boldsymbol\beta. \tag{4.6}
It gives the multiple linear regression model Y_i=\boldsymbol X_i'\boldsymbol\beta+U_i, \qquad E[U_i\mid\boldsymbol X_i]=0. \tag{4.7}
The CT and LIE imply the population moment condition E[\boldsymbol X_iU_i]=\boldsymbol 0.
Substituting Equation 4.7 gives \begin{align*} \boldsymbol 0 &=E[\boldsymbol X_i(Y_i-\boldsymbol X_i'\boldsymbol\beta)] \\ &=E[\boldsymbol X_iY_i] -E[\boldsymbol X_i\boldsymbol X_i']\boldsymbol\beta. \end{align*}
Using the notation from Section 3, \boldsymbol q=\boldsymbol Q\boldsymbol\beta.
If \boldsymbol Q is invertible, its inverse \boldsymbol Q^{-1} exists. The population coefficient vector is then \boxed{ \boldsymbol\beta=\boldsymbol Q^{-1}\boldsymbol q }. \tag{4.8}
The matrix \boldsymbol Q=E[\boldsymbol X_i\boldsymbol X_i'] is a second-moment matrix, not the covariance matrix \mathrm{Var}(\boldsymbol X_i).
Because \boldsymbol X_i contains a constant, its covariance matrix cannot be inverted. The second-moment matrix \boldsymbol Q may nevertheless be invertible.
OLS Estimator
The sample matrix \widehat{\boldsymbol Q} must be invertible to calculate the OLS estimator.
Replacing the population moments by their sample counterparts gives \begin{aligned} \widehat{\boldsymbol\beta} &=\widehat{\boldsymbol Q}^{-1}\widehat{\boldsymbol q} \\ &=\left(\frac{1}{n}\sum_{i=1}^n \boldsymbol X_i\boldsymbol X_i'\right)^{-1} \left(\frac{1}{n}\sum_{i=1}^n \boldsymbol X_iY_i\right). \end{aligned} \tag{4.9}
Equivalently, using the data matrix \boldsymbol X and outcome vector \boldsymbol Y, \widehat{\boldsymbol\beta} =(\boldsymbol X'\boldsymbol X)^{-1}\boldsymbol X'\boldsymbol Y.
The OLS estimator also minimizes \sum_{i=1}^n(Y_i-\boldsymbol X_i'\boldsymbol b)^2 over \boldsymbol b\in\mathbb R^k.
Define the multiple-regression residual \widehat u_i :=Y_i-\boldsymbol X_i'\widehat{\boldsymbol\beta}.
The residuals satisfy \frac{1}{n}\sum_{i=1}^n\boldsymbol X_i\widehat u_i =\boldsymbol 0.
Because \boldsymbol X contains a constant, this means that the residuals have sample mean zero and sample covariance zero with every nonconstant included regressor.
4.6 Correct Specification and Nonlinear CEFs
A regression is linear when it is linear in its coefficients.
The CEF may therefore be nonlinear in an original regressor.
Let A_i denote years of experience. For example, m_A(a)=E[Y_i\mid A_i=a] =\gamma_1+\gamma_2a+\gamma_3a^2 \tag{4.10} is linear in \boldsymbol\gamma=(\gamma_1,\gamma_2,\gamma_3)'.
It is a multiple linear regression with regressor vector \boldsymbol X_i=(1,A_i,A_i^2)'.
The curved CEF of wage given experience from Section 2 illustrates why a quadratic term may be useful:
If we used only a straight line for this population CEF, the difference V_i=Y_i-b_1-b_2A_i would generally satisfy E[V_i\mid A_i]\neq0.
The straight line would then be a misspecified model for the conditional mean.
NoteBest linear approximation
If the specified linear function is not the exact CEF, finite second moments and an invertible \boldsymbol Q define the best mean-squared linear approximation \boldsymbol\beta^* :=\arg\min_{\boldsymbol b\in\mathbb R^k} E[(Y_i-\boldsymbol X_i'\boldsymbol b)^2] =\boldsymbol Q^{-1}\boldsymbol q.
For U_i^*=Y_i-\boldsymbol X_i'\boldsymbol\beta^*, we have E[\boldsymbol X_iU_i^*]=\boldsymbol 0 but generally E[U_i^*\mid\boldsymbol X_i]\neq0.
Under the sampling conditions below, OLS is consistent for \boldsymbol\beta^*, but these coefficients need not describe marginal effects of the actual CEF.
4.7 Consistency of OLS
Section 3 established that, for a fixed number of regressors, i.i.d. sampling and finite second moments give \widehat{\boldsymbol Q}\overset{p}{\to}\boldsymbol Q, \qquad \widehat{\boldsymbol q}\overset{p}{\to}\boldsymbol q.
If \boldsymbol Q is invertible, probability-limit arithmetic gives \begin{align*} \widehat{\boldsymbol\beta} &=\widehat{\boldsymbol Q}^{-1}\widehat{\boldsymbol q} \\ &\overset{p}{\to} \boldsymbol Q^{-1}\boldsymbol q. \end{align*}
The consistency argument requires:
- i.i.d. observations (Y_i,\boldsymbol X_i')',
- finite second moments,
- an invertible population second-moment matrix \boldsymbol Q.
These conditions give consistency for \boldsymbol Q^{-1}\boldsymbol q=\boldsymbol\beta^*.
Under the exact linear CEF in Equation 4.6, \boldsymbol\beta^*=\boldsymbol\beta. Therefore, \boxed{ \widehat{\boldsymbol\beta} \overset{p}{\to}\boldsymbol\beta }.
4.8 Marginal Effects
A marginal effect describes how the population CEF changes when a regressor changes, holding the other underlying regressors fixed.
If the CEF is differentiable with respect to a continuous regressor x_j, j=2,\ldots,k, its marginal effect at \boldsymbol x is ME_j(\boldsymbol x) :=\frac{\partial m(\boldsymbol x)}{\partial x_j}. \tag{4.11}
For an additive linear CEF, m(\boldsymbol x) =\beta_1+\beta_2x_2+\cdots+\beta_kx_k, we have ME_j(\boldsymbol x)=\beta_j.
The corresponding estimated marginal effect is \widehat\beta_j.
Education is measured in discrete years. Its one-year conditional-mean difference is m_Z(z+1)-m_Z(z)=\beta_2 whenever both education levels are possible.
More generally, the effect of changing a discrete regressor from a to b is m(x_j=b,\boldsymbol x_{-j}) -m(x_j=a,\boldsymbol x_{-j}), provided both (a,\boldsymbol x_{-j}) and (b,\boldsymbol x_{-j}) are possible values. The vector \boldsymbol x_{-j} contains the regressors held fixed.
Treating experience as continuous, the quadratic CEF in Equation 4.10 has marginal effect \frac{\partial m_A(a)}{\partial a} =\gamma_2+2\gamma_3a.
When experience changes, both A_i and its transformation A_i^2 change. The marginal effect therefore depends on a.
A regression coefficient is not always itself a marginal effect.
The next section develops marginal effects for nonlinear specifications and discusses when conditional-mean differences can be interpreted causally.