Multiple Linear Regression

The motivation we develop Multiple Linear Regression is that Simple Linear Regression only contains one variable that could explaining $Y$.

That is, it will face a serious Omitted Variable Bias (OVB) problem.

OLS under OVB

The true model:

$$Y_i={\beta }_0+{\beta }_1{X}_{1i}+{\beta }_2{X}_{2i}+U_I$$

However, we incorrectly assume:

$$Y_i={\alpha }_0+{\alpha }_{1i}{X}_{1i}+V_i$$

Where $E[V_i|X_{1i}]=0$

---

From Simple Linear Regression in Matrix Notation, we could also use Matrix to denote the model.

The initial model is:

$$y_i+{\beta }_1+{\beta }_2{x}_{2i}+{\beta }_3{x}_{3i}+\dots {\beta }_k{x}_{ki}+{ϵ}_i$$

We have $k$ unknowns, and $n$ equations. Only when equations are more than unknowns, we could get the answers.

However, once we change the unknowns to:

$$\mathbf{X}=\left(\begin{array}{cccc}1& x_{21}& \dots & x_{k1}\\ \dots & \dots & \dots & \dots \\ 1& x_{2n}& \dots & x_{kn}\end{array}\right)$$

We could denote the model to a much more simpler form:

$$\mathbf{y}=\mathbf{X}\beta +ϵ$$

recall that from Simple Linear Regression in Matrix Notation, the FOC of $S(b))$ is exactly the same, thanks to the matrix:

$$\frac{\delta \mathbf{S}(\mathbf{b})}{\delta \mathbf{b}}=2{\mathbf{X}}^{\prime }\mathbf{Xb}-2{\mathbf{X}}^{\prime }\mathbf{y}=0$$

The difference is that the ${\mathbf{X}}^{\prime }\mathbf{X}$ is now a $k\times k$ matrix, not $1\times 1$

We now could derive $e$:

$$\mathbf{y}=\mathbf{Xb}+\mathbf{e}$$ $$\mathbf{e}=\mathbf{y}-\mathbf{Xb}$$ $$\mathbf{e}=\mathbf{y}-\mathbf{X}({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\prime }\mathbf{y}$$ $$\mathbf{e}=({\mathbf{I}}_{\mathbf{k}\times \mathbf{k}}-\mathbf{X}({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\prime })\mathbf{y}$$ $${\mathbf{X}}^{\prime }\mathbf{e}={\mathbf{X}}^{\prime }({\mathbf{I}}_{\mathbf{n}\times \mathbf{n}}-\mathbf{X}({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\prime })\mathbf{y}$$

We called $\mathbf{M}=\mathbf{I}-\mathbf{X}({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\prime }$ “残差生成矩阵”

$${\mathbf{X}}^{\prime }\mathbf{e}={\mathbf{X}}^{\prime }{\mathbf{I}}_{\mathbf{k}\times \mathbf{k}}\mathbf{y}-\underset{{\mathbf{I}}_{\mathbf{k}}}{\underbrace{{\mathbf{X}}^{\prime }\mathbf{X}({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}}}{\mathbf{X}}^{\prime }\mathbf{y}$$

Thus,

$${\mathbf{X}}^{\prime }\mathbf{e}=0$$

Thus we prove that the residual vector $e$ is orthogonal to the matrix of explanatory variables $X$, i.e. ${\mathbf{X}}^{\prime }\mathbf{e}=0$.

OLS part

Ordinary Least Squares

We could then write $\hat{y}$:

$$\hat{\mathbf{y}}=\mathbf{Xb}=\mathbf{X}\underset{\mathbf{b}}{\underbrace{({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\prime }\mathbf{y}}}$$

We denote $\mathbf{H}$:

$$\mathbf{H}=\mathbf{X}({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\prime }$$ $$\mathbf{y}=\mathbf{Hy}+\mathbf{My}=\hat{\mathbf{y}}+\mathbf{e}$$

Try to figure out that $\mathbf{MH}=0$ ($\mathbf{H}$ and $\mathbf{M}$ are orthogonal).

Hint: $\mathbf{H}=\mathbf{I}-\mathbf{M}$

1. identity matrix
2. orthogonal projection matrix

are both Idempotency.

The Least Squares is unbiased

See Unbiasedness.

$$b=(X^{\prime }X{)}^{-1}{X}^{\prime }(X\beta +ϵ)=\beta +(X^{\prime }X{)}^{-1}{X}^{\prime }ϵ$$

To see why the OLS estimator is most efficient, see attached: Proof of Gauss-Markov Theorem.

Estimating the disturbance Variance

That is to check $e$.

Recall that we have $$ \mathbf{e = (I_{k \times k} - X (X’X)^{-1}X’ )y}

We denote $M = I_{k \times k} - X (X'X)^{-1}X'$ so $e = My$. we already know that $MX = 0$ so $$ e = My = M(X\beta + \epsilon) = \underbrace{ MX\beta }_{ =0 } +M\epsilon = M\epsilon

Thus $E(e)=E[Mϵ]=ME[ϵ]=M0=0$

Under the current assumptions, $M=I_{n\times n}-X(X^{\prime }X{)}^{-1}{X}^{\prime }$ is a fixed matrix, and $E(ϵ{ϵ}^{\prime })={\sigma }^{2}I$.

$$\begin{array}{rl}Var(e)& =E[e{e}^{\prime }]=E[Mϵ{ϵ}^{\prime }M]\\ & ={\sigma }^2{M}^2\\ & ={\sigma }^{2}M\end{array}$$

Hint: $M$ is idempotent. See Idempotency.

So we know that:

$$E({ϵ}^{\prime }Mϵ)={\sigma }^2\mathrm{Tr}(M)$$

A simple proof: to see why the expectation of a matrix is equal to its Trace, see here

$$\begin{array}{rl}\mathrm{Tr}(M)& =\mathrm{Tr}(I)-\mathrm{Tr}(X(X^{\prime }X{)}^{-1}{X}^{\prime })\\ & =n-k\end{array}$$

This shows that $E[e^{\prime }e]=(n-k){\sigma }^2$ so that:

$$s^2=\frac{e^{\prime }e}{n-k}$$

is unbiased estimator of ${\sigma }^2$. $s$ is called standard error of the regression.

$n-k$ is also the degree of freedom.

$$R^2=1-\frac{SSR}{SST}$$

Omitting Relevant Variables

The source of omitted relevant variables: if we ignore some of the dependent variables that have significant relation with the explanatory variables, we could cause bias in the regressors.

Eg: we are focusing on the relation between wage and education, but we forget to include Experience as a explanatory variable.

Discussion

Suppose the true model is:

$$y=X\beta +ϵ=X_1{\beta }_1+X_2{\beta }_2+ϵ$$

Suppose we omit $X_2$, the model now becomes:

$$y=X_1{\beta }_1+ϵ$$

Now the restricted estimator is:

$$\begin{array}{rl}{b}_R& =(X_1^{\prime }{X}_1{)}^{-1}{X}_1^{\prime }y\\ & =(X_1^{\prime }{X}_1{)}^{-1}{X}_1^{\prime }(X_1{\beta }_1+X_2{\beta }_2+ϵ)\\ & ={\beta }_1+(X_1^{\prime }{X}_1{)}^{-1}{X}_1^{\prime }{X}_2{\beta }_2+(X_1^{\prime }{X}_1{)}^{-1}{X}_1^{\prime }ϵ\end{array}$$ $$E(b_R)={\beta }_1+(X_1^{\prime }{X}_1{)}^{-1}{X}_1^{\prime }{X}_2{\beta }_2$$

It shows that $b_R$ will be biased in a predictable direction: the bias is equal to the regression effect of $X_2$ predicted by a regression on $X_2$.

We use $e_R=y-X_1{b}_R$ to represent the corresponding restricted residuals.

Now we have a clear thinking that there must be a difference between $b_R$ and $b_1$.

Comparison between residuals

Now we compare $e_R^{\prime }{e}_R$ and $e^{\prime }e$:

First, we try to get $e_R$:

$$e_R=M_{1}y=M_1(X_1{b}_1+X_2{b}_2+e)=M_1{X}_2{b}_2+e$$

如何形象理解$M$ 的意义？ 其实简单来说，$M$ 是一个投影矩阵，它的作用就是将一个向量投影到$X$ 的列空间的正交补空间上。简单来说，$M$会移除任何在$X$ 列空间内的成分。

After calculating the results, we could figure out:

$$e_R^{\prime }{e}_R\ge e^{\prime }e$$

Only when $M_1{X}_2{b}_2=0$, the equal sign is satisfied.