Simple Linear Regression in Matrix Notation

One of the reasons that we want to introduce Matrix to linear regression is that it has greater generality. We could use simpler notations to represent the model. You may first feel not familiar, just try to get used to it.

Recall the classic notation of Simple Linear Regression:

$$y_i=\alpha +\beta x_i+{ϵ}_i$$

We change the notation a bit…

$$y_i={\beta }_1+{\beta }_2{x}_i+{ϵ}_i$$

Now, considering there are $n$ equations here, we could rewrite the group as:

• $\mathbf{y}=(y_1,y_2,\dots y_n{)}^{\prime }$
• $\mathbf{X}$ denotes an $n\times 2$ matrix:

$$\mathbf{X}=\left(\begin{array}{cc}1& x_1\\ 1& x_2\\ \dots & \dots \\ 1& x_n\end{array}\right)$$

• the two-element column vector (In the Simple Linear Regression, we only have those two): $\beta =({\beta }_1,{\beta }_2{)}^{\prime }$
• and an error vector.

In short, we could rewrite the Simple Linear Regression in this way:

$$\mathbf{y}=\mathbf{X}\beta +ϵ$$

Recall the assumption of Simple Linear Regression, we could represent the error terms as:

这里缺一个引用，关于assumption的

$$E\{ϵ\}=0,E\{ϵ{ϵ}^{\prime }\}={\sigma }^2{\mathbf{I}}_{\mathbf{n}}$$

Hint: Just a multiplication:

$$ϵ=\left(\begin{array}{c}{ϵ}_1\\ {ϵ}_2\\ \dots \\ {ϵ}_n\end{array}\right),\text{and }{ϵ}^{\prime }=\left(\begin{array}{c}{ϵ}_1,{ϵ}_2,\dots ,{ϵ}_n\end{array}\right)$$

the estimator of $\beta$ is now $\mathbf{b}=(b_1,b_2{)}^{\prime }$.

$$S(\mathbf{b})={\mathbf{e}}^{\prime }\mathbf{e}=(\mathbf{y}-\mathbf{Xb}{)}^{\prime }(\mathbf{y}-\mathbf{Xb})$$ $$={\mathbf{y}}^{\prime }\mathbf{y}-{\mathbf{y}}^{\prime }\mathbf{Xb}-{\mathbf{b}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{y}+{\mathbf{b}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{Xb}={\mathbf{y}}^{\prime }\mathbf{y}-2{\mathbf{b}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{y}+{\mathbf{b}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{Xb}$$

Take the FOC here:

If you are not familiar with how to take derivatives under Matrix environment, see attached: 矩阵求导. In here, we conclude some useful results of differentiation of vectors:

$$\frac{\delta ({\mathbf{x}}^{\prime }\mathbf{b})}{\delta \mathbf{b}}=\frac{\delta ({\mathbf{b}}^{\prime }\mathbf{x})}{\delta \mathbf{b}}=\mathbf{x}$$ $$\frac{\delta ({\mathbf{b}}^{\prime }\mathbf{Ab})}{\delta \mathbf{b}}=(\mathbf{A}+{\mathbf{A}}^{\prime })\mathbf{b}$$ $$\frac{{\delta }^2({\mathbf{b}}^{\prime }\mathbf{Ab})}{\delta \mathbf{b}\delta {\mathbf{b}}^{\prime }}=(\mathbf{A}+{\mathbf{A}}^{\prime })$$

So it is clear that

$$\frac{\delta ({\mathbf{y}}^{\prime }\mathbf{y})}{\delta b}=0$$ $$\frac{\delta ({\mathbf{b}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{y})}{\delta \mathbf{b}}={\mathbf{X}}^{\prime }\mathbf{y}$$ $$\frac{\delta {\mathbf{b}}^{\prime }{\mathbf{X}}^{\prime }\mathbf{Xb}}{\delta \mathbf{b}}=2{\mathbf{X}}^{\prime }\mathbf{Xb}$$

Thus,

To obtain the correct value of $\mathbf{b}$, we continue calculating:

$${\mathbf{X}}^{\prime }\mathbf{Xb}-{\mathbf{X}}^{\prime }\mathbf{y}=0$$ $$({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}[{\mathbf{X}}^{\prime }\mathbf{Xb}-{\mathbf{X}}^{\prime }\mathbf{y}]=0$$ $$\mathbf{b}-({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\prime }\mathbf{y}=0$$ $$\mathbf{b}=({\mathbf{X}}^{\prime }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\prime }\mathbf{y}$$

This is what we want to know.