Bootstrap

EC484: Econometric Analysis

Bootstrap

Taisuke Otsu

London School of Economics

2025/6

Contents

1. Basic idea (Hansen Ch. 10.6-8)
2. Bootstrap theory (Hansen Ch. 10.9-13, 16, 19, 21)
3. Bootstrap for OLS and GMM (Hansen Ch. 10.27)

1. Basic idea

Example

• Consider wage equation estimated by CPS subsample

$$\log (wage)=\begin{array}{l}0.698\\ (0.493)\end{array}+\begin{array}{l}0.155educ\\ (0.031)\end{array}+\hat{e}$$ $${\hat{\sigma }}^2=\underset{(0.043)}{0.144},n=20$$

$\bullet |\mathfrak{f}e|educ\sim N(0,{\sigma }^2)$

$$\mu =\mathbb{E}[\text{w a g e}|\text{e d u c}=16]=\exp \left({\beta }_0+16{\beta }_1+{\sigma }^2/2\right)$$

By delta method

$$\hat{\mu }=25.80\text{\hspace{1em}(2.29)}$$

t-test or CI for ${\beta }_1$ or $\mu$ is based on asymptotic approximation
Bootstrap provides alternative se and CI

Bootstrap algorithm

• Original sample for Y = log(wage) and X = educ

i	1	2	3	4	5		18	19	20
Yi	3.64	3.71	2.65	2.82	3.50	···	3.18	3.59	3.14
Xi	18	18	13	16	16	···	16	18	16

• Bootstrap (re)sample is constructed by randomly drawing integers of size $n=20$ from $\{1,2,\dots ,20\}$ with replacement

. E.g. If we draw, {16, 5, 17, 20, 20, … , 7, 1, 8}, then bootstrap sample is (same unit can appear)

$$\begin{array}{cccccccccc}i& 1& 2& 3& 4& 5& & 18& 19& 20\\ Y_i^{\ast }& Y_{16}& Y_5& Y_{17}& Y_{20}& Y_{20}& \dots & Y_7& Y_1& Y_8\\ X_i^{\ast }& X_{16}& X_5& X_{17}& X_{20}& X_{20}& \dots & X_7& X_1& X_8\end{array}$$

By $\{Y_i^{\ast },X_i^{\ast }:i=1,\dots ,n\}$ , we can compute ${\hat{\beta }}_1^{\ast }$ and ${\hat{\mu }}^{\ast }$ , say
• We can repeat this process as many as we want to get $\{{\hat{\beta }}_{1b}^{\ast },{\hat{\mu }}_b^{\ast }:b=1,\dots ,B\}$

Bootstrap variance and se

• Generally consider estimator θˆ and its bootstrap counterparts $\{{\hat{\theta }}_b^{\ast }:b=1,\dots ,B\}$
• Bootstrap variance estimator of $\hat{\theta }$ is

$${\hat{\mathbf{V}}}_{\hat{\theta }}^{\mathrm{boot}}=\frac{1}{B-1}\sum _{b=1}^B({\hat{\theta }}_b^{\ast }-\overline{{\hat{\theta }}^{\ast }})({\hat{\theta }}_b^{\ast }-\overline{{\hat{\theta }}^{\ast }}{)}^{\prime }$$ $$where\overline{{\hat{\theta }}^{\ast }}=\frac{1}{B}\sum _{b=1}^B{\hat{\theta }}_b^{\ast }$$

If θˆ is scalar, bootstrap standard error is

$${\mathbf{s}}_{\hat{\mathbf{\theta }}}^{\mathrm{boot}}=\sqrt{{\hat{\mathbf{V}}}_{\hat{\mathbf{\theta }}}^{\mathrm{boot}}}$$

and normal-approximation bootstrap CI is

$${\mathrm{C}}^{\mathrm{nb}}=[\hat{\theta }-z_{1-\alpha /2}{s}_{\hat{\theta }}^{\mathrm{boot}},\hat{\theta }+z_{1-\alpha /2}{s}_{\hat{\theta }}^{\mathrm{boot}}]$$

Percentile CI

Again $\hat{\theta }$ is scalar. Several ways to construct CI
• Based on $\{{\hat{\theta }}_b^{\ast }:b=1,\dots ,B\}$ , compute its sample quantiles $q_{\alpha }^{\ast }$ . Percentile bootstrap CI for $\theta$ is

$${\mathrm{C}}^{\mathrm{pc}}=[q_{\alpha /2}^{\ast },q_{1-\alpha /2}^{\ast }]$$

• This CI is transformation respecting, i.e. for any monotone transform $m(\theta )$ , its CI is given by

$$[m(q_{\alpha /2}^{\ast }),m(q_{1-\alpha /2}^{\ast })]$$

2. Bootstrap theory

Bootstrap

• Consider iid sample $\{Z_1,\dots ,Z_n\}$ from population distribution function F
• Consider scalar estimator $\hat{\theta }$ with cdf

$$G_n(u,F)=\mathbb{P}[\hat{\theta }\le u|F]$$

• Usually $G_n(u,F)$ is very complicated function depending on n, so need some approximation
Asymptotic approximation to $G_n(u,F)$ is

$$G(u,F)=\underset{n\to \infty }{\lim }{G}_n(u,F)$$

• Bootstrap provides an alternative approximation to $G_n(u,F)$ , that is

$$G_n^{\ast }(u)=G_n(u,F_n)$$

where $F_n$ is empirical distribution function (EDF)

• $G_n^{\ast }(u)$ is called bootstrap distribution and bootstrap conducts inference based on $G_n^{\ast }(u)$
• Note: If $\{Z_1^{\ast },\dots ,Z_n^{\ast }\}$ is iid sample from $F_n$ , then ${\hat{\theta }}^{\ast }$ satisfies

$$G_n^{\ast }(u)=\mathbb{P}\left[T_n^{\ast }\le u\mid F_n\right]$$

i.e. in ”bootstrap world”, $F_n$ plays role of population

Empirical distribution

• Popular choice for $F_n$ is empirical distribution function (EDF)

$$F_n(u)=\frac{1}{n}\sum _{i=1}^n\mathbb{I}\{Z_i^{(1)}\le u^{(1)}\}\dots \mathbb{I}\{Z_i^{(d)}\le u^{(d)}\}$$

which is consistent and asymptotically normal

$$\sqrt{n}(F_n(u)-F(u))\stackrel{d}{\to }N(0,F(u)(1-F(u)))$$

for each $u\in {\mathbb{R}}^d$

• From EDF $F_n$ , bootstrap sample $\{Z_1^{\ast },\dots ,Z_n^{\ast }\}$ is obtained by drawing from observed sample $\{Z_1,\dots ,Z_n\}$ with equal 1/n weights (with replacement)

Distribution of bootstrap observations

• Let $Z^{\ast }$ be (scalar) random draw from EDF $F_n$ by original sample $\{Z_1,\dots ,Z_n\}$
• How can we think about distribution of $Z^{\ast }?$ There are two randomness: (i) randomness for resampling from $F_n$ and (ii) randomness of original sample $\{Z_1,\dots ,Z_n\}$
• To separate these randomness, introduce conditional probability and mean given $F_n$

$${\mathbb{P}}^{\ast }[Z^{\ast }\le c]=\mathbb{P}[Z^{\ast }\le c|F_n]$$ $${\mathbb{E}}^{\ast }[Z^{\ast }]=\mathbb{E}[Z^{\ast }|F_n]$$

• Note: conditional distribution $Z^{\ast }|F_n$ is always discrete (with support $\{Z_1,\dots ,Z_n\})$

Conditional mean and variance

• Conditional mean of $Z^{\ast }$ is

$$\begin{array}{l}{\mathbb{E}}^{\ast }[Z^{\ast }]={\sum }_{i=1}^n{Z}_i{\mathbb{P}}^{\ast }[Z^{\ast }=Z_i]\\ ={\sum }_{i=1}^n{Z}_i\frac{1}{n}=\bar{Z}\end{array}$$

• Conditional variance of $Z^{\ast }$ is

$$\begin{array}{l}{\mathrm{var}}^{\ast }[Z^{\ast }]={\mathbb{E}}^{\ast }[Z^{\ast 2}]-({\mathbb{E}}^{\ast }[Z^{\ast }]{)}^2\\ ={\sum }_{i=1}^n{Z}_i^2{\mathbb{P}}^{\ast }[Z^{\ast }=Z_i]-{\bar{Z}}^2\\ ={\sum }_{i=1}^n{Z}_i^2\frac{1}{n}-{\bar{Z}}^2\equiv {\hat{\sigma }}^2\end{array}$$

Similarly for $\begin{array}{r}{\bar{Z}}^{\ast }=\frac{1}{n}{\sum }_{i=1}^n{Z}_i^{\ast }\end{array}$

$${\mathbb{E}}^{\ast }[{\bar{Z}}^{\ast }]=\frac{1}{n}\sum _{i=1}^n{\mathbb{E}}^{\ast }[Z_i^{\ast }]=\bar{Z}$$

and

$${\mathrm{var}}^{\ast }[{\bar{Z}}^{\ast }]=\frac{1}{n^2}\sum _{i=1}^n{\mathrm{var}}^{\ast }[Z_i^{\ast }]=\frac{1}{n}{\hat{\sigma }}^2$$

Convergence for bootstrap statistics

• How can we think about convergence or asymptotic distribution for ${\bar{Z}}^{\ast }?$
• Definition: Bootstrap statistic $W_n^{\ast }$ converges in bootstrap probability to W as $n\to \infty$ (denoted by $W_n^{\ast }\stackrel{p^{\ast }}{}W)$ if

$${\mathbb{P}}^{\ast }\left[\left|W_n^{\ast }-W\right|>ϵ\right]\stackrel{p}{\to }0\text{f o r e a c h}ϵ>0$$

. Note: “ p→” is (standard) convergence in probability for original sample $\{Z_1,\dots ,Z_n\}$
Property: If $W_n\stackrel{p}{}W$ , then $W_n\stackrel{p^{\ast }}{}W$

Bootstrap WLLN

• Theorem: If $\{Z_1,\dots ,Z_n\}$ is iid and $\mathbb{E}[|Z_i{|}^2]<\infty$ , then

$${\bar{Z}}^{\ast }-\bar{Z}\stackrel{p^{\ast }}{\to }0$$ $${\bar{Z}}^{\ast }\stackrel{p^{\ast }}{\to }\mu =\mathbb{E}[Z_i]$$

• Proof: Since $\bar{Z}\stackrel{p}{}\mu$ implies $\bar{Z}\stackrel{p^{\ast }}{}\mu$ , it is enough to show the first statement

Pick any ϵ > 0. By Markov inequality

$$\begin{array}{l}{\mathbb{P}}^{\ast }\left[\left|{\bar{Z}}^{\ast }-\bar{Z}\right|>ϵ\right]\le {ϵ}^{-2}{\mathbb{E}}^{\ast }\left[{\left({\bar{Z}}^{\ast }-\bar{Z}\right)}^2\right]={ϵ}^{-2}{\mathrm{var}}^{\ast }\left[{\bar{Z}}^{\ast }\right]\\ ={ϵ}^{-2}\frac{1}{n^2}{\sum }_{i=1}^n{\mathrm{var}}^{\ast }\left[Z_i^{\ast }\right]\\ ={ϵ}^{-2}\frac{1}{n}{\hat{\sigma }}^2\stackrel{\rho }{\to }0\end{array}$$

where $\begin{array}{r}{\hat{\sigma }}^2=\frac{1}{n}{\sum }_{i=1}^n(Z_i-\bar{Z}{)}^2\end{array}$ (sample var) satisfying σˆ2 p→ var[Zi ]

Asymptotic distribution

• For asymptotic distribution of ${\bar{Z}}^{\ast }$ , we need bootstrap version of convergence in distribution
• Definition: Bootstrap statistic $W_n^{\ast }$ converges in bootstrap distribution to W as $n\to \infty$ (denoted by $W_n^{\ast }\stackrel{d^{\ast }}{}W)$ if

$${\mathbb{P}}^{\ast }\left[W_n^{\ast }\le u\right]\stackrel{p}{\to }\mathbb{P}\left[W\le u\right]$$

for each u at which $\mathbb{P}[W\le u]$ is continuous

Bootstrap CLT

• Theorem: If $\{Z_1,\dots ,Z_n\}$ is iid and $\mathbb{E}[|Z_i{|}^4]<\infty$ , then

$$\sqrt{n}\left({\bar{Z}}^{\ast }-\bar{Z}\right)\stackrel{d^{\ast }}{\to }\mathrm{N}\left(0,{\sigma }^2\right)$$

where ${\sigma }^2=\mathrm{var}[Z_i]$

• Proof follows by applying Lindeberg-Feller CLT under conditional distribution given $F_n$ . We cannot use Lindeberg-Levy because conditional distribution varies with n
See Hansen, Ch. 10.14 for detail

Bootstrap CMT

• Theorem: If $W_n^{\ast }\stackrel{p^{\ast }}{}c$ and $g(\cdot )$ is continuous at c, then

$$g\left(W_n^{\ast }\right)\stackrel{p^{\ast }}{\to }g(c)$$

• Theorem: If $W_n^{\ast }\stackrel{d^{\ast }}{}W$ and $g(\cdot )$ has set of discontinuity points $D_g$ such that ${\mathbb{P}}^{\ast }[Z^{\ast }\in D_g]=0$ , then

$$g\left(W_n^{\ast }\right)\stackrel{d^{\ast }}{\to }g(c)$$

Consistency of bootstrap variance estimator

• Based on above tools, now consider bootstrap variance estimator of θˆ. Since $\hat{\theta }$ itself typically does not have limiting distribution, suppose

$$\begin{array}{l}{Z}_n=a_n(\hat{\theta }-\theta )\stackrel{d}{\to }\xi \\ Z_n^{\ast }=a_n\left({\hat{\theta }}^{\ast }-\hat{\theta }\right)\stackrel{d^{\ast }}{\to }\xi \end{array}$$

for some sequence $a_n$ (typically $\sqrt{n})$ and distribution ξ (typically normal)

• Three variance concepts

$${\mathbf{V}}_{\theta }=\mathrm{var}[\xi ](\text{t a r g e t})$$ $${\hat{\mathbf{V}}}_{\theta }^{\text{b o o t}}={\mathrm{var}}^{\ast }\left[Z_n^{\ast }\right](\text{c o n c e p t u a l b o o t v a r})$$ $${\hat{\mathbf{V}}}_{\theta ,B}^{\mathrm{boot}}=\frac{1}{B-1}\sum _{b=1}^B\left(Z_{n,b}^{\ast }-\overline{Z_n^{\ast }}\right){\left(Z_{n,b}^{\ast }-\overline{Z_n^{\ast }}\right)}^{\prime }\text{(a c t u a l b o o t v a r)}$$

Under certain conditions (see Theorem 10.9 of Hansen-B)

$${\hat{\mathbf{V}}}_{\theta ,B}^{\mathrm{boot}}\stackrel{p^{\ast }}{\to }{\hat{\mathbf{V}}}_{\theta }^{\mathrm{boot}}\mathrm{as}B\to \infty$$ $${\hat{\mathbf{V}}}_{\theta }^{\mathrm{boot}}\stackrel{p^{\ast }}{\to }\mathrm{var}[\xi ]$$

Consistency of bootstrap percentile CI

• For percentile CI, suppose

$$\begin{array}{l}{Z}_n=a_n(\hat{\theta }-\theta )\stackrel{d}{\to }\xi \\ Z_n^{\ast }=a_n\left({\hat{\theta }}^{\ast }-\hat{\theta }\right)\stackrel{d^{\ast }}{\to }\xi \end{array}$$

where pdf ξ is symmetric around zero

• Then

$$\mathbb{P}[\theta \in {\mathrm{C}}^{\mathrm{pc}}]=\mathbb{P}[q_{\alpha /2}^{\ast }\le \theta \le q_{1-\alpha /2}^{\ast }]\to 1-\alpha$$

where $q_{\alpha }^{\ast }$ is $\alpha$ -th quantile of ${\hat{\theta }}^{\ast }$

• Sketch: Let $H(\cdot )$ be cdf of $\xi$ . Since $\alpha /2$ -th quantile of $a_n({\hat{\theta }}^{\ast }-\hat{\theta })$ is $a_n(q_{\alpha /2}^{\ast }-\hat{\theta })$ , we can show

$$a_n\left(q_{\alpha /2}^{\ast }-\hat{\theta }\right)\stackrel{P}{\to }{q}_{\alpha /2}^{\xi }\left(\alpha /2\text{- t h q u a n t i l e o f}\xi \right)$$

Thus

$$\begin{array}{l}\mathbb{P}[\theta \in {\mathrm{C}}^{\mathrm{pc}}]\\ =\mathbb{P}\left[-a_n\left(q_{\alpha /2}^{\ast }-\hat{\theta }\right)\ge a_n(\hat{\theta }-\theta )\ge -a_n\left(q_{1-\alpha /2}^{\ast }-\hat{\theta }\right)\right]\\ \to \mathbb{P}\left[-q_{\alpha /2}^{\xi }\ge \xi \ge -q_{1-\alpha /2}^{\xi }\right]\\ =H\left(-q_{\alpha /2}^{\xi }\right)-H\left(-q_{1-\alpha /2}^{\xi }\right)\\ =H\left(q_{1-\alpha /2}^{\xi }\right)-H\left(q_{\alpha /2}^{\xi }\right)(\text{b y})\\ =1-\alpha \end{array}$$

Percentile-t interval

• We can make better CI than Cpc which does not require symmetry and theoretically more accurate
• Consider t-statistic

$$T=\frac{\hat{\theta }-\theta }{s(\hat{\theta })}$$

Let ${\mathbf{q}}_{n,\alpha }^T$ be the $\alpha .$ -th quantile of $\begin{array}{r}T=\frac{\hat{\theta }-\theta }{s(\hat{\theta })}\end{array}$
• If we know ${\mathbf{q}}_{n,\alpha }^T$ , then

$$\begin{array}{l}1-\alpha =\mathbb{P}\left[q_{n,\alpha /2}^T\le T\le q_{n,1-\alpha /2}^T\right]\\ =\mathbb{P}\left[q_{n,\alpha /2}^T\le \frac{\hat{\theta }-\theta }{s(\hat{\theta })}\le q_{n,1-\alpha /2}^T\right]\\ =\mathbb{P}[\hat{\theta }-s(\hat{\theta })q_{n,\alpha /2}^T\ge \theta \ge \hat{\theta }-s(\hat{\theta })q_{n,1-\alpha /2}^T]\end{array}$$

• Thus (infeasible) exact CI is given by

$${\mathrm{C}}^{\mathrm{ideal}}=[\hat{\theta }-s(\hat{\theta })q_{n,1-\alpha /2}^T,\hat{\theta }-s(\hat{\theta })q_{n,\alpha /2}^T]$$

Estimate ${\mathbf{q}}_{n,\alpha }^T$ by $q_{n,\alpha }^{T\ast }$ , which is $\alpha .$

$$T_{(1)}^{\ast }\le T_{(2)}^{\ast }\le \cdots \le T_{(B)}^{\ast }$$

where $\begin{array}{r}{T}_b^{\ast }=\frac{{\hat{\theta }}_b^{\ast }-\hat{\theta }}{s({\hat{\theta }}_b^{\ast })}(s({\hat{\theta }}_b^{\ast })\end{array}$ is s.e. of ${\hat{\theta }}_b^{\ast }$ computed by b-th resample)

Then percentile-t bootstrap CI is

$${\mathrm{C}}^{\mathrm{pt}}=[\hat{\theta }-s(\hat{\theta })q_{n,1-\alpha /2}^{T\ast },\hat{\theta }-s(\hat{\theta })q_{n,\alpha /2}^{T\ast }]$$

which does not require symmetry of ξ

Remark: Higher order refinement

• Indeed percentile-t CI, Cpt, is more accurate than other methods in the sense that

$$\mathbb{P}[\theta \in {\mathrm{C}}^{\mathrm{pt}}]=1-\alpha +O(n^{-1})$$

• In contrast, percentile CI, ${\mathrm{C}}^{\mathrm{pc}}$ , or asymptotic CI (say Casy) exhibit

$$\mathbb{P}[\theta \in {\mathrm{C}}^{\mathrm{pc}}]=1-\alpha +O\left(n^{-1/2}\right)$$ $$\mathbb{P}[\theta \in C^{\text{a s y}}]=1-\alpha +O\left(n^{-1/2}\right)$$

• All CIs are asymptotically valid (in the sense of $\mathsf{Pr}[\theta \in \mathrm{C}\cdot ]1-\alpha )$ but approximation error of ${\mathrm{C}}^{\mathrm{pt}}$ has smaller order
See Hansen, Ch. 10.20 for detail

Bootstrap test

• Consider testing H0 : θ = c against H1 : $\theta \ne c$ for scalar θ
For t-statistic $\begin{array}{r}T=\frac{\hat{\theta }-c}{s(\hat{\theta })}\end{array}$ , bootstrap counterpart is

$$T^{\ast }=\frac{{\hat{\theta }}^{\ast }-\hat{\theta }}{s({\hat{\theta }}^{\ast })}$$

• Note: $T^{\ast }$ should be centered at $\hat{\theta }$ , not c. Because θˆ is true value in bootstrap world
• Bootstrap estimate for critical value is obtained by $(1-\alpha )\cdot$ -th quantile $q_{1-\alpha }^{\ast }$ of $|T^{\ast }{|}_{(1)}\le \cdots \le |T^{\ast }{|}_{(B)}$
Reject H0 if $|\tau |\ge q_{1-\alpha }^{\ast }$

Bootstrap p-value

We can also estimate p-value by bootstrap
• Recall that p-value is defined as

$$p=1-G_n(|T|)$$

where $G_n(\cdot )$ is null distribution of |T |

Thus bootstrap estimate for p is

$$p^{\ast }=1-G_n^{\ast }(|T|)$$

where $G_n^{\ast }(\cdot )$ is bootstrap distribution of $|T^{\ast }|$

• By bootstrap algorithm $p^{\ast }$ is obtained as

$$p_B^{\ast }=\frac{1}{B}\sum _{b=1}^B\mathbb{I}\{|T_b^{\ast }|>|T|\}$$

Bootstrap bias estimation

• Let $\hat{\theta }$ be an estimator of θ. We are interested in evaluation of bias $\mathbb{E}[\hat{\theta }-\theta ]$
• Let $T_n=\hat{\theta }-\theta$ . Then bias is written as E[Tn]
• Bootstrap counterpart of $T_n$ is

$$T_n^{\ast }={\hat{\theta }}^{\ast }-\hat{\theta }$$

and bootstrap counterpart of E[Tn] is

$${\mathbb{E}}^{\ast }\left[T_n^{\ast }\right]$$

• Based on ${\hat{\theta }}_b^{\ast }$ for $b=1,\dots ,B,{\mathbb{E}}^{\ast }[T_n^{\ast }]$ ${\mathbb{E}}^{\ast }[T_n^{\ast }]$ can be estimated by

$$\frac{1}{B}\sum _{b=1}^B{T}_{nb}^{\ast }=\frac{1}{B}\sum _{b=1}^B{\hat{\theta }}_b^{\ast }-\hat{\theta }\equiv \overline{{\hat{\theta }}^{\ast }}-\hat{\theta }$$

• Given the estimated bias $\overline{{\hat{\theta }}^{\ast }}-\hat{\theta }$ , bias corrected estimator is obtained as

$$\hat{\theta }-(\overline{{\hat{\theta }}^{\ast }}-\hat{\theta })=2\hat{\theta }-\overline{{\hat{\theta }}^{\ast }}$$

Bootstrap MSE estimation

• Similarly MSE $\mathbb{E}[(\hat{\theta }-\theta ])]$ of θˆ can be estimated by

$${\mathbb{E}}^{\ast }[({\hat{\theta }}^{\ast }-\hat{\theta }{)}^2]$$

which is estimated by simulation as

$$\frac{1}{B-1}\sum _{b=1}^B({\hat{\theta }}_b^{\ast }-\hat{\theta }{)}^2$$

3. Bootstrap for OLS and GMM

Bootstrap

• Consider projection model

$$Y=X^{\prime }\beta +e,\mathbb{E}[Xe]=0$$

• Projection coefficient $\beta =\mathbb{E}[X{X}^{\prime }{]}^{-1}\mathbb{E}[XY]$ is estimated by OLS

$$\hat{\beta }={\left(\sum _{i=1}^n{X}_i{X}_i^{\prime }\right)}^{-1}\left(\sum _{i=1}^n{X}_i{Y}_i\right)$$

• Based on bootstrap sample $\{(Y_i^{\ast },X_i^{\ast }):i=1,\dots ,n\},$ , bootstrap counterpart of $\hat{\beta }$ is given by (called pairs bootstrap)

$${\hat{\beta }}^{\ast }={\left(\sum _{i=1}^n{X}_i^{\ast }{X}_i^{\ast \prime }\right)}^{-1}\left(\sum _{i=1}^n{X}_i^{\ast }{Y}_i^{\ast }\right)$$

Bootstrap CI and test are applicable

Bootstrap for OLS: Regression model

• Consider regression model

$$Y=X^{\prime }\beta +e,\mathbb{E}[e|X]=0$$

• Pairs bootstrap is still valid. But to improve precision, want to impose conditional moment restriction $\mathbb{E}[e|X]=0$ in the bootstrap world
• One way is to hold Xi fixed and draw $e_i^{\ast }$ to satisfy conditional moment restriction, that is

$$Y_i^{\ast }=X_i^{\prime }\hat{\beta }+e_i^{\ast }$$ $$e_i^{\ast }={\hat{e}}_i{\xi }_i^{\ast }$$

where ˆei is OLS residual and ${\xi }_j^{\ast }$ is iid draws (by computer) with $\mathbb{E}[{\xi }_i]=0$ and var[ξi ] = 1

It satisfies ${\mathbb{E}}^{\ast }[e_i^{\ast }|X_i]=0\mathrm{and}{\mathbb{E}}^{\ast }[e_i^{\ast 2}|X_i]={\hat{e}}_i^2$ ${\mathbb{E}}^{\ast }[e_i^{\ast }|X_i]=0$

Bootstrap for GMM

• Consider moment restriction model

$$\mathbb{E}[g(Z,{\theta }_0)]=0$$

• For just identified case (dim g = dim θ), all results above apply
• For over identified case (dim g > dim θ), there is some issue
• In bootstrap world, empirical distribution $F_n$ plays role of population, so bootstrap counterpart of $\mathbb{E}[g(Z,{\theta }_0)]$ is

$$\bar{g}(\hat{\theta })=\frac{1}{n}\sum _{i=1}^{n}g(Z_i,\hat{\theta })$$

where θˆ is GMM estimator, but generally $\bar{g}(\hat{\theta })\ne 0$

Confidence interval

Let $\{Z_i^{\ast }{\}}_{i=1}^n$ be bootstrap resample from $F_n$ . Bootstrap counterpart of GMM estimator $\hat{\theta }$ is

$${\hat{\theta }}^{\ast }=\arg \underset{\theta }{\min }{\bar{g}}^{\ast }(\theta {)}^{\prime }{\hat{\Omega }}^{\ast -1}{\bar{g}}^{\ast }(\theta )$$

where $\begin{array}{r}{\bar{g}}^{\ast }(\theta )=\frac{1}{n}{\sum }_{i=1}^{n}g(Z_i^{\ast },\theta )\end{array}$ and ${\hat{\Omega }}^{\ast -1}$ is optimal weight based on $\{Z_i^{\ast }{\}}_{i=1}^n$

• All methods based on ${\hat{\theta }}^{\ast }$ are asymptotically valid
• However, in order to achieve higher-order refinement, we need to modify bootstrap method

Recentered bootstrap

• Hall & Horowitz (1996) suggested to compute bootstrap counterpart of $\hat{\theta }$ by

$${\hat{\theta }}_r^{\ast }=\arg \underset{\theta }{\min }\{{\bar{g}}^{\ast }(\theta )-\bar{g}(\hat{\theta }){\}}^{\prime }{\hat{\Omega }}^{\ast -1}\{{\bar{g}}^{\ast }(\theta )-\bar{g}(\hat{\theta }){\}}^{\prime }$$

i.e. use “recentered” moments $\{g(Z_i^{\ast },\theta )-\bar{g}(\hat{\theta })\}$ to do GMM

• Percentile-t CI based on ${\hat{\theta }}_r^{\ast }$ achieves higher-order refinement

Specification test

• Test validity of overidentified moment restrictions

$${\mathbb{H}}_0:\mathbb{E}[g(Z,\theta )]=0\text{f o r s o m e}\theta \in \Theta$$ $${\mathbb{H}}_1:\mathbb{E}[g(Z,\theta )]\ne 0\text{f o r a l l}\theta \in \Theta$$

. Test statistic (called J-statistic)

$$\begin{array}{l}J=n{\min }_{\theta \in \Theta }\bar{g}(\theta {)}^{\prime }{\hat{\Omega }}^{-1}\bar{g}(\theta )\\ =n\bar{g}(\hat{\theta }{)}^{\prime }{\hat{\Omega }}^{-1}\bar{g}(\hat{\theta })\end{array}$$

• To approximate distribution of J by bootstrap, we should use

$$J^{\ast }=n\underset{\theta \in \Theta }{\min }\{{\bar{g}}^{\ast }(\theta )-\bar{g}(\hat{\theta }){\}}^{\prime }{\hat{\Omega }}^{\ast -1}\{{\bar{g}}^{\ast }(\theta )-\bar{g}(\hat{\theta })\}$$

If we use $J^{\ast }$ without recentering, it fails to approximate distribution of $J$ under H0