7 Hypothesis Testing
Section 6 used point estimates, standard errors, and critical values to construct confidence intervals.
Hypothesis tests use the same ingredients to evaluate a claim about a population parameter.
7.1 Statistical Hypotheses
Let \theta denote a population parameter and let \theta_0 be a specified value.
A two-sided hypothesis test considers H_0:\theta=\theta_0 \qquad\text{against}\qquad H_1:\theta\neq\theta_0. \tag{7.1}
The null hypothesis H_0 is the claim being tested.
The alternative hypothesis H_1 describes the departures from H_0 that the test is designed to detect.
The parameter \theta may be a regression coefficient, a marginal effect, or another population quantity.
Test Decisions
A test statistic measures how strongly the sample disagrees with H_0.
Its null distribution describes the values it would take across repeated samples if H_0 were true.
A rejection region \mathcal R_\alpha contains values that are unlikely under H_0: \begin{aligned} \text{reject }H_0 &\quad\text{if }T\in\mathcal R_\alpha,\\ \text{fail to reject }H_0 &\quad\text{if }T\notin\mathcal R_\alpha. \end{aligned}
The significance level \alpha controls the probability of rejecting a true null hypothesis.
For the large-sample tests in this section, \lim_{n\to\infty} P(\text{reject }H_0\mid H_0\text{ is true}) =\alpha.
Such a test has asymptotic size \alpha.
Rejecting H_0 means that the sample provides evidence against it at the selected significance level.
Failing to reject H_0 does not prove that H_0 is true.
- A Type I error is the rejection of a true null hypothesis.
- A Type II error is the failure to reject a false null hypothesis.
- The power at a particular alternative is the probability of rejecting H_0 when that alternative is true.
The significance level controls the Type I error probability.
Larger samples generally increase power against a fixed alternative.
7.2 Testing One Restriction
Let \widehat\theta estimate \theta and let \mathrm{se}_{HC1}(\widehat\theta) be its HC1 standard error.
For Equation 7.1, define the standardized distance T :=\frac{\widehat\theta-\theta_0} {\mathrm{se}_{HC1}(\widehat\theta)}. \tag{7.2}
Under H_0 and the large-sample conditions from Section 6, T\overset{d}{\to}\mathcal N(0,1).
Large positive and negative values of T provide evidence against the two-sided null.
We reject H_0 at significance level \alpha if |T|>z_{1-\alpha/2}. \tag{7.3}
For a 5% test, the critical values are approximately -1.96 and 1.96.
Class Size Example
We return to Model (2) from Sections 5 and 6: \mathrm{score}_i =\beta_1+\beta_2\mathrm{STR}_i +\beta_3\mathrm{english}_i +\beta_4\mathrm{lunch}_i+U_i.
Consider the hypothesis that the population coefficient on the student-teacher ratio is zero: H_0:\beta_2=0 \qquad\text{against}\qquad H_1:\beta_2\neq0.
The estimate and its HC1 standard error are -0.998 and 0.270.
Therefore, T=\frac{-0.998-0}{0.270}=-3.696.
Because |T|>1.96, we reject H_0 at the 5% level.
This is evidence of a conditional association in the specified model. It does not by itself establish a causal effect.
For a right-sided test, the hypotheses divide the parameter space as H_0:\theta\leq\theta_0 \qquad\text{against}\qquad H_1:\theta>\theta_0.
We reject when T>z_{1-\alpha}. The approximate p-value is 1-\Phi(T_{\mathrm{obs}}).
For a left-sided test, H_0:\theta\geq\theta_0 \qquad\text{against}\qquad H_1:\theta<\theta_0.
We reject when T<-z_{1-\alpha}. The approximate p-value is \Phi(T_{\mathrm{obs}}).
The direction of a one-sided test should be selected before examining the estimate.
7.3 p-Values
The p-value measures how extreme the observed test statistic is under the null hypothesis.
For the two-sided test in Equation 7.1, p =P\bigl(|Z|\geq|T_{\mathrm{obs}}|\bigr) =2\left[1-\Phi(|T_{\mathrm{obs}}|)\right], \qquad Z\sim\mathcal N(0,1), \tag{7.4} where T_{\mathrm{obs}} is the value calculated from the observed sample and \Phi is the standard normal CDF.
The decision can equivalently be written as \begin{aligned} \text{reject }H_0 &\quad\text{if }p<\alpha,\\ \text{fail to reject }H_0 &\quad\text{if }p\geq\alpha. \end{aligned}
For the class-size example, p=2\left[1-\Phi(3.696)\right] \approx0.0002.
The p-value is calculated under the assumption that H_0 is true.
It is not the probability that H_0 is true.
It also does not measure the size, practical importance, or causal validity of an estimated effect.
For example, a p-value of 0.04 means that the reference distribution assigns 4% probability to test statistics at least as extreme as the observed one.
Relation to Confidence Intervals
Suppose the test and confidence interval use the same standard error and critical value.
Then p<\alpha \quad\Longleftrightarrow\quad \theta_0\notin I_{1-\alpha}. \tag{7.5}
The HC1 interval for the class-size coefficient from Section 6 is [-1.528,-0.469].
It excludes zero but contains -1:
| Hypothesis | T | p-value | Decision at 5% |
|---|---|---|---|
| H_0:\beta_2=0 | -3.696 | 0.0002 | Reject H_0 |
| H_0:\beta_2=-1 | 0.006 | 0.995 | Fail to reject H_0 |
Thus, a hypothesis test need not concern a zero coefficient.
7.4 Statistical Significance
A result is statistically significant at level \alpha when the corresponding null hypothesis is rejected at that level.
Regression tables often use stars for tests of H_0:\beta_j=0:
| Stars | p-value |
|---|---|
*** |
p<0.01 |
** |
0.01\leq p<0.05 |
* |
0.05\leq p<0.10 |
Stars provide a compact summary of p-values.
They do not show the magnitude or uncertainty of the estimate, whether the model is credible, or whether an association is causal.
Point estimates and confidence intervals should therefore remain central when reporting results.
7.5 Testing Several Restrictions
Some hypotheses impose several restrictions at the same time.
The general linear hypotheses are H_0:\boldsymbol R\boldsymbol\beta=\boldsymbol r \qquad\text{against}\qquad H_1:\boldsymbol R\boldsymbol\beta\neq\boldsymbol r, \tag{7.6} where
- \boldsymbol R is a q\times k restriction matrix with rank q,
- \boldsymbol r is a q\times1 vector of hypothesized values,
- and q is the number of restrictions.
The estimated distance from the restrictions is \widehat{\boldsymbol d} :=\boldsymbol R\widehat{\boldsymbol\beta}-\boldsymbol r.
Its estimated covariance matrix is \widehat{\mathrm{Var}} (\widehat{\boldsymbol d}) =\boldsymbol R \widehat{\mathrm{Var}}_{HC1} (\widehat{\boldsymbol\beta}) \boldsymbol R'.
Wald Test
The Wald statistic is the squared, standardized distance from the restrictions: W :=\widehat{\boldsymbol d}' \left[ \boldsymbol R \widehat{\mathrm{Var}}_{HC1} (\widehat{\boldsymbol\beta}) \boldsymbol R' \right]^{-1} \widehat{\boldsymbol d}. \tag{7.7}
Under H_0 and the large-sample conditions from Section 6, W\overset{d}{\to}\chi_q^2.
The \chi_q^2 distribution is the distribution of the sum of squares of q independent standard normal variables.
It takes only nonnegative values. Large values provide evidence against H_0.
We reject at significance level \alpha if W>\chi^2_{q,1-\alpha}.
The corresponding approximate p-value is p=1-F_{\chi_q^2}(W).
For one linear restriction using the same covariance estimator, W=T^2.
The Wald test and the corresponding two-sided test therefore agree.
Class Size Slopes for Two Groups
Let \mathrm{HiEL}_i equal one for districts with at least 10% English learners and zero otherwise.
Consider the interaction model \mathrm{score}_i =\beta_1+\beta_2\mathrm{STR}_i +\beta_3\mathrm{HiEL}_i +\beta_4\mathrm{STR}_i\mathrm{HiEL}_i+U_i.
Code
CASchools$HiEL = as.numeric(CASchools$english >= 10)
group_fit = lm(score ~ STR * HiEL, data = CASchools)
group_vcov = sandwich::vcovHC(group_fit, type = "HC1")The class-size slope is \beta_2 when \mathrm{HiEL}_i=0 and \beta_2+\beta_4 when \mathrm{HiEL}_i=1.
The table reports HC1 standard errors in parentheses.
| (1) | |
|---|---|
| * p < 0.1, ** p < 0.05, *** p < 0.01 | |
| Student-teacher ratio | -0.968 |
| (0.589) | |
| High English-learner share | 5.639 |
| (19.515) | |
| STR × high English-learner share | -1.277 |
| (0.967) | |
| Num.Obs. | 420 |
Neither \widehat\beta_2 nor \widehat\beta_4 is individually significant at the 5% level.
Having no class-size association in either group requires H_0:\beta_2=0 \qquad\text{and}\qquad \beta_4=0.
In the notation of Equation 7.6, \boldsymbol R =\begin{pmatrix} 0&1&0&0\\ 0&0&0&1 \end{pmatrix}, \qquad \boldsymbol r =\begin{pmatrix}0\\0\end{pmatrix}.
Code
group_wald = car::linearHypothesis(
group_fit,
c("STR = 0", "STR:HiEL = 0"),
vcov. = group_vcov,
test = "Chisq"
)
round(c(
Wald = group_wald$Chisq[2],
df = group_wald$Df[2],
p_value = group_wald$`Pr(>Chisq)`[2]
), 4) Wald df p_value
11.2762 2.0000 0.0036 The Wald statistic is 11.276 and its p-value is 0.0036.
We reject the joint null at the 1% level.
The joint test accounts for the covariance between the coefficient estimates.
Separate coefficient tests do not answer a joint hypothesis.
As before, the result concerns conditional associations unless the causal identification conditions hold.
The main results in this section use HC1 standard errors with standard normal and \chi^2 reference distributions.
Under H_0 and the homoskedastic Gaussian regression model from Section 6, calculating Equation 7.2 with the homoskedastic standard error gives T_{hom}\mid\boldsymbol X\sim t_{n-k}.
For q linear restrictions, let W_{hom} denote Equation 7.7 calculated with the homoskedastic covariance matrix.
Then F:=\frac{W_{hom}}{q} \ \Big|\ \boldsymbol X \sim F_{q,n-k}.
These results give exact conditional tests under the Gaussian and homoskedasticity assumptions.
Software sometimes compares robust T and W/q statistics with t and F critical values as a finite-sample convention.
Under heteroskedasticity, this convention is not an exact finite-sample result.
7.6 Testing Workflow
For a coefficient or effect:
- State the null and alternative hypotheses.
- Calculate the estimate and a covariance estimator appropriate for the sampling structure.
- Standardize the distance from the hypothesized value.
- Use the corresponding reference distribution to obtain a critical value or p-value.
- Report the estimate, standard error, confidence interval, and test result together.
- Interpret the population parameter actually tested.
Section 8 extends inference to dependent observations.