7 Testing
Section 6 used point estimates, standard errors, and critical values to construct confidence intervals.
Hypothesis tests use the same ingredients to evaluate a claim about a population parameter.
7.1 Statistical Hypotheses
Let \theta denote a population parameter and let \theta_0 be a specified value.
A two-sided hypothesis test considers H_0:\theta=\theta_0 \qquad\text{against}\qquad H_1:\theta\neq\theta_0. \tag{7.1}
The null hypothesis H_0 is the claim being tested.
The alternative hypothesis H_1 describes the departures from H_0 that the test is designed to detect.
The parameter \theta may be a regression coefficient, a marginal effect, or another population quantity.
Test Decisions
A test statistic measures how strongly the sample disagrees with H_0.
Its null distribution describes the values it would take across repeated samples if H_0 were true.
A rejection region \mathcal R_\alpha contains values that are unlikely under H_0: \begin{aligned} \text{reject }H_0 &\quad\text{if }T\in\mathcal R_\alpha,\\ \text{fail to reject }H_0 &\quad\text{if }T\notin\mathcal R_\alpha. \end{aligned}
The significance level \alpha controls the probability of rejecting a true null hypothesis.
For the large-sample tests in this section, \lim_{n\to\infty} P(\text{reject }H_0\mid H_0\text{ is true}) =\alpha.
Such a test has asymptotic size \alpha.
Rejecting H_0 means that the sample provides evidence against it at the selected significance level.
Failing to reject H_0 does not prove that H_0 is true.
- A Type I error is the rejection of a true null hypothesis.
- A Type II error is the failure to reject a false null hypothesis.
- The power at a particular alternative is the probability of rejecting H_0 when that alternative is true.
The significance level controls the Type I error probability.
Larger samples generally increase power against a fixed alternative.
7.2 Testing One Restriction
Let \widehat\theta estimate \theta and let \mathrm{se}_{HC1}(\widehat\theta) be its HC1 standard error.
For Equation 7.1, define the standardized distance T :=\frac{\widehat\theta-\theta_0} {\mathrm{se}_{HC1}(\widehat\theta)}. \tag{7.2}
Under H_0 and the large-sample conditions from Section 6, T\overset{d}{\to}\mathcal N(0,1).
Large positive and negative values of T provide evidence against the two-sided null.
We reject H_0 at significance level \alpha if |T|>z_{1-\alpha/2}. \tag{7.3}
For a 5% test, the critical values are approximately -1.96 and 1.96.
Class Size Example
We return to Model (2) from Sections 5 and 6: \mathrm{score}_i =\beta_1+\beta_2\mathrm{STR}_i +\beta_3\mathrm{english}_i +\beta_4\mathrm{lunch}_i+u_i.
Consider the hypothesis that the population coefficient on the student-teacher ratio is zero: H_0:\beta_2=0 \qquad\text{against}\qquad H_1:\beta_2\neq0.
The estimate and its HC1 standard error are -0.998 and 0.270.
Therefore, T=\frac{-0.998-0}{0.270}=-3.696.
Because |T|>1.96, we reject H_0 at the 5% level.
Within this model, this is evidence of a mean test-score difference between districts with different student-teacher ratios but the same percentages of English learners and students eligible for subsidized lunch.
It does not by itself establish a causal effect.
For a right-sided test, the hypotheses divide the parameter space as H_0:\theta\leq\theta_0 \qquad\text{against}\qquad H_1:\theta>\theta_0.
We reject when T>z_{1-\alpha}. The approximate p-value is 1-\Phi(T_{\mathrm{obs}}).
For a left-sided test, H_0:\theta\geq\theta_0 \qquad\text{against}\qquad H_1:\theta<\theta_0.
We reject when T<-z_{1-\alpha}. The approximate p-value is \Phi(T_{\mathrm{obs}}).
The direction of a one-sided test should be selected before examining the estimate.
7.3 p-Values
The p-value measures how extreme the observed test statistic is under the null hypothesis.
For the two-sided test in Equation 7.1, p =P\bigl(|Z|\geq|T_{\mathrm{obs}}|\bigr) =2\left[1-\Phi(|T_{\mathrm{obs}}|)\right], \qquad Z\sim\mathcal N(0,1), \tag{7.4} where T_{\mathrm{obs}} is the value calculated from the observed sample and \Phi is the standard normal CDF.
The decision can equivalently be written as \begin{aligned} \text{reject }H_0 &\quad\text{if }p<\alpha,\\ \text{fail to reject }H_0 &\quad\text{if }p\geq\alpha. \end{aligned}
For the class-size example, p=2\left[1-\Phi(3.696)\right] \approx0.0002.
The p-value is calculated under the assumption that H_0 is true.
It is not the probability that H_0 is true.
It also does not measure the size, practical importance, or causal validity of an estimated effect.
For example, a p-value of 0.04 means that the reference distribution assigns 4% probability to test statistics at least as extreme as the observed one.
Relation to Confidence Intervals
Suppose the test and confidence interval use the same standard error and critical value.
Then p<\alpha \quad\Longleftrightarrow\quad \theta_0\notin I_{1-\alpha}. \tag{7.5}
The HC1 interval for the class-size coefficient from Section 6 is [-1.528,-0.469].
It excludes zero but contains -1:
| Hypothesis | T | p-value | Decision at 5% |
|---|---|---|---|
| H_0:\beta_2=0 | -3.696 | 0.0002 | Reject H_0 |
| H_0:\beta_2=-1 | 0.006 | 0.995 | Fail to reject H_0 |
Thus, a hypothesis test need not concern a zero coefficient.
7.4 Statistical Significance
A result is statistically significant at level \alpha when the corresponding null hypothesis is rejected at that level.
Regression tables often use stars for tests of H_0:\beta_j=0:
| Stars | p-value |
|---|---|
*** |
p<0.01 |
** |
0.01\leq p<0.05 |
* |
0.05\leq p<0.10 |
Stars provide a compact summary of p-values.
They do not show the magnitude or uncertainty of the estimate, whether the model is credible, or whether an association is causal.
Point estimates and confidence intervals should therefore remain central when reporting results.
7.5 Testing Several Restrictions
Some hypotheses impose several restrictions at the same time.
The general linear hypotheses are H_0:\boldsymbol R\boldsymbol\beta=\boldsymbol r \qquad\text{against}\qquad H_1:\boldsymbol R\boldsymbol\beta\neq\boldsymbol r, \tag{7.6} where
- \boldsymbol R is a q\times k restriction matrix with rank q,
- \boldsymbol r is a q\times1 vector of hypothesized values,
- and q is the number of restrictions.
The estimated distance from the restrictions is \widehat{\boldsymbol d} :=\boldsymbol R\widehat{\boldsymbol\beta}-\boldsymbol r.
Its estimated covariance matrix is \widehat{\mathrm{Var}} (\widehat{\boldsymbol d}) =\boldsymbol R \widehat{\mathrm{Var}}_{HC1} (\widehat{\boldsymbol\beta}) \boldsymbol R'.
Wald Test
The Wald statistic is the squared, standardized distance from the restrictions: W :=\widehat{\boldsymbol d}' \left[ \boldsymbol R \widehat{\mathrm{Var}}_{HC1} (\widehat{\boldsymbol\beta}) \boldsymbol R' \right]^{-1} \widehat{\boldsymbol d}. \tag{7.7}
Under H_0 and the large-sample conditions from Section 6, W\overset{d}{\to}\chi_q^2.
The chi-squared distribution was defined in Section 6.
It takes only nonnegative values. Large values provide evidence against H_0.
We reject at significance level \alpha if W>\chi^2_{q,1-\alpha}.
The corresponding approximate p-value is p=1-F_{\chi_q^2}(W).
For one linear restriction using the same covariance estimator, W=T^2.
The Wald test and the corresponding two-sided test therefore agree.
Class Size Slopes for Two Groups
Let \mathrm{HiEL}_i equal one for districts with at least 10% English learners and zero otherwise.
Consider the interaction model \mathrm{score}_i =\beta_1+\beta_2\mathrm{STR}_i +\beta_3\mathrm{HiEL}_i +\beta_4\mathrm{STR}_i\mathrm{HiEL}_i+u_i.
Code
CASchools$HiEL = as.numeric(CASchools$english >= 10)
group_fit = lm(score ~ STR * HiEL, data = CASchools)
group_vcov = sandwich::vcovHC(group_fit, type = "HC1")The class-size slope is \beta_2 when \mathrm{HiEL}_i=0 and \beta_2+\beta_4 when \mathrm{HiEL}_i=1.
Each slope compares districts within the same English-learner group, not necessarily with exactly the same percentage of English learners.
The table reports HC1 standard errors in parentheses.
| (1) | |
|---|---|
| * p < 0.1, ** p < 0.05, *** p < 0.01 | |
| Student-teacher ratio | -0.968 |
| (0.589) | |
| High English-learner share | 5.639 |
| (19.515) | |
| STR × high English-learner share | -1.277 |
| (0.967) | |
| Num.Obs. | 420 |
Neither \widehat\beta_2 nor \widehat\beta_4 is individually significant at the 5% level.
Having no class-size association in either group requires H_0:\beta_2=0 \qquad\text{and}\qquad \beta_4=0.
In the notation of Equation 7.6, \boldsymbol R =\begin{pmatrix} 0&1&0&0\\ 0&0&0&1 \end{pmatrix}, \qquad \boldsymbol r =\begin{pmatrix}0\\0\end{pmatrix}.
Code
group_wald = car::linearHypothesis(
group_fit,
c("STR = 0", "STR:HiEL = 0"),
vcov. = group_vcov,
test = "Chisq"
)
round(c(
Wald = group_wald$Chisq[2],
df = group_wald$Df[2],
p_value = group_wald$`Pr(>Chisq)`[2]
), 4) Wald df p_value
11.2762 2.0000 0.0036 The Wald statistic is 11.276 and its p-value is 0.0036.
We reject the joint null at the 1% level.
The joint test accounts for the covariance between the coefficient estimates.
Separate coefficient tests do not answer a joint hypothesis.
As before, the result concerns conditional associations. A causal interpretation requires a credible causal argument for the underlying comparison.
The main results in this section use HC1 standard errors with standard normal and \chi^2 reference distributions.
Under H_0 and the homoskedastic Gaussian regression model from Section 6, calculating Equation 7.2 with the homoskedastic standard error gives T_{hom}\mid\boldsymbol X\sim t_{n-k}.
For q linear restrictions, let W_{hom} denote Equation 7.7 calculated with the homoskedastic covariance matrix.
Then F:=\frac{W_{hom}}{q} \ \Big|\ \boldsymbol X \sim F_{q,n-k}.
These results give exact conditional tests under the Gaussian and homoskedasticity assumptions.
Software sometimes compares robust T and W/q statistics with t and F critical values as a finite-sample convention.
Under heteroskedasticity, this convention is not an exact finite-sample result.
7.6 Testing Workflow
For a coefficient or effect:
- State the null and alternative hypotheses.
- Calculate the estimate and a covariance estimator appropriate for the sampling structure.
- Standardize the distance from the hypothesized value.
- Use the corresponding reference distribution to obtain a critical value or p-value.
- Report the estimate, standard error, confidence interval, and test result together.
- Interpret the population parameter actually tested.
7.7 Regression Diagnostics
After estimating a regression, diagnostics can reveal patterns that the specification may have missed.
They provide warning signs. They cannot prove that the CEF is correctly specified, that exogeneity holds, or that an effect is causal.
Residual Plots
A residual-versus-fitted plot compares \widehat u_i with \widehat Y_i.
- A systematic pattern in the local average around zero may indicate that the fitted function misses part of the conditional mean.
- A changing vertical spread indicates that the conditional variance may depend on the regressors.
The absolute residuals in the right panel make changes in the spread easier to see.
The smooth curves suggest some curvature and a changing residual spread, particularly near the ends of the fitted-value range.
These patterns suggest further investigation. Their absence would not prove that the model is correct.
Goodness of Fit and R^2
For a regression with an intercept, the R-squared is R^2 :=1- \frac{\sum_{i=1}^n\widehat u_i^2} {\sum_{i=1}^n(Y_i-\overline Y)^2}. \tag{7.8}
It measures the share of the observed sample variation in Y_i accounted for by the fitted regression.
For a regression with an intercept, 0\leq R^2\leq1.
For the class-size regression,
Thus, the fitted model accounts for about 77.5% of the sample variation in test scores.
A high R^2 does not establish a correct CEF or a causal relationship.
A low R^2 does not by itself imply that an estimated conditional-mean effect is uninformative.
Moreover, adding regressors cannot reduce the ordinary R^2 when the outcome and estimation sample remain unchanged. Control variables should therefore not be selected by maximizing R^2.
Leverage and Influence
Section 6 defined the leverage value h_{ii}. High leverage means that an observation has an unusual combination of regressor values.
A large residual means that its outcome lies far from its fitted value.
An observation may be influential when removing it would noticeably change the fitted regression. This can occur when high leverage is combined with an unusual outcome.
High leverage alone does not show that an observation is erroneous and is not a reason to delete it.
We should check the observation and its context, and examine whether the main conclusions are sensitive to it.
The Breusch–Pagan test considers the homoskedasticity hypothesis H_0:\mathrm{Var}(u_i\mid\boldsymbol X_i)=\sigma^2.
It uses an auxiliary regression of \widehat u_i^2 on the original regressors.
If R_{aux}^2 is its R-squared, a common version of the test statistic is BP=nR_{aux}^2.
Under H_0 and suitable large-sample conditions, BP has an approximate \chi^2_{k-1} distribution.
lmtest::bptest(class_size_fit)
studentized Breusch-Pagan test
data: class_size_fit
BP = 9.9375, df = 3, p-value = 0.0191The p-value is about 0.019. The test therefore provides evidence that the conditional variance changes with the regressors.
For independent cross-sectional observations, we continue to use HC1 as the practical default whether or not the Breusch–Pagan test rejects.
Failure to reject homoskedasticity does not prove that it holds. Neither the test nor a residual plot repairs a misspecified conditional mean.
Section 8 extends inference and residual diagnostics to dependent observations.