Residuals, Error Terms, and Disturbance Terms
I have studied econometrics and often use econometric methods in my research. However, I must admit that I have reached this point without being very confident in the distinction between the terms 'residuals,' 'error terms,' and 'disturbance terms.' Although I have been curious about it, I have been using them somewhat interchangeably until now, so I would like to organize my understanding once and for all.
This time, I learned about it through a dialogue with generative AI. Finally, I asked it to summarize the information with a focus on public economics, my field of expertise. The following is the result. I am leaving this here as a memo.
In regression analysis, the terms 'error term,' 'disturbance term,' and 'residual' are often confused as similar concepts, but they each play different roles. First, an error term or disturbance term in a theoretical model includes all factors that cannot be explained by the model and cannot be observed. In many econometric contexts, the two are used almost synonymously, but 'error term' is a general and comprehensive expression, while 'disturbance term' is often used explicitly when emphasizing stochastic exogenous shocks in theoretical models or in contexts where errors are structurally decomposed, such as in Stochastic Frontier Analysis (SFA), which will be discussed later. On the other hand, a residual is an 'observed deviation' calculated from actual data after estimation, which can be handled directly in empirical analysis. For this reason, while the assumptions and the core of the problems in econometrics concern the error (disturbance) term, actual testing and diagnostics are performed through residuals.
This organization is important for understanding problems such as endogeneity, heteroskedasticity, and serial correlation. Endogeneity refers to a state where explanatory variables and the error (disturbance) term are correlated (Cov(X, u) ≠ 0), in which case the OLS estimator loses its unbiasedness and consistency. The important point is that because the error term is unobservable, it is difficult to test this correlation directly. In tests using residuals, due to the properties of OLS, the correlation between explanatory variables and residuals is constructed to be zero, so endogeneity is not detected on the surface. Therefore, it is necessary to verify it indirectly through the comparison of different estimators, such as the instrumental variable method or the Hausman test.
In contrast, heteroskedasticity and serial correlation are problems concerning the variance or covariance structure of the error (disturbance) term. As long as the assumption that the conditional mean is zero (E[u|X]=0) is maintained, the OLS estimator remains unbiased and consistent. However, if these problems exist, the assumptions regarding the variance structure of the error term are violated, making the calculation of standard errors incorrect. As a result, inferences such as t-values, p-values, and confidence intervals become unreliable. Therefore, the practical issue is not the estimated value itself, but the judgment of its significance.
In this regard, the role of robust standard errors can be understood. Robust standard errors are a method to correct the bias in standard errors by directly estimating the variance using residuals without making specific assumptions about the variance structure of the error term. Since it does not change the coefficient estimates themselves but only corrects the variance, it is effective against heteroskedasticity and serial correlation.
Furthermore, in the evaluation of estimators, the concept of efficiency is important in addition to unbiasedness and consistency. Efficiency refers to the smallness of the variance of an estimator, and the estimator with the smallest variance under the same conditions is considered the most efficient. According to the Gauss-Markov theorem, under the assumption that the error terms are homoskedastic and uncorrelated, OLS is the most efficient (BLUE) among linear unbiased estimators. However, if heteroskedasticity or serial correlation exists, this condition is violated, so OLS, while still unbiased, is no longer the most efficient.
On the other hand, the instrumental variable method (IV estimator) used to address endogeneity has the property of being consistent but inefficient, meaning it has a large variance. Thus, the expression 'consistent but inefficient' means that while it converges to the correct value as the sample size increases, the estimates have large variations in finite samples. The Hausman test uses this property to test the difference between OLS and IV estimates; if the difference between the two is statistically significant, the existence of endogeneity is suggested.
To summarize the above, endogeneity is a problem related to the mean structure of the error (disturbance) term and introduces bias into the estimates themselves. On the other hand, heteroskedasticity and serial correlation are problems related to the variance structure of the error (disturbance) term; while the estimates are maintained, the inferences are distorted. Therefore, the former requires a change in estimation method (such as IV), while the latter is effectively addressed by correcting standard errors (robust estimation).
In this way, many problems in econometrics can be understood uniformly as differences in assumptions regarding the properties of the error (disturbance) term. Distinguishing between theoretical error terms and empirical residuals while properly grasping their roles is essential for evaluating the validity of empirical analysis.
I have copied the output of the generative AI, although it has become quite long. By the way, I had a different AI check it as well, and while it pointed out some issues with the rigor of the details, it evaluated it as generally appropriate.
Indeed, these are all things I was taught in econometrics classes long ago, and I use them somewhat intuitively in practice. However, I had not organized my thoughts to this extent. In particular, the distinction between error terms and residuals—that error terms are theoretical errors and residuals are errors calculated from model estimation—was something I vaguely recognized, but it has now become clear.