The Out-of-Sample Performance of Stochastic Methods in Forecasting Age-Specific Mortality Rates

 

Save/Print 

Introduction

Mortality is one of the key demographic variables affecting the flow of income and expenditures in pay-as-you-go public retirement programs. Indeed, a combination of population aging and declining fertility rates largely drives the currently projected financial imbalance in the U.S. Social Security system. In recent years, official mortality forecasts in a number of industrialized nations have come under greater scrutiny. The deterministic nature of these projections and the role that expert judgment plays in shaping them are often viewed by academics as sources of contention. Meanwhile, demographers and other social scientists are increasingly turning to statistical time series techniques to generate mortality forecasts that are consistent with a probabilistic representation of uncertainty.

This paper will evaluate the performance of two alternative stochastic approaches that can be applied to project age-specific mortality rates. Mortality data from 16 industrialized nations is used to carry out an extensive out-of-sample validation exercise comparing actual mortality rates to the pseudo-forecasts generated by the models. This analysis differs from other ex-post assessments published in the literature in two respects: First, in addition to reporting several single-valued aggregate measures of performance, it also investigates how forecast error is distributed across age groups and forecast horizons. Second, this paper is not only concerned with a model's ability to produce accurate point projections, but also with its capacity to generate a realistic depiction of forecast uncertainty in terms of the empirical probability content of its interval forecasts.

The remainder of the paper is structured as follows: an introduction of the two models that are the focus of the investigation; a description of the experimental design, followed by the proposed ex-post validation exercise, and a discussion on the most salient features of the mortality data used in the paper; a presentation on the out-of-sample performance results; and the conclusion.

The Models

Stochastic forecasts are typically generated based on some underlying time-series statistical model. This time series approach often involves a specified random disturbance shock process, as well as a recursive expression that posits current values of the series in question as a function of previous values. Once the model is fit to a particular data set and estimates of its parameters obtained, future values of the series can be produced by iterating the model forward. For simple models, the forecast distribution may be available in closed-form. Otherwise, the researcher can turn to simulation by drawing from the disturbance process to generate random sample paths of forthcoming observations. In either case, the result is not only a single point forecast but an entire probability distribution describing the uncertainty associated with future outcomes.

Modeling and projecting age-specific mortality rates over time is a high-dimensional forecasting problem. Following the taxonomy suggested in Bell (1997), stochastic projection models can be categorized as parametric or nonparametric, although this can be a somewhat artificial distinction. The parametric or curve-fitting approach involves fitting a curve defined by a finite set of time-varying parameters to the mortality data, based on some optimization criterion. The resulting parameter estimates are then treated as a time series that is projected to recover the different paths of the curve into the future (that is, the mortality forecasts). The nonparametric approach relies on principal components analysis to yield a linear transformation of the data, often in terms of an approximation of reduced dimensions (one or a few principal components).

Stochastic projection methods can be further classified as univariate or multivariate depending on whether the generated forecasts take into account the interdependencies across the age groups. The former proceed by individually fitting each age-specific mortality rate to a univariate time series equation. Although the forecasts produced by univariate methods ignore the typically high cross-correlation among the different age series, they do not necessarily perform worse than multivariate models. For instance, in an ex-post validation experiment, Bell (1997) found that a random-walk with drift applied to each age series led to better short-term forecast performance than any of a variety of multivariate approaches. Nonetheless, there are several problems associated with the univariate route. First, while the projections may be more accurate for each individual age group, they can jointly imply unreasonable behavior. In particular, univariate methods can lead to odd shapes in the fairly regular structure of mortality over the entire age profile. Similarly, since this approach ignores the high degree of correlation among the age series, it is unlikely to provide an accurate picture of overall forecast uncertainty.

This paper focuses on the forecast performance of two models. First, the multivariate nonparametric method proposed by Lee and Carter (1992). This model has gained increasing recognition over the years, becoming a benchmark technique to the most recent technical advisory panels to the Social Security Administration, the U.S. Census Bureau, and several agencies around the world. The second model involves one of the approaches suggested by Denton, Feaver, and Spencer (2005). This parametric model fits first-order autoregressive processes to each age group separately. The resulting residuals are then used to estimate the covariance matrix of the multivariate disturbance process driving joint future variation in the age-specific mortality rates.

The Lee-Carter Model

The approach to mortality modeling proposed by Lee and Carter (1992) postulates the logarithm of a set of age-specific mortality rates as the sum of a time-invariant age-specific element and a second component that changes over time. Formally, let M represent the A\times T dimensional matrix of mortality rates with individual elements m_{a,t} denoting the death rate for the population of individuals at age $a$ and time $t.$ Then,

The age-specific set of parameters {\alpha }_a describes the average shape of the log-mortality rates for every age category. The second component is the product of a time-varying index or trend of the general level of mortality k_t and a set of coefficients {\beta }_a that determine both the direction and magnitude by which mortality at every age varies with the index. Notice also that the parameters {\beta }_a and k_t are not uniquely identified, since for any given constant $c,$ an equivalent representation results by using {\beta }_a/c and k_{t}c. Thus, Lee and Carter (1992) suggest imposing the following constraints:

The Lee-Carter model represents a special case of the principal components (PC) analysis applied by Bell and Monsell (1991) to forecast age-specific mortality rates. Intuitively, PC analysis yields an approximation of the $A$ age-specific mortality rates as the linear combination of $p$ "basic elements" or principal components estimated from the data, where typically $p\le A.$ One way to compute the latter is via singular value decomposition (SVD). Specifically, let $M$ define the matrix of centered age profiles obtained by subtracting the $A$ sample logarithmic mortality means from the columns of the matrix $\log (M).$ The singular value decomposition of M yields a representation involving the product of the following three matrices:

Once equation (1) is fitted to the data, the parameter estimates {\widehat{\alpha }}_a and {\widehat{\beta }}_a are taken to be fixed, while the log mortality index {\widehat{k}}_t provides a univariate time series whose future values can be forecasted using standard Box-Jenkins techniques. In most applications, a random-walk with drift is empirically found to yield a suitable fit to {\widehat{k}}_t

The Lee-Carter model yields a parsimonious stochastic approach to mortality forecasting that is easy to implement and often produces reasonable forecasts for all-cause age-specific mortality. The method, however, is not without its limitations. First, a linear trend in the mortality index k_t does not hold empirically in very long data sets. It entails a constant geometric rate of decline for each age-specific mortality

Finally, the Lee-Carter model incorporates uncertainty through a single source (the sampling uncertainty derived from forecasting k_t). It is also possible to accommodate additional uncertainty about the trend in mortality linked to the estimate of the drift parameter \mu , as Lee and Carter (1992) originally discussed. However, this still ignores uncertainty in the estimation of the {\beta }_a coefficients associated with k_t, as well as the error from fitting the model using only the first principal component. Some demographers have criticized the model's interval forecasts as implausibly narrow.

Some Extensions of the Lee-Carter Model

There have been a number of refinements to the Lee-Carter specification. In fact, in their original article, Lee and Carter (1992) addressed the two modifications considered in this paper. In particular, the authors observed that the models' forecasts do not match the initial conditions in the jump-off year (that is, the forecasts are not linked to the actual mortality rates at the end of the base period). One easy way to solve this problem is to set {\alpha }_a equal to the most recently observed logarithmic age-specific rates, instead of their time average. However, Lee and Carter (1992) caution that such an approach might extrapolate features of mortality that are specific to the jump-off year and could have a negative impact on model fit and forecast performance. In subsequent papers, Lee (2000) and Lee and Miller (2002) seem to have reconsidered this position, favoring the modified value of {\alpha }_a for forecasting purposes. Bell (1997), who also supports this bias correction step, finds dramatic improvements in short-term out-of-sample forecast performance when setting {\alpha }_a equal to the logarithm of the age-specific rates in the base year.³

Another improvement to the Lee-Carter model is concerned with the fact that the OLS estimates of its parameters are the values minimizing error in the logarithm of the death rates, rather than the death rates themselves. Consequently, the total number of deaths predicted by the model is not guaranteed to match the observed death counts in the sample. Lee and Carter (1992) propose a second stage reestimation of the mortality index by holding {\widehat{\alpha }}_a and {\widehat{\beta }}_a fixed, while searching for a new estimate {\widehat{k}}_t^{\ast } satisfying the following equation

The first model this paper entertains is a variant of Lee-Carter, incorporating the bias corrections described in the previous paragraphs. Specifically, after some preliminary experimentation, a decision was made to settle on the following estimation approach: first, the model's parameters are computed by applying SVD on the matrix \tilde{M} of centered logarithmic age profiles. Next, {\alpha }_a is set equal to the logarithm of the age-specific rates corresponding to the last period in the sample. Finally, a second stage reestimation of k_t is performed to match total observed and fitted deaths.

A First-Order Autoregressive Approach

In a recent paper, Denton, Feaver, and Spencer (2005) suggest a number of multivariate time-series econometric specifications as alternatives to the Lee-Carter method. One such possibility is to model the first difference of logarithmic mortality \Delta \log (m_{a,t})=\log (m_{a,t})-\log (m_{a,t-1}) as a p^{\text{th}}\text{-} order autoregressive process AR $(p)$:

Future changes in the individual mortality rates are determined by their own past values plus a random disturbance term e_{a,t}. The age-specific series are estimated within a system of seemingly unrelated regression equations (SURE) to accommodate the significant contemporaneous correlation characterizing mortality data. Denton, Feavor, and Spencer (2005) further suggest a second specification, which they refer to as a quasi-vector autoregressive approach QVAR $(p)$

The second model this paper entertains is a variant of equation (13) with p=1 lags. Formally, let m_{a,t}^{\ast } denote the annual rate of improvement in mortality expressed as the negative of the percent change in the central death rate:

The process m_{a,t}^{\ast } can be shown to be covariance stationary if \left|{\varphi }_a\right|<1, with mean and variance respectively equal to

For a covariance stationary process, the mean $h$-step-ahead forecast, conditional on the previous observations is given by

Data and Experimental Design

The data sets used to carry out the ex-post validation exercise proposed in this paper were obtained from the Human Mortality Database (HMD) and consist of mortality rates from 16 industrialized nations for males and females combined.⁵ Wilmoth (2004) documents the methods by which the raw data were converted into mortality rates. The investigation in this paper focuses on period death rates rather than cohort rates. In other words, the mortality rates are indexed by year of occurrence rather than year of birth, so that m_{a,t} denotes mortality at age $a$ occurring in year $t,$ rather than the death rate of individuals aged $a$ born in year $t.$ While analysis of rates on a cohort basis might be preferable, a complete set of cohort mortalities requires a much longer time frame and can involve significant missing data problems.

Formally, the period death rate m_{a,t} is defined as the ratio

Ex-post validation analysis involves using an initial portion of the available data to estimate a set of models that are then used to generate forecasts for the remaining time period. This way, it is possible to compare the models' projections to the actual observations to determine how well the models would have performed in the past. The design of any ex-post validation experiment always requires somewhat arbitrary decisions. For instance, the researcher must select the specific time frame and length over which the behavior of the models should be investigated, the fraction of the data used for estimation purposes, and the evaluation criteria employed to measure forecast performance. The particular objectives of the analysis shape these decisions and constrain the applicability of any conclusions. Table 1 lists the historical period of mortality data available for each of the 16 countries.⁶ The shortest sample corresponds to the United States (1959–2002) with 44 observations, while the longest sample belongs to Sweden (1751–2003), with 253 observations.

This paper uses all available data regardless of potential country-specific concerns about variation in quality, particularly when the estimated mortality rates date back more than one century. This decision is justified by treating the selected stochastic models as general algorithms, whose mechanically-generated forecasts should be tested under multiple scenarios. Furthermore, to make the generated forecasts comparable across countries, the initial jump-off year (the first period to be forecast) is fixed to 1980 in all cases. This particular choice was made based on the shortest available data set, by roughly adhering to two guiding principles: First, for each series there should be at least as many in-sample observations as the length of the forecast horizon. Second, the estimation sample per series should be at least as large as the total number of variables to be projected (21 age groups).

To minimize the impact of the selected jump-off year on the resulting projections, it is common practice in out-of-sample validation exercises to focus on the forecast error corresponding to fixed lead times, using different forecast origins. In other words, for every country, each model is fitted using all observations from the beginning of the series until 1979 and mortality projections generated from 1980 to the end of the data set. Then, the sample is expanded to include the next observation (1980). Upon reestimation, new forecasts are generated from 1981 to the end of the series. This process is repeated until the only period left to forecast is the last available observation. For instance, for each age group in the United States, the projections generated over the various jump-off years (1980, 1981,…, 2001) yield a set of 23 forecasts involving a 1-year horizon, 22 forecasts involving a 2-year period, and eventually a single forecast 23 years ahead. The fourth column of Table 1 shows the size of the longest forecast horizon (from 1980 to the end of each series). By design, the analysis centers on evaluating forecast performance over the short- to medium-range (23 to 25-year horizons in most cases), a fact that is determined by the choice of initial jump-off year, given the small sample sizes of some of the data. The last column in Table 1 presents the total number of projected observations per age group, over all forecast horizons.

Chart 1 displays three-dimensional surfaces, as well as contours of the logarithmic age profile of mortality corresponding to the United States and the United Kingdom. They serve to illustrate a number of features in all-cause mortality common to most nations. One characteristic of the data is the regularity in the shape of mortality over the ages. For any given period, mortality declines smoothly from birth until about ages 10–14, then increases almost linearly for the remaining ages until death. Moreover, in the second half of the twentieth century, the death rates experience a sharp increase associated with motor-vehicle fatalities in the 15–19 age group, often referred to as "the accident hump." Notice also how the surface for the United Kingdom appears far less smooth than the one corresponding to the United States. The former contains a much longer data sample (1841–2003) that includes spikes in mortality associated with the two world wars and the 1918 Spanish influenza pandemic.

Chart 1.
Surfaces and contours of the age profile of logarithmic mortality for the United States and the
United Kingdom

Log mortality age profile (surface)

Log mortality age profile

SOURCE: Author's calculations from the Human Mortality Database.

When modeling mortality, some researchers treat unusual data spikes as outliers by introducing dummy variables to remove their influence. An alternative view is that these observations represent rare but nonetheless potentially recurring shocks, and thus, their exclusion is likely to underestimate true uncertainty. The analysis in this paper subscribes to the latter practice, treating all observations equally. Yet a third possibility, as Lee and Carter (2000) suggest, is to incorporate additional uncertainty in every forecast period due to such events. For example, this can be accomplished by introducing a 1/(T-1) chance of a shock to the mortality index k_t the size of the 1918 influenza pandemic, where $T$ denotes the sample size. Nevertheless, the authors report that this practice has a negligible effect in the resulting projections.

A second characteristic in the age profile of all-cause mortality is a downward trend. That is, while mortality across the age groups maintains its regular shape, it also shifts downward over time. This can be clearly seen in the bottom graphs of Chart 1, which show the logarithm of the age mortality profile at three different points in time for the same two countries. It is evident that the death rates among the various age groups tend to move together. Thus, it is not surprising that a third feature of all-cause mortality involves a high degree of cross-correlation among the rates for different ages.

Table 2 presents the sample correlation between each age series and its immediately adjacent group for all 16 countries. For instance, the top entry in the column of Table 2 corresponding to Austria indicates that the estimated correlation between the age 0 and age 1–4 groups is 0.988. Similarly, the correlation between age groups 90–94 and 95 or older is 0.748 (the last entry in the same column). Evidently, mortality across the ages shows a high degree of positive association. Finally, Table 3 shows the sample standard deviation of the mortality rates corresponding to several age groups. Clearly, there is much more variation in mortality within the older age groups (particularly for the last series age 95 or older), as well as before age 1. Typically, the standard deviation decreases rapidly from a relatively high value at birth (age 0), until it reaches the 10–14 age group. Then, it increases steadily from ages 15–19 to the last series (age 95 or older), where it often attains its highest value.

Before turning to the ex-post validation results presented in the next section, it is important to discuss a number of findings in the literature that are relevant to this paper. Denton, Feavor, and Spencer (2005) use Canadian mortality data from 1926 to 2000 to produce long-term forecasts of life expectancy at birth and ages 65 and 80, based on the specification in equation (13) with p=2 lags. The authors utilize a partially parametric method to generate random variation via a bootstrap procedure. They also implement a fully parametric approach by drawing from a multivariate normal disturbance process, much like the second model entertained in this paper. Although Denton, Feaver, and Spencer (2005) do not conduct an analysis of the out-of-sample forecast performance of these models, they do find the point forecasts generated by the fully parametric approach much closer to the projections of the Lee-Carter method than the official forecasts of the Canada Pension Plan.

Lee and Miller's (2001) ex-post validation analysis also focuses on life expectancy at birth e_0, comparing actual and hypothetical forecast errors in the Lee-Carter model with those of the Social Security Administration (SSA).⁷ Using U.S. data from 1900 to 1998 (with 1921 as the initial jump-off year), the authors find that the empirical distribution of the actual forecast error matches well its hypothetical counterpart within a 10-year period, but deteriorates over time. Generally, the Lee-Carter model tends to underpredict life expectancy, although not by as much as the official SSA projections. In addition, the interval forecasts of e_0 appear to be "too wide" up to the first 50 forecast horizons, while underestimating their hypothetical probability content for longer periods. Lee and Miller (2001) reach similar conclusions in more limited pseudo-forecast experiments using data from Japan, Canada, France, and Sweden.

Finally, Bell (1997) implements an evaluation of the short-term out-of-sample forecast behavior of multiple models using U.S. central death rates for white males and females from 1940 to 1991 (with 1981 as the initial jump-off year). Unlike Lee and Miller (2001), Bell reports forecast error over the entire age profile instead of relying on life expectancy as a single-valued measure of forecast performance. He finds that a univariate random-walk with drift fitted separately to each age group outperforms all of the parametric and nonparametric multivariate approaches considered. Only the Lee-Carter model with the type of bias correction discussed previously yields a similar forecast error to the univariate approach.

Out-of-Sample Forecast Performance

As previously mentioned, ex-post validation analysis provides a means to determine how well a set of models would have performed in the past, by comparing the forecasts generated by the models to the actual observations. This kind of analysis is not without its limitations, and should not be confused with forecasts that are generated in real time. The latter are produced prior to the forecast period, when the future outcome is truly uncertain. The former enjoy the advantage of perfect foresight, and are therefore based on an information set that was not available during the forecast period. Keeping these drawbacks in mind, ex-post validation is still a very valuable tool that cannot be replaced by in-sample goodness-of-fit measures. In particular, ex-post validation provides answers to "what if" type scenarios that are useful in specifying and calibrating models to be used for real time forecasting.

To compare forecast performance among several models, the following elements must be specified a priori: (1) the variables of interest to be projected; (2) the estimators used to measure these variables; and (3) an appropriate criterion to evaluate the variables' forecast performance. Clearly, with respect to the first point, the ultimate object of investigation is the 21 different age-specific mortality rates being modeled simultaneously. This paper looks at both the accuracy of the point projections produced by the models, and the ability of these projections to provide a realistic representation of forecast uncertainty. The means and medians of the generated forecast distributions are presented as two alternative point estimators. On the other hand, the capacity of the models to gauge forecast uncertainty is assessed by the behavior of their interval projections. To this aim, 90-percent confidence interval forecasts are also estimated using the 5th and 95th quantiles of the resulting forecast distributions.

The performance of the point estimates is evaluated using the traditional root mean squared error (RMSE) measure. Conversely, the performance of the interval projections is determined in terms of their empirical probability content (that is, the fraction of times the generated intervals actually include the observed ex-post mortality rates). If the interval forecasts enjoy an empirical probability content that is close to its hypothetical 90 percent level, it is likely that the model does a good job at accommodating the uncertainty associated with its point projections and can be used reliably for inference. However, coverage alone is only part of the picture. Since by design a fixed forecast interval between 0 and 1 covers the entire sample space, it is guaranteed to contain the ex-post mortality rates 100 percent of the time. Yet, such an interval has no practical use for inference, as it does not convey any information not already known a priori. Hence, the average width of the generated forecast intervals is also reported. Clearly, one unequivocal way to rank the interval estimates generated by several models involves the trade-off between probability coverage and interval width. In particular, an interval forecast that is narrower than all others and also enjoys greater empirical coverage should be the preferred choice.

Typically, when comparing multivariate forecast models, it is unusual for one model to outperform all others for every series projected at every forecast horizon. This is particularly likely in this application, given both the relatively large number of data sets and variables (21 age-specific mortality series for each of 16 samples). Therefore, to evaluate overall model performance, it is useful to adopt a single-valued measure that combines all the variables. One quantity to consider is life expectancy at birth e_{0,t}, defined as the average number of remaining years an individual born at time $t$ is expected to live. Following the discussion in Wilmoth (2004), let l_{a,t} denote the number of survivors at age $a$ in year $t$

Evidently, life expectancy at birth is a highly nonlinear function of all of the age-specific mortality rates that carries a natural interpretation. For this reason, it is often reported in practice as an overall summary measure of forecast performance, as in Lee and Miller's (2001) ex-post analysis. Unfortunately, such an aggregate quantity can be deceiving in that the forecast error associated with individual age groups could potentially cancel each other out in the computation of person-years remaining at birth, masking the extent of the forecast error experienced at particular ages.

Bell (1997) uses an alternative gauge of overall performance that looks at forecast error over the entire age profile. For instance, for the point projections, the RMSE corresponding to a particular forecast horizon is computed by averaging over the squared difference of observed and projected mortality at every age. This kind of measure is typical in multivariate time-series econometric applications. While not nearly as intuitive in its interpretation as life expectancy, it does not suffer from the potential problem that the forecast errors at different ages might cancel each other out. However, the measure is not without its drawbacks. In particular, suppose that a few of the series experience error that is disproportionately high relative to the remaining ages. Then, those few groups will largely determine the resulting total forecast error. A more robust measure of forecast error in the age profile might entail using a weighted average, with weights determined by the sample precision of each age series (that is, the inverse of its sample standard deviation). This more robust measure would define the importance of the error contributed by every age group as a function of how much variation mortality at that age displays in the sample, relative to the remaining series. Nevertheless, for the purposes of the forthcoming analysis, equal weights are assumed throughout.

In addition to both life expectancy at birth and the entire age profile as overall measures of forecast performance, the impact that mortality has on the program's future finances is an even more relevant criterion for a pay-as-you-go pension system. This impact is typically defined by the age distribution of the population, in terms of the old-age dependency ratio (the ratio of retired to working age population). Of course, to generate population forecasts, we would also need to model fertility and net migration, which is outside the scope of this paper. Nonetheless, it is still possible to evaluate the manner in which the age-specific mortality rates would actually enter into a population projection of the old-age dependency ratio, and thus, measure the effect that the mortality projections alone have on the program's finances. Specifically, recalling the previously defined number of survivors l_{a,t} at age $a$ in equation (21), the following dependency ratios implied by the individual age mortality rates are entertained:

For the purpose of illustration, Charts 2 through 4 display a number of projections generated at the initial jump-off year using the mortality data for the United States and United Kingdom. The top graphs in Chart 2 show actual mortality for the 10–14 age group, along with the median and 90-percent interval forecasts generated by the models from 1980 to the end of each series. The bottom graphs display forecasts corresponding to the 70–74 age group. The top of Chart 3 presents similar projections of life expectancy at birth, while the bottom graphs illustrate different measures of the dependency ratios defined above. In particular, the thickest solid lines in the bottom part of Chart 3 respectively represent the historical values of {\delta }_{1,t} and {\delta }_{2,t}, based on the actual population figures from the Human Mortality Database. For instance, for the United States, the dependency ratios in the year 2000 were {\delta }_{1,t}=0.212 and {\delta }_{2,t}=\mathrm{0.698.}⁹ By contrast, the remaining lines in the same graphs represent the dependency ratios based on the mortality rates alone, abstracting from fertility and net migration. These are the quantities relevant to the ex-post analysis in this paper. Their values for the United States in 2000 were {\delta }_{1,t}=0.351 and {\delta }_{2,t}=\mathrm{0.817.} Chart 4 shows projections of these two dependency measures.

Chart 2. Mortality projections for ages 10–14 and ages 70–74 for the United States (1980–2002) and the United Kingdom (1980–2003)

SOURCE: Author's calculations.

NOTE: AR(1) = Lag 1 autoregression.

Chart 3.
Life expectancy at birth and old-age dependency ratios for the United States and the United Kingdom

Life expectancy at birth e_0

Old-age dependency ratios

SOURCE: Author's calculations.

NOTE: AR(1) = Lag 1 autoregression.

Chart 4.
Projections of old-age dependency ratios for the United States (1980–2002) and the United Kingdom
(1980–2003)

SOURCE: Author's calculations.

NOTE: AR(1) = Lag 1 autoregression.

The experimental design of the ex-post analysis implemented in this paper looks at forecast error at fixed lead times, using different forecast origins. For every data set and model, N=20,000 random paths are simulated from 1980 forward for each of the 21 age groups. Since the Lee-Carter model takes as inputs the logarithmic death rates while the AR(1) approach models the rates of mortality improvement, the generated paths are transformed back into mortality rates prior to computing the features of interest of the forecast distribution. The mortality paths are then used to calculate the mean, median, and 5th and 95th quantiles for each age group and forecast period. In addition, the simulations corresponding to all 21 ages are also used to compute similar estimates for life expectancy at birth e_{0,t} and the dependency ratios {\delta }_{1,t} and {\delta }_{2,t}. Finally, the same process is repeated with other jump-off years (1981, 1982, 1983 and so on). This is done to limit the influence of any particular forecast origin on the results, and thus, improve the robustness of the findings. At the end of the exercise, there are $n$ projections of the quantities of interest with a 1-year forecast horizon, $n-1$ projections 2 years ahead, and eventually, 1 projection $n$ years into the future, where $n$ denotes the longest forecast horizon available, as the fourth column of Table 1 shows.

Once all the projections are obtained it is a simple matter to evaluate forecast error using the specified performance criteria. For the point estimators (means and medians), performance is measured in terms of RMSE. Formally, let {\widehat{m}}_{a,t,\Delta t} represent the \Delta t-step-ahead forecast of the mortality rate for age group $a$ in year $t.$ The RMSE associated with a particular age series and fixed lead time \Delta t is calculated as follows:

The performance of the interval forecasts is determined by computing the actual fraction of times ex-post mortality rates lie inside the intervals. Let {\widehat{C}}_{a,t,\Delta t} denote the area covered by the 90 percent \Delta t-step-ahead interval forecast of mortality for age-group $a$ in year $t.$ Furthermore, define an indicator function taking a value of 1 if the calculated {\widehat{C}}_{a,t,\Delta t} includes the observed mortality rate, and 0 otherwise:

Finally, the overall measures of performance are a function of either all or most of the 21 age groups. In this case, the forecast error associated with the point estimates of the entire age profile at a particular forecast horizon \Delta t is simply obtained by averaging over the ages

Forecast Performance of Point Projections

The first four columns of Table 4 present the resulting RMSE corresponding to the median forecasts of the Lee-Carter (LC) model for the following measures of overall forecast performance: the age profile, life expectancy at birth e_0, and the two age-dependency ratios {\delta }_1 and {\delta }_2 defined in equations (25) and (26), respectively. These quantities are computed over all available forecast horizons. Since both the means and medians of the forecast distribution are entertained as plausible point estimators, columns 5 through 8 in Table 4 display the ratio of RMSE between the two. Clearly, for the first three measures (the age profile, e_0, and {\delta }_1), the median is a better performing point estimator than the mean in the large majority of cases, as most of the ratios exceed 1. Only for the more comprehensive measure of dependency ({\delta }_2) do the mean projections generally exhibit lower RMSE than their median counterpart, although the differences between the two are fairly small. Moreover, while not shown for the sake of conciseness, the results corresponding to the AR(1) model are qualitatively similar. In light of these findings, this paper focuses exclusively on the median forecasts from this point forward.

To facilitate comparison, columns 1 through 4 in Table 5 present the ratios of RMSE in the median forecasts between the LC and AR(1) models (again, over all forecast horizons). Notice how forecast performance can vary across the different specified criteria. For example, for the Netherlands or the United States, the AR(1) approach outperforms Lee-Carter over the age profile, while the latter model actually exhibits lower RMSE for the projections of life expectancy and the dependency ratios. Conversely, for Finland, Germany and Japan, the LC model enjoys lower RMSE over the age profile but is outranked by the first-order autoregressive approach in the remaining measures.

The LC model outranks the autoregressive approach in half of all cases for {\delta }_1 and the age profile, while the AR(1) model displays lower RMSE in the other half. For life expectancy at birth e_0, the LC model does better in 7 of the data sets but is outperformed in the remaining 9 cases. For the broader dependency measure {\delta }_2, the AR(1) approach outperforms the LC model in 11 out of the 16 countries. Furthermore, in most instances, the differences in performance between the models are relatively small (that is, most of the ratios are fairly close to 1). There are a few notable exceptions to this finding for the forecasts of e_0. For instance, for France, the AR(1) approach reduces forecast error in life expectancy at birth by almost half relative to the LC model-likewise for the Netherlands and the United States. Overall, however, both models seem to display rather similar performance.

The remaining columns in Table 5 report the percentage of times the median projections in both models fall below the actual values for each of the four evaluation criteria. Clearly, for a given measure, the percentages corresponding to each model are very close to one another, suggesting that both models generate forecasts that are roughly biased in the same direction. With the exceptions of Japan, Spain, and the United States, notice that over the age profile, the percentages in Table 5 fall below 50 percent, so that the models tend to moderately overestimate actual mortality in most cases. By contrast, the large majority of the forecasts of life expectancy at birth and the age-dependency ratios underestimate their observed values. Specifically, only the median projections corresponding to Japan and the United States overpredict life expectancy, while the dependency ratios are also overestimated only for the United States. For the remaining data sets, between 70 percent to 99 percent of all generated forecasts underpredict e_0, {\delta }_1, and {\delta }_2.

To gain insight into the results presented in Table 5 (the models' forecasts overestimate actual mortality but underestimate life expectancy and the age dependency ratios), it is important to consider the mechanism via which the age-specific mortalities enter the calculation of e_0, {\delta }_1, and {\delta }_2. In all cases, the quantities that matter are the longitudinal numbers of survivors out of some initial population, as defined in equation (21). Suppose that a particular forecast {\widehat{m}}_{a,t} overpredicts actual mortality at age $a$ and time $t.$ Then, the implied survival rate {\widehat{s}}_{a,t}=(1-{\widehat{m}}_{a,t}) will underestimate the projected number of people that graduate into the next age category. Of course, whether the resulting future estimates of life expectancy at birth will underproject the observed values depends not only on the fraction of the age-specific rates that overestimate mortality, but also on the magnitude of their forecast error. Dependency ratios are further complicated by the fact that they comprise the quotient of longitudinal numbers of survivors at different ages, so that the distribution of both the bias and magnitude of forecast error across the ages plays a large role.

Performance by Age Group

One way to measure how error is distributed among the ages is to determine the percentage that each particular age group contributes to the value of total forecast error. For ease of presentation and to maintain consistency with how the dependency ratios have been defined, the individual age groups are aggregated into three broad categories (ages 0–19, ages 2–64, and ages 65–95 or older, respectively), containing 5, 9, and 7 of the 21 original groups. Broadly, these three categories encompass birth to young adulthood, the working population, and individuals in retirement ages. Following the discussion in the previous section, the RMSE associated with some individual age group $a$ over all forecast horizons \Delta t is determined by

Table 6 displays the percentage of forecast error over the entire age profile that is attributed to two broad sets of ages. The first set comprises the initial 14 age groups being modeled (from birth to age 64). These series make up less than 1 percent of total forecast error in most cases, and less than 3 percent in both models and all 16 data sets. By contrast, the retirement ages account for 97 percent to 99 percent of total MSE. In terms of model performance by age, the first three columns in Table 7 present the ratio of RMSE between models for the three broad age categories specified by the dependency measures. The first-order autoregressive approach outperforms the Lee-Carter model in 11 out of the 16 countries for the youngest age groups (ages 0–19), 7 countries for the working population (ages 20–64), and half of all countries for the retirement category (ages 65–95 or older). Furthermore, a comparison of the ratios in the first column of Table 5 with those in the third column of Table 7 reveals that they are virtually identical in magnitude, confirming once more that the oldest age groups overwhelmingly determine total forecast error over the age profile. The remaining columns in Table 7 show the percentage of the median forecasts that fall below the observed ex-post mortality rates by model and broad age category. Clearly, in all but one case (the United States), the models are far more likely to overestimate actual mortality for the oldest ages than for any age group. In most cases, over three-fourths of the generated projections for the 65–95 or older ages overpredict observed mortality.

Two obvious patterns concerning the models' median forecasts emerge from Tables 6 and 7. First, the bulk of forecast error is heavily concentrated among the oldest ages. Second, the majority of the forecasts corresponding to these age groups overestimate observed mortality. These findings shed additional light on the results shown previously in Table 5. In particular, a very high proportion of the forecasts for the 65–95 or older age groups overestimates mortality and hence, underestimates the number of population survivors at these ages. Furthermore, since these groups carry greater importance in determining how the magnitude of the forecast error is distributed across the ages, they are more likely to underpredict the total number of person-years remaining at birth, and thus e_0. Similarly, the 65–95 or older age groups enter the computation of the dependency ratios through the numerator. Consequently, if the number of survivors at these ages is underestimated, so are likely to be the values of {\delta }_1 and {\delta }_2. The exception to this pattern involves the projections for the United States, where mortality is underestimated at the oldest ages instead, while the forecasts of life expectancy and the dependency ratios overestimate their ex-post values.

Performance by Forecast Horizon

To assess how the median forecasts change with the length of the forecast horizon, the generated projections are grouped into four periods: 1–5 years, 6–10 years, 11–15 years, and 16 or more years ahead. Notice that the last category varies with the final year of data available for each series, involving 16–23 years ahead in most cases. Table 8 presents the ratio of RMSE between models over the age profile, as well as the percentage of the median forecasts that fall below observed ex-post mortality over the various forecast horizons. Tables 9 through 11 display similar quantities for the projections of life expectancy at birth e_0 and the dependency ratios {\delta }_1 and {\delta }_2, respectively. Although not discernible from the ratios in the first four columns of each table, as expected, forecast error generally increases with the distance of the forecast horizon.

Beginning with the age profile in Table 8, the first-order autoregressive approach outperforms the Lee-Carter model in ten cases for the 1–5 and 11–15 year horizons and in eight cases for the 6–10 and 16 or more year periods. While not always true, the differences in model performance tend to increase with the length of the forecast horizon, with the largest divergence corresponding to Austria in the 16 or more year period, where RMSE over the age profile for the AR(1) model is approximately 25 percent greater than for the LC approach. In most cases, a moderately larger proportion of the mortality forecasts tend to overpredict their observed values, except for Japan, where roughly three-fourths of the mortality forecasts involve underpredictions. The same pattern holds true for Spain and the United States, where approximately 50 percent of all forecasts underpredict mortality. Moreover, in about half of all countries, the percentage of projections overpredicting mortality increases as a function of the forecast horizon.

Turning to the median projections of life expectancy at birth in Table 9, the LC model outperforms the AR(1) approach in thirteen countries for the 1–5 year horizon, nine countries for the 6–10 year period, eight countries for the 11–15 horizon, and seven countries for 16 or more years ahead. Barring a few exceptions typically involving the longest forecast horizons (such as France, the Netherlands, and the United States), most of these ratios are relatively close to 1. Moreover, excluding the United States and Japan, the projections generated by both models overwhelmingly underpredict life expectancy, particularly as the distance of the forecast horizon increases. In fact, at the 16 or more year horizon 100 percent of the forecasts of life expectancy at birth underpredict their ex-post values for the majority of the data sets in both models.

Finally, Tables 10 and 11 show the performance of the point projections of the dependency ratios. For the {\delta }_1 ratio (survivors at ages 65–95 or older over ages 20–64), the LC model outperforms the AR(1) approach in ten cases for the 1–5 and 6–10 year horizons, nine cases for the 11–15 year period and eight cases for 16 or more years ahead. On the other hand, for the broader measure of dependency {\delta }_2 (ages 0–19 and 65–95 or older over 20–64), the LC approach outranks the autoregressive model in ten cases for the 1–5 year forecast period, six cases for the 6–10 year horizon, and only five cases for 11–15 and 16 or more years ahead. For both measures of dependency and all sixteen data sets, the largest difference in performance between the models does not exceed 26 percent at any forecast horizon. In all but one instance (the United States), the median projections of both dependency ratios underestimate their observed values increasingly as a function of the forecast period. At the 16 or more year horizon, virtually all of the generated forecasts underestimate the observed dependency values, while the converse is also true for the U.S. data (none of the median projections fall below the corresponding ex-post quantities).

Forecast Performance of Interval Projections

The first two columns in Table 12 display the empirical probability content of the 90-percent forecast confidence intervals generated by the models for the age profile, over all forecast horizons. The third and fourth columns in the same table respectively present the average width of these intervals for the Lee-Carter model, and the ratio of average width between models. The last four columns in Table 12 show similar quantities for life expectancy at birth e_0, while Table 13 displays analogous coverage and width measures for the two age dependency ratios {\delta }_1 and {\delta }_2.

Beginning with the age profile, it is evident that over all the age groups, the first-order autoregressive approach yields interval projections in every single case that exhibit greater probability content than the Lee-Carter model, but are also much wider. In general, the LC model seems more likely to generate mortality intervals that are "too narrow" (that is., that fall below their nominal 90-percent level of coverage). Conversely, the AR(1) model tends to produce intervals that are "too wide." For instance, with the LC model, only 4 nations exhibit coverage greater than or equal to 90 percent (France, the Netherlands, Sweden, and Switzerland), while in 9 of the 16 cases empirical coverage falls below 80 percent. By contrast, with the first-order autoregressive approach, probability content is in excess of 90 percent in nine of the data sets, whereas only three countries exhibit coverage below 80 percent (Austria, Germany and Japan). On average, the interval forecasts of mortality produced by the AR(1) model are wider than those of the LC approach by a factor ranging from less than one-and-a-half times wider for the United States, to nearly 10 times wider for Finland (fourth column in Table 12).

Turning to the projections of life expectancy at birth, it is clear that both models tend to generate intervals that are "too wide." With the exceptions of Austria and Germany in the AR(1) model and Austria, Canada, and Germany in the LC approach, empirical coverage exceeds 90 percent for the remaining countries and is either equal or closer to 100 percent in most cases. Moreover, the differences in size between the interval forecasts generated by the two models are far less pronounced than for the age profile. In roughly half of the data sets, each model produces narrower intervals on average than the other. These findings highlight the type of cancellation effects that can occur when the age-specific mortality forecasts are combined to produce such a highly nonlinear aggregate measure of overall performance. Consider, for instance, the interval projections corresponding to Japan. In this case, over all age groups and forecasts horizons the 90 percent interval projections generated by the Lee-Carter model contain observed mortality only 36 percent of the time. However, when the simulated paths are used to compute e_0, all 276 interval forecasts of life expectancy at birth contain the corresponding ex-post values, resulting in 100-percent probability coverage.¹⁰ The converse can also be the case. In the AR(1) approach, the interval projections of mortality for Austria over the age profile have an empirical probability content of 74 percent, while those associated with the LC model yield 66-percent coverage. Yet, for the latter model, the interval forecasts of life expectancy display 71-percent coverage, with an average width of 3.6 years over all forecast horizons. By contrast, the projections of life expectancy generated by the AR(1) model exhibit extremely poor coverage (24 percent) and are half the size of those produced by the LC approach.

For the OASDI program, a more useful performance evaluation criterion regarding the age-specific mortality forecasts generated by the models involves the forecast error associated with the age dependency ratios presented in Table 13. In this case, with the exceptions of Austria and Germany, where empirical coverage is quite poor, the first-order autoregressive approach produces interval forecasts with probability content in excess of 90 percent for both measures {\delta }_1 and {\delta }_2. On the other hand, the Lee-Carter model generates intervals that are "too narrow" for half of the data sets and "too wide" for the other half. Specifically, empirical coverage in Austria, Belgium, Canada, Finland, Germany, Italy, Norway and the United Kingdom falls below 60 percent. Not surprisingly, the AR(1) model generates wider interval projections than the LC model in 11 cases for {\delta }_1, and 15 cases for {\delta }_2.

Finally, Table 14 shows the performance of the interval projections generated by the models over the three broad age categories previously defined. In general, the Lee-Carter model tends to produce interval forecasts of mortality that exceed their hypothetical probability content at the youngest ages, but seriously underestimate it for the older age groups. For instance, in the 0–19 age category there are 12 cases with coverage in excess of 90 percent and only 3 countries with coverage below 80 percent (Germany, Japan and the United States). By contrast, for the retirement ages (65–95 or older), coverage stays above 90 percent in 2 countries (France and the Netherlands), while it falls below 80 percent in the remaining 14 countries. On the other hand, for all three age categories (0–19, 20–64, and 65–95 or older), the first-order autoregressive approach generates interval forecasts with over 90-percent probability content in the majority of instances. Moreover, in every single case the AR(1) interval projections are narrower than those of the LC model for the youngest ages, but much wider for the 65–95 or older age class.

Performance by Forecast Horizon

Table 15 displays the empirical probability content and ratio of average width corresponding to the 90-percent interval projections of the models over the age profile and various forecast horizons (1–5, 6–10, 11–15 and 16 or more years ahead). Tables 16 through 18 present similar quantities for the interval projections of life expectancy at birth e_0 and the two age dependency measures {\delta }_1 and {\delta }_2. Although not always the case, coverage over the age profile tends to decrease with the length of the forecast horizon. For the 1–5 year period, the LC and AR(1) models generate interval forecasts with over 80-percent coverage in 10 and all 16 countries, respectively. Out of these countries, coverage exceeds the hypothetical 90-percent level in 6 cases for the LC model and 12 cases for AR(1) approach. On the other hand, for the most distant forecast period (the 16 or more year horizon), probability content lies above 80 percent in 6 countries for the LC model and 10 countries for AR(1) approach. Even at this forecast length, coverage exceeds 90 percent in half of all cases for the latter model. In terms of the size of the generated intervals, the LC model generates narrower projections over the age profile than the AR(1) model across all forecast horizons. As previously mentioned, this is because interval projections for the oldest age groups in the first-order autoregressive approach are much wider.

Table 16 shows the empirical content of the interval projections of life expectancy at birth. Clearly, with the exceptions of Austria and Germany, where coverage deteriorates with the length of the forecast horizon, both models generate intervals that are "too wide." For most of the data sets there is 100 percent coverage at every forecast horizon. The forecast intervals of e_0 produced by the LC model are narrower than those of the AR(1) approach in 11 cases for the 1–5 year period, and 10 cases for the remaining forecast horizons.

Finally, Tables 17 and 18 present the empirical probability coverage of the interval forecasts for the age-dependency ratios {\delta }_1 and {\delta }_2. Clearly, for the Lee-Carter model, performance tends to deteriorate dramatically with the distance of the forecast horizon. By contrast, with the exceptions of Austria and Germany, the AR(1) approach yields interval projections with 100-percent probability content in the large majority of cases across all forecast periods. For instance, over the 1–5 year horizon, coverage in the LC model exceeds 80 percent in 15 cases for {\delta }_1, and in 13 cases for {\delta }_2. On the other hand, over the longest forecast period (16 or more years ahead), these quantities drop down to 8 and 6 cases, respectively. In fact, for this same period, probability content in the LC model is actually 0 percent in five and eight nations for the {\delta }_1 and {\delta }_2 ratios, respectively. Conversely, over the 16 or more years horizon, the AR(1) approach yields coverage in excess of 90 percent in 13 and 12 cases. Generally, the interval forecasts corresponding to the first-order autoregressive approach are wider on average than those of the LC model.