Testing for Changes in Forecasting Performance

Testing for Changes in Forecasting Performance Abstract We consider the issue of forecast failure (or breakdown) and propose methods to assess retrospectively whether a given forecasting model provides forecasts which show evidence of changes with respect to some loss function. We adapt the classical structural change tests to the forecast failure context. First, we recommend that all tests should be carried with a fixed scheme to have best power. This ensures a maximum difference between the fitted in and out-of-sample means of the losses and avoids contamination issues under the rolling and recursive schemes. With a fixed scheme, Giacomini and Rossi’s (2009) (GR) test is simply a Wald test for a one-time change in the mean of the total (the in-sample plus out-of-sample) losses at a known break date, say , the value that separates the in and out-of-sample periods. To alleviate this problem, we consider a variety of tests: maximizing the GR test over values of within a pre-specified range; a Double sup-Wald (DSW) test which for each performs a sup-Wald test for a change in the mean of the out-of-sample losses and takes the maximum of such tests over some range; we also propose to work directly with the total loss series to define the Total Loss sup-Wald (TLSW) and Total Loss UDmax (TLUD) tests. Using theoretical analyses and simulations, we show that with forecasting models potentially involving lagged dependent variables, the only tests having a monotonic power function for all datagenerating processes considered are the DSW and TLUD tests, constructed with a fixed forecasting window scheme. Some explanations are provided and empirical applications illustrate the relevance of our findings in practice. 1 Introduction We consider the issue of forecast failure (or breakdown) and propose methods to detect changes in the forecasting performance over time. The aim is to assess retrospectively whether a given forecasting model provides forecasts which show evidence of changes (improvements or deterioration) with respect to some loss function. Since the losses can change because of changes in the variance of the shocks (e.g., good luck), detection of a forecast failure does not necessarily mean that a forecast model should be abandoned. Care must be exercised to assess the source of the changes. But if a model is shown to provide stable forecasts, it can more safely be applied in real time. In practice, such forecasts are made at the time of the last available data, using a fixed, recursive or rolling window. Hence, there is a natural separation between the in and out-of-sample periods dictated by the last data point. Such is not the case when trying to assess retrospectively whether a given model provides stable forecasts. There is a need for a separation between the in and out-of-sample periods at some date labelled , say. This date should be such that the model in the insample period is stable, i.e., yielding stable forecasts. This can create problems since one needs a truncation point  to assess forecast failures but the choice of this value is itself predicated on some knowledge of stability. An example of such test is that of Giacomini and Rossi (2009), GR hereafter. It is a global and retrospective test which compares the in and out-of-sample averages of the sequence of forecast losses. See Casini (2017) for an extension as well as Casini and Perron (2018) and Perron (2006) for a review of the relevant issues. We adapt classical structural change tests to the forecast failure context. First, we recommend that all tests should be carried with a fixed scheme to have best power, which ensures the maximum difference between the fitted in and out-of-sample means of the losses; contamination issues under the rolling and recursive schemes induce power losses. With this fixed scheme, GR’s test is simply a Wald test for a one-time change in the mean of the total (in-sample plus out-of-sample) losses at a known break date . To alleviate this problem, which leads to power losses when the forecast breakdown is not at , one can follow Inoue and Rossi (2012) and maximize the GR test over values of  within a pre-specified range; i.e., a sup-Wald test for a single change at a date constrained to separate the in and out-of-sample periods. The test is still not immune to non-monotonic power problems when multiple changes occur. Hence, we propose a Double sup-Wald ( ) test which for each  performs a sup-Wald test for a change in the mean of the out-of-sample losses and takes the maximum over the range  ∈ [0  1 ]:  = max∈[0  1 ]  () , where  () is 1 the sup-Wald test for a change in the mean of the out-of-sample losses for a forecast horizon  ,  (̂) for  =  +        , defined by  () = max ()∈[+ + + +(1−)] [ () −  ( ()) () ]̂ () , where  =  −− +1,  () is the restricted sum of squared residuals (SSR),  ( ()) () is the SSR with a change at  (), and ̂ () is the long-run variance estimate of the out-of-sample losses ( is a small trimming parameter set at 01 throughout). We also propose working directly with the total loss series  () to define the Total Loss sup-Wald (  ) and UDmax tests ( ). Using simulations based on the original design of GR, we show that with forecasting models with lagged dependent variables, the only tests with monotonic power for all data-generating processes considered are the  and   tests, constructed with a fixed forecasting window scheme. The benefits of forecast breakdown tests over structural change tests applied to the forecasting model are the following, among others. As stated in GR, they can detect breaks in variance (with a squared error loss); they allow model misspecifications; breaks in coefficients and in variance may offset; they allow for instability in the distribution of the regressors. More interestingly, the two types of tests are complementary. Consider data generated with the predictor changing from  to  at some date;  is not accessible and a regression of + on  is used. For a given variance of  , when the variance of  is small the power of the forecasting tests is small, while that of the structural change tests is large (and vice versa), since  is part of the error term. This holds for more general tests with unknown multiple breaks. It shows a complementarity between the power of the forecast breakdown and structural change tests. See Supplement (a) for theoretical and simulation analyses. The paper is structured as follows. Section 2 introduces the statistical framework and some tests: 2.1 reviews the single break case at a known date; 2.2 discusses why using a fixed scheme is preferable; 2.3 considers unknown break dates and proposes new tests. Some limit distributions are stated in Section 3. Section 4 considers the size and power of the tests: 4.1 for the finite sample size of the proposed tests; 4.2 describes the setup to evaluate the power functions; 4.3 contains theoretical results about the shapes of the loss functions. Section 4.4 presents a summary of the results and 4.5 expands on the reasons for non-monotonic power functions. Section 5 provides empirical applications and Section 6 concluding remarks. A supplement contains technical derivations, additional discussions and results. 2 The framework and the tests We have data (   ) with  a scalar variable to be forecasted and  a -dimensional vector of predictors for  = 1   . Consider a model forecasting + at period , a  -period ahead 2 forecast obtained using the direct method is ̂+ =  (̂  ;  ), where  is a known function that defines the model, ̂  is the estimate of the parameter  ( × 1) obtained from an in-sample window of size  ≥ ; e.g., for the linear model,  (̂  ;  ) = ̂   and ̂  is the OLS estimate from a regression of + on  using data from the in-sample window. The outof-sample forecast procedure divides the full sample into an in-sample window of size  and an out-of-sample window of size  =  −  −  + 1. The model is estimated in the in-sample window and the out-of-sample window is used for forecast error evaluation. We consider three popular forecast schemes: fixed window, with the in-sample consisting of observations 1 to ; rolling window, with the in-sample consisting of observations −+1 to ; recursive window, with the in-sample consisting of observations 1 to . We denote the sequence of in-sample losses by  (̂  ), defined by the in-sample fitted values ̂ =  (̂  ;  ), and the sequence of out-of-sample losses by  (̂  ), defined by the forecast values ̂+ . With the popular squared error loss function, + (̂  ) = (+ −  (̂  ;  ))2 and + (̂  ) = (+ − ̂+ )2 . We also define the in-sample loss sequence:  () = ( +1 (̂  )   (̂  )), the out-of-sample loss sequence:  () = (+ (̂  )   (̂  )) and the total loss series as the stacked vector of both, i.e., () = ( +1 (̂  )   (̂  ), + (̂  )   (̂  )), a ( − 2 + 1) vector. −2 +1 . Note that we use a “direct A time-indexed total loss series is denoted by {+ }=1 forecast” method when   1. Supplement (g) shows that the main theoretical results also apply when using an indirect forecast method. The goal is to assess whether there are instabilities in forecast accuracy; e.g., a deterioration usually referred to as a “forecast breakdown”. This can occur because of a genuine change in the stability of the forecasting regression, via the conditional mean, or from changes in the variance of the errors. It can also occur if the forecasting model is misspecified in which case an over-fitting problem is possible, so that the out-of-sample losses are inflated relative to the in-sample losses irrespective of whether a change in the stability of the forecasts is present or not. We shall be concerned about the former case. If one wants to guard about potential changes related to over-fitting, one can simply adjust the out-of-sample losses by subtracting a correction factor (see GR for the exact expressions for a linear model and a quadratic loss function). The null hypothesis considered is 0 : [ ( ∗ )] = 0 , for all  =  + 1   −  + 1 for some  ∗ =  lim→∞ ̂  , which implicitly assumes that the probability limit of ̂  is the same for all  under the null hypothesis. The alternative hypothesis is 1 : [ ( ∗ )] 6= [+1 ( ∗ )] for at least one  =  + 1   −  . Hence, we are concerned with testing the stability of the forecast performance in population as opposed to in finite samples using the terminology of Clark and McCracken (2013). 3 2.1 The case with a single break occurring at a known date We consider first a single break in forecast accuracy at a known date  , say, so that the alternative is  [ ( ∗ )] = 1 for  ≤  , 2 for    . The obvious thing is then to apply a test for a change in the mean of the total loss series at date  , fixed for any given  but increasing with  so that lim →∞   = , say, when performing an asymptotic analysis. This is achieved by setting  =  and assessing whether the averages of the in and out-ofsample losses are different. This is the test proposed by GR. Define the surprise loss as the out-of-sample loss minus the mean of the in-sample losses, i.e., + (̂  ) ≡ + (̂  ) − P  ̄ (̂  ) where ̄ (̂  ) ≡ ( −  )−1  = +1  (̂  ), with the out-of-sample losses adjusted P − + (̂  ))Ω̂12 , for over-fitting if desired. The test they propose is  = (−12 = where  =  −  −  + 1 is the size of the out-of-sample window and Ω̂ is an estimate of the long-run variance of the loss sequence (see GR for the exact form suggested). It is easy to verify that the square of this test is equivalent to an F-test for a change in mean occurring at date  when applied to the total loss series (). This simple observation leads to the following comments. First, it is really inconsequential if we work with the original loss functions (in-sample plus out-of-sample) with a given value of  that assumes a pre-sample with no change or with the “surprise loss functions” for which the average of the in-sample losses is subtracted from the out-of-sample losses. Hence, below, we shall also consider tests constructed via the original (not-demeaned) out-of sample losses, again assuming the model to be stable prior to date  (the demarcation between the in and out-of-sample). Second, the test of GR is problematic since the true break date is unknown in practice even if only one is present. This makes the test sensitive to the choice of . As will be shown via simulations, it can have non-monotonic power (decreasing as the mean-change of the losses increases) for a range of choices for . Hence, we also consider tests that allow  to vary within some pre-specified range, whether with the surprise or original losses. 2.2 The choice of the forecasting scheme Before considering tests that do not assume a known break date, we discuss the merits and drawbacks of the fixed, rolling or recursive forecasting schemes. To get better forecasts it is, in general, better to adopt a recursive or rolling forecasting scheme in the presence of instabilities, or a combination of both; see, e.g., Clark and McCracken (2009). The parameter estimates then adapt to the data-generating process to fit the data better and provide more accurate forecasts. A fixed forecasting scheme fails to provide such adjustments. A rolling one 4 provides adjustments but at the expense of increased variability due to a smaller in-sample window. However, when trying to detect retrospectively whether a change has occurred the opposite ranking applies. The best scheme to adopt is a fixed one. Suppose that the break date is known and  is set accordingly. A fixed scheme ensures the maximum difference between the fitted in and out-of-sample means of the losses. With a recursive scheme, the in-sample fitted mean of the loss series is pushed towards the fitted mean of the out-of-sample losses inducing a loss of power. With a rolling window scheme, the same occurs but in a more pronounced way since the in-sample fitted mean can eventually reach the post-break mean if the window is small. Hence, when testing for changes in forecast accuracy, it is preferable to use a fixed window scheme. This will remain true with one or multiple breaks occurring at unknown dates. We provide explanations with some theory and simulations later. 2.3 The case with unknown break dates A simple method to alleviate the dependence of the  test on  is to take the supremum over a range of , say [0  1 ], a version denoted by  = max∈[0 1 ] | |. This test is tailored to the alternative hypothesis with  unknown. The limit distribution and critical values of  are in Inoue and Rossi (2012) for typical values of 0 and 1 . Alternatively, 2 , which is equivalent to a sup-Wald test for a change one could use 2 = max∈[0 1 ]  in mean and use the critical values in Andrews (1993). This modification will, however, not be immune from power problems when multiple changes occur. To see why, consider the case with two breaks. Then for any choice of  in the range [0  1 ] at least one segment will be contaminated due to biased parameter estimates and the average loss will be reduced thereby decreasing the power of the test. As we shall see, this problem can be especially severe when the range [0  1 ] is large. To avoid it, one can perform a sup-Wald test for a change in the mean of the out-of-sample losses for each value of  and take the maximum of such tests over a range  ∈ [0  1 ], labelled as the Double sup-Wald ( ) test, defined by  = max∈[0 1 ] 0 () , where  () is the sup-Wald test for a change in the mean of the out-of-sample loss series  (̂) for  =  +     , defined by  () = max [ () − ( ()) () ]̂ ()   ()∈[+ ++ +(1−)] (1) where  () is the restricted , ( ()) () is the  with a change at time  (), and ̂ () is the long-run variance estimate of the demeaned out-of-sample loss series with the mean changing at date  (), obtained using the method of Andrews (1991). The parameter  is a small trimming value set at 01. Too small a value leads to size distortions 5 and large ones to power problems. The results are not sensitive to minor variations, e.g.,  = 005 or 015; see Supplement (c). The limit distribution of the  test is stated in Section 3. To construct it: 1) Start with an out-of-sample method with an in-sample length 0 small but large enough to estimate the model. Let  ≡  −  −  + 1. 2) Compute − . 3) Consider a regression with only a constant: the out-of-sample loss series {+ }=  + =  + + . Apply a sup-Wald test for the constancy of , with a HAC variance estimate if there is evidence of serial correlation in the losses. Store the value as  () . 4) Update  to  + 1 and repeat Steps 2-3 up to  = 1 . The range of , 1 − 0 , is 1 some fraction of , denoted by ̄. 5) Take the maximum of the sequence of { () } =0 . The reason why the  test improves upon the  test is because it produces three segments instead of only two, which is beneficial with more than one break. Two are defined by  and the other by the date at which the Wald test is maximized in the range [ +  +   +  + (1 − )]. Hence, three segments can be inserted within the total sample, which guarantees that the two segments with the largest difference can be separated by a break, thereby increasing power. With a single break the mean of at least one segment is contaminated by the values of the means in the other segments, reducing power. The idea is akin to that of Qu (2007) who showed that when searching whether any part of a sample is stationary all one needs is a search with two breaks or three segments. To go further, one can also consider a test similar to the UDmax test for multiple changes of Bai and Perron (1998). However, the size distortions were rather high and we shall not consider it further. An alternative is to work directly with the total loss series () instead of only using the out-of-sample losses. This can yield higher power given that more information is used. We consider two tests following this approach: the Total Loss sup-Wald test (  ) and the Total Loss UDmax test (  ). More precisely,   = max∈[0 1 ] () , where () is the sup-Wald test applied for a mean-change in the total loss series (): () = [() − ( ())() ]̂()  max  ()∈[ +( −2 +1) +(1−)( −2 +1)] where () is the restricted SSR, ( ())() is the SSR assuming a one-time change at time  () and ̂() is the long-run variance estimate of the total loss series using demeaned total loss series with the mean changing at date  ();  is some small  , where trimming parameter set at 01. Also,   () = max∈[0 1 ] ()  = max  () max =1 {  ()}=1 ∈Λ  [() − ({ ()}=1 )() ]̂() ¯ ¯ with Λ = {(1 ()   ()) : ¯ () − −1 ()¯ ≥  +( −2 +1), 0 () = 1  () ≤ 6  + (1 − )( − 2 + 1)} and ̂() the long-run variance estimate of the total loss series using the demeaned total loss series with the mean changing at dates 1 ()   (). We set  = 5. To summarize the construction of the TL tests: 1) start with a value of the −2 +1 ; 3) for in-sample length; e.g., 0 = b015 c; 2) compute the total loss series {+ }=1 the regression with only a constant, + =  + + , apply the sup-Wald or UDmax test for the constancy of  with a HAC variance estimate if there is evidence of serial correlation in  . 4) Update  to  + 1 and repeat Steps the losses. Store the value as () or () 2-3. Take a choice of 1  0 , say 1 = b085 c, and continue up to  = 1 . The choice of 0 and 1 does not affect the asymptotic critical values (Theorem 2 below). 5) Take the  1 1 maximum of the sequence of {() } =0 or {() }=0 . 3 Asymptotic distributions of the proposed tests This section discusses the asymptotic distributions of the proposed test statistics under  the null hypothesis. We let “→” denote convergence in probability and “⇒” denote weak convergence in distribution. We first require the following assumption. Throughout, we assume that  , , and  go to infinity at the same rate unless otherwise stated.  Assumption 1 Under no change in forecast accuracy: i) ̂  →  ∗ for all  ∈ [0  1 ], with 0 and 1 the smallest and largest values of the in-sample lengths; ii) For + ≡ P −2 +1)c  −2 +1 ,  [+ ] =  for all  and  −1 [ b( (+ − )]2 → Ω, as {+ ( ∗ )}=1 =1 Pb( −2 +1)c (+ −  → ∞ , for  ∈ [0 1] with  fixed, Ω a non-random matrix and  −12 =1 ) ⇒ Ω12  (), with  () a standard Wiener process defined on  ∈ [0 1]. These high level assumptions characterize the properties of the loss series under the null hypothesis. It is informative to see what they imply for the linear forecasting model + = 0  + + . Then Assumption 1 basically requires that  is stable over time under the null hypothesis of no change in forecast accuracy and the loss sequence satisfies a standard functional limit theorem with long-run variance Ω. Another important feature is that the loss series do not depend on  when evaluated at the limit value  ∗ . The relevance of this assumption is examined using the same example of a correctly specified linear model. Suppose we compute loss series using two distinct in-sample lengths 1 and 2 . The coeffi1 2 cient estimates are denoted by ̂ and ̂ , say, and the forecasting errors are + − 0 ̂ 1 and + −0 ̂ 2 , respectively. Under the null hypothesis, these series are asymptotically equivalent since, roughly speaking, both estimators converge to a unique limit value  ∗ for all . In general, under the null hypothesis of no change in forecast accuracy, in large samples, and under 7 a quadratic loss function, the losses are proportional to 2 so that, if  has constant unconP −2 +1)c 2 (+ −  2 ) ⇒ Ω12  (), ditional variance  2 , Assumption 1 is satisfied if  −12 b( =1 P −2 +1 2 (+ −  2 )]2 is the long-run variance of the centered where Ω =  lim →∞  −1 [ =1 values of the squared errors. This allows considerable forms of dependence in the higher moments of  , in particular the second moment so that conditional heteroskedasticity or serial correlation in the squared errors is allowed, in which case Ω is different from 2 and the test statistics need to be scaled by an estimate of the long-run variance of 2 . Hence, the conditions are quite general when the model is stable. Under Assumption 1, () and () have the same null limiting distribution as the sup-Wald test for a change in mean (Andrews, 1993) and the  max test of Bai and Perron (1998), respectively. We next present the asymptotic distribution of the  test, whose proof is in Supplement (b). Theorem 1 Under Assumption 1, the limit distribution of DSW is given by [( − ) (1) + (1 − ) () − (1 − ) ()]2  (1 − )(1 − )( − ) ∈[0̄]∈[+(1−)1−(1−)]  ⇒ sup sup as    → ∞ at the same rate, where  () is a standard Wiener process defined on  ∈ [0 1],  is the trimming parameter and ̄ = lim →∞ (1 −0 )0 , with 0 =  −0 − +1. The critical values of the  test were tabulated using 5,000 replications with 5,000 steps to approximate the Wiener process as partial sums of  (0 1) random variables. We report results for a grid of values for ̄ in the range [020 080] and we set  = 01 (used throughout in the simulations and applications). The results are presented in Table 1 (see Supplement (c) for  = 005 and 015). We next consider the limit distribution of the −2 +1 does not   and    tests. Exploiting the fact that the loss series {+ ( ∗ )}=1 asymptotically depend on  under the null hypothesis, we obtain the following theorem. Theorem 2 Under Assumption 1 and the null hypothesis: a) the limit distribution of   is the same as the sup-Wald test for a change in mean (Andrews, 1993) for any 0 and 1 (1 ≤ 0 ≤ 1 ≤  ); b) the limit distribution of    is the same as the UDmax test of Bai and Perron (1998) for any 0 and 1 (1 ≤ 0 ≤ 1 ≤  ). −2 +1 does not depend Under Assumption 1, Theorem 2 follows trivially since {+ ( ∗ )}=1 on  under the null hypothesis and the tests computed with different ’s are asymptotically perfectly correlated. This implies no effect of taking the maximum of the statistics over  on the limiting distribution. Note also that, unlike for the  test, the choices of 0 and 1 do not affect the limiting distribution of the   and    tests. 8 4 Analysis of the size and power of the tests We present simulation and theoretical results to address the following issues: 1) the finite sample size of the tests proposed (Section 4.1); 2) the power function of the tests: Section 4.2 describes the experimental design, Section 4.3 provides theoretical results useful to understand the main features of the power functions and Section 4.4 provides a summary of the main results. Section 4.5 expands on the causes of various non-monotonic power functions. 4.1 Finite sample size of the proposed tests We first examine the size of the   and    tests using the asymptotic distribution of Theorem 2. The DGP specifies  ∼ (0 1) of lengths  = 150 300. We consider the squared error loss for the static model: + =  + + , and the dynamic model: + =  +  + + , with  = 1 both estimated by OLS and tests with or without a HAC correction for serial correlation in the losses. The HAC variance estimate is constructed using Andrew’s (1991) data dependent method with an AR1 approximation and the Bartlett kernel. For all cases, we consider a fixed, rolling and recursive forecasting scheme. The exact sizes of the tests are presented in Table 2 for  = 300 and ̄ = 025 05 and 075 (for  = 150, see Supplement (d)). The number of replications is 1,000. We label the test without the HAC variance estimate by “non-robust” and with it by “robust”. We also set  = 01, 0 = b015 c and 1 = b085 c; any reasonable variations of these choices do not change the results qualitatively. The exact size is, in general, close to the nominal size. Some distortions are present with the robust version, which decrease as  increases. The results for the size of the  tests are in Table 3, with the same specifications as above (for  = 150, see Supplement (e)). For ̄ = 025 and 05, the test shows little size distortions, if any, for all cases. Some liberal size distortions are present when ̄ is as large as 075 (caused by parameter uncertainty from unreported simulations). With larger sample sizes, the distortions somewhat decrease but remain substantial for ̄ = 075. Hence, we recommend using ̄ = 025 or ̄ = 05. 4.2 The experimental design for the power analysis In order to ensure that our simulation design is not biased in favor of the tests we propose, we adopt the same design as in GR. Note, however, that we do not set  to be equal to the date of the first break. GR mention that this corresponds to the worst case scenario from a forecasting point of view. But what is more relevant in the context of assessing the presence 9 of forecast instabilities is the fact that it corresponds to the best case possible for the power of the tests. Hence, such a choice can distort the power properties of the tests which are relevant in practice, given that the date of the break is unknown. There are five different DGPs involving single or multiple changes in level or in variance. They are: DGP1: (single variance shift):  =  ,  ∼ (0 2 ), with  2 = 1 +   (   2); DGP2: (multiple variance shifts):  =  ∼  (0 2 ),  2 = 1 +   ( * Λ0 ); DGP3: (single mean shift):  =   (   2) +   with  ∼ (0 1); DGP4: (multiple mean shifts):  = −  +  if  ∈ Λ0 ,   +  otherwise,  ∼ (0 1); DGP5: (mean shifts at unequal intervals):  =   ( ≤  4) − 3  ( 4   ≤  2) + p   (   2) +  ,  ∼ (0 1). We set  = 150 and with multiple breaks as in DGPs 2 and 4, these occur every 50 periods, i.e. Λ0 ∈ {(1 50) (101 150)}. The following tests are considered with the squared error loss function: the  test of GR with  = 40 (1) and  = 100 (2); the supremum of the  tests, labelled  over 02 ≤  ≤ 08 (1) and 03 ≤  ≤ 07 (2), the Double sup-Wald test  with ̄ = 025 ( 1) and ̄ = 050 ( 2) with 0 = 02 for both cases; the sup-Wald test for a single break and the UDmax test for multiple breaks (up to 5) applied to the total loss series, denoted   and  , with 015 ≤  ≤ 085 . Two forecasting models are used: a static model: + =  + + , and the dynamic one: + =  +  + + , with  = 1, both estimated by OLS, and for each, two versions with or without a HAC correction for serial correlation in the losses. In the dynamic model an irrelevant lagged dependent variable is included (i.e.,  = 0), which is completely inconsequential. We could extend the DGPs to include genuine dynamics with  6= 0. The qualitative features would remain the same. The nonmonotonicities reported would simply be more severe (e.g., Section 4.5). The static model with no HAC correction is labelled “static, non-robust” and with a HAC correction “static, robust”. The dynamic model with no HAC correction is labelled “dynamic, non-robust” and the one with a HAC correction “dynamic, robust”. We consider forecasting schemes using a fixed, rolling or recursive window. The number of replications is 1,000. The results are presented in Tables 4-1 to 4-5. The foremost criterion adopted to compare the tests is whether the power function is monotonically increasing as the magnitude of the change(s) in forecast accuracy increases. We view this as an essential feature for any reasonable test. For tests with monotonically increasing power, we compare the relative power functions. We start with some theoretical results about the limit value of the expected loss function that will help understand the sources of the power differences across various forecasting schemes. 10 4.3 Theoretical results about the limit of the loss function We consider the limit of the loss sequence for a single coefficient or variance break. Our results pertain to the loss sequence in large samples, which are used for all tests. The surprise losses are used for the  and  tests but since these simply differ by subtracting the average of the in-sample losses, the dynamics of both sequences are similar. Hence, we consider only the expected values of  lim →∞  = ∗ (), defined on the unit interval  ∈ [0 1] where  = lim( ). The squared loss function is applied. For simplicity, and without substantive loss of generality, we consider the single break model: + =    + +  for  = 1   −   (2) where   =  1 for  ≤ [ 0 ] and   =  2 for   [ 0 ]. Again for simplicity, the predictor  is a scalar that satisfies (2 ) =  2 and ( − ) =   . Also,  ∼ (0 2 ), with  2 =  21 for  ≤ [ 0 ] and  2 =  22 for   [ 0 ]. The in-sample length is  = [ ] chosen so that  ≤ 0 . We consider the following two cases: a coefficient change, i.e.,  1 = 0 and  2 = ∆ with  21 =  22 =  2 ; a variance change, i.e.,  21 =  2 and  22 =  2 + ∆2 with  1 =  2 = . Suppose we use the static regression model of + on  to produce  -period ahead forecasts at time . For the out-of-sample procedure, we use the estimate of  obtained from the insample information, labelled as ̂ [1] . We consider the three window schemes and estimate the coefficient using OLS for the sample period [1 − ] with the fixed scheme, [−+1 − ] with the rolling scheme, and [1  −  ] with the recursive scheme. Hence, ̂ [1] with the same  can be different depending on the window scheme. When the static regression is used,  [∗ ()] = [lim →∞ (+ −  ̂ [1] )2 ] =  2 +  2 (  −  ∗[0] )2 , where  ∗[0] =  lim →∞ ̂ [1] . We also denote the limit true value of   and  2 , defined on the interval  ∈ [0 1], by   and  2 . Next, we consider a dynamic regression, i.e., + =  +    + + . Including  as a predictor while the true model is (2) is inconsequential in a stable environment since the dynamic model nests the DGP. Things are quite different when instabilities are present. For the dynamic model, with ∗[0] =  lim →∞ ̂[1] , and  ∗[0] =  lim →∞ ̂ [1] , ∗ ∗2 2 ∗ 2  [∗ ()] = (1 + ∗2 [0] )  +   [(1 − [0] )  −  [0] ] (see Supplement (f)). Figures S.1S.2 present [∗ ()] for the variance change case using the static and dynamic regressions, respectively. The limit of the loss sequence always has a stepwise change, whose magnitude depends on ∆2 . Hence, all tests should have power monotonically increasing in ∆2 . Since the expressions for the case of a coefficient change are quite complex and yields little insights per se, we only present numerical values in the text for [∗ ()]. We set 2 = 1,  = 01 and 0 = 05 and a small (∆  ∆2 = 1) and large break (∆  ∆2 = 5) 11 to investigate how the size of the break affects the shape of the loss sequence. Figures 12 present [∗ ()] for the coefficient change case for the static and dynamic regressions, respectively. The upper (lower) panels report the case of a small (large) break. When the static model is used (Figure 1), we see a stepwise change in the loss sequence when the fixed scheme is used. However, the same change in coefficient translates into a spiked shape with the rolling scheme and a triangular shape with the recursive scheme. This explains why the fixed scheme is to be preferred and why using the rolling or recursive scheme induces a loss of power. Note, however, that the break magnitude only changes the height of the change in the loss sequence, not the shapes of the loss sequences. Hence, increasing the magnitude of the break size should still increase power under all schemes. Things are different when considering the dynamic model. We still have the same general shapes for the loss sequences. However, the spikes and the triangular shape for the rolling and the recursive schemes become more narrow and closer to an outlier as the break becomes larger. This explains why, when using a dynamic model, the use of a rolling or recursive scheme leads to a non-monotonic power function, i.e., the power decreases as the break magnitude increases. We shall use the insights provided by these results to explain the power differences across the tests and the forecasting schemes. We show in Supplement (g) that the main theoretical results continue to hold using an indirect forecast method. The results are also similar with multiple breaks, in which case the shapes essentially repeat themselves for each break date; see Figures S.5 and S.6 in Supplement (h), which present a “typical” realization of the loss sequences for DGPs 3-5 under the three forecasting schemes for the tests 2,  2 and   . 4.4 Summary of the main power results The main findings of interest can be illustrated by the results for DGPs 4-5 for a dynamic forecasting model with a correction for serial correlation in the loss sequence. Only three tests have a monotonically increasing power function: the two versions of  and the    tests using a fixed forecasting window. All the other tests have a power function that eventually decreases to zero as the magnitude of changes increases in at least one and most often many cases. One exception is the  test with a rolling window, whose power appears high because of large size-distortions. The distortions are reduced as  increases as in 2 and 2 and the power is then decreasing to 0 as the magnitude of the change increases. Cases with tests having a power function that eventually reaches zero as the magnitude of the change(s) increases can also be found when dealing with other DGPs with shifts in the conditional mean (i.e., not DGPs 1-2, which only affects the variance) and other 12 forecasting methods. The  tests can have zero or trivial power even in the “static, non-robust” case; see DGP4 (fixed and rolling), and DGP5 (recursive). To compare the power of the tests, we focus on   = 05, a value with power not close to one or zero. We disregard the  test with the rolling window given the large sizedistortions. For a single break (DGPs 1 and 3) the   and  tests have equally the highest power. However, with multiple breaks (DGPs 2, 4 and 5), the   and  tests outweigh the  tests in all cases. In summary, the test with highest power is the    (for DGP1 the   and  tests have nearly the same power). The    test has a monotonically increasing power and also highest power for small values of the alternative. Hence, we recommend using the   test followed by the  , both with a fixed forecasting scheme. The loss in power when using the  test instead of the    test may, however, be DGP-specific since the changes involved, i.e., recurrent regimes, are those most prone to cause power problems for the  tests. With non-recurrent regimes, the power of the  tests would be closer to that of  . 4.5 Explanations for the power properties DGPs 1-2 are cases with single and multiple variance changes (Tables 4-1 and 4-2). For such type of instability the forecast model is unaffected by the choice of  or the forecasting scheme. The forecast model is still consistently estimated because the conditional mean of the variable to be forecasted is unchanged. All tests have nontrivial power in all cases. The substantial difference between 1 and 2 is caused solely by the choice of , showing its importance.  resolves this problem by maximizing the test statistics over all permissible  and achieves a reasonably high power in all cases. All tests proposed ( ,   and   ) have, overall, high power. Under DGP1,  has a slightly lower power than   and  , while under DGP2,  and   have a higher power than   , because the latter accounts for a single break. Also, under DGP2, the power of these tests does not reach one because of the nature of the breaks; i.e., two breaks with the first and last regimes being the same, which is the most difficult case to detect with a single break test (e.g., Bai and Perron, 2006). The problem is alleviated allowing for multiple changes so that the   and  tests are the most powerful in this setting. These observations are consistent with the theoretical results reported in Section 4.3. We now turn to models with mean breaks so that the conditional mean of the variable to be forecasted changes, i.e., DGPs 3-5 (results reported in Tables 4-3 to 4-5). The power functions exhibit non-monotonicity or a significant power loss because of three potential 13 sources. The first is the “robust effect” indicated with an “R” in the last row of a case with non-monotonic power. The second is the “window effect” indicated with a “W”. The third cause is the “dynamic effect”, indicated with a “D”. The simulation results pertaining to the window and dynamic effects follow from the theoretical ones in Section 4.3. The “robust effect” is due to a failure to properly account for serial correlations in the loss sequence. As is well known, when neglected breaks are present in the losses when constructing the HAC variance estimate, they inflate the sample autocovariances and the value of the bandwidth, thereby increasing HAC variance estimates and reducing power. This is a standard problem that has been discussed at length (e.g., Vogelsang , 1999, Crainiceanu and Vogelsang, 2007, Deng and Perron, 2008, Kim and Perron, 2009, Perron and Yamamoto, 2016, Martins and Perron, 2016, Chang and Perron, 2018). The “window effect” refers to the change in the loss sequence induced by using some window that separates the in and out-of-sample data and causes a loss in power. This applies, e.g., when breaks occur in the in-sample partition so that the model is not consistently estimated. The “dynamic effect” is the most pronounced and caused by in-sample contaminations when using a dynamic model. It is well known that if a dynamic model is estimated in the presence of mean breaks the coefficient estimate for the lagged dependent variable is biased toward one as the break magnitude becomes larger (Perron, 1989, 1990, 2019). This results in forecast errors being roughly the first-differences of those from a static model. Hence, the mean breaks become outliers in the loss sequence and the tests have no power. Note that the “dynamic effect” will not occur with the fixed scheme if  is sufficiently small and there is a chance that the model can be consistently estimated in a stable in-sample window. See the theoretical results in Section 4.3. We now explain some power functions reported in Tables 4-3 to 4-5. First, as shown in panel (a), with the static model and the fixed scheme, all non-robust tests have a monotonic power. However, the “robust effect” applies to tests constructed with a HAC variance estimate in panel (b), which is pronounced for 1 and 2 and also applies to 1, 2 and   when multiple breaks are present. With the fixed scheme, the “robust effect” applies to 1 under DGPs 3 and 5, and to 1, 2, 1, 2 and   under DGP4. With either the rolling or recursive scheme, it applies to 2 and  1 under DGP3, to 1, 2, 1, 2,  1 and  2 under DGP4 and to 1, 2 and 2 under DGP5. The results suggest not using  and  with a HAC variance estimate under any forecasting scheme nor   with multiple breaks. Note that even in panel (a) the power of 1 and 2 is sometimes very low. 1 has nonmonotonic power with the rolling scheme under DGP3, with both the fixed and the rolling 14 schemes under DGP4 and with the recursive scheme under DGP5, while 2 does so with the recursive scheme under DGP5. This is because they are affected by the “window effect” when a break occurs in the in-sample window. With a fixed scheme, the “window effect” applies to 2 under DGP4. With the rolling scheme, it applies to 2 under DGPs 3-4 and with the recursive scheme to 1 under DGP5. Also,  may lose power with the rolling or recursive scheme because the loss sequence takes a triangular shape (Figure 1). This applies to  1 and  2 under DGP4. For  , the “window effect” can be exacerbated by the “robust effect” (labelled “,  ” in panel (b)); cf.  1 with the rolling scheme under DGP3 and  1 and  2 with the rolling and recursive schemes under DGP4. The source of the “window effect” can be explained by the results in Figure 1. With the fixed scheme, the loss sequence takes a step-wise pattern for all three DGPs and tests. With the rolling or recursive scheme, it shows an abrupt increase followed by a gradual decline. For DGP-3, the increase occurs when the in-sample window covers a stable period and the initial date of the out-of-sample period coincides with the true break date. After, the window increasingly contains post-break data, which gradually causes a bias in the estimated forecast model and thus a decline of the loss sequence. More importantly, the shape is robust with either the static or dynamic model, except when the first break is included in the in-sample window (e.g., 2 under DGP5 with the dynamic model). The non-monotonic power functions are more pronounced with a dynamic model. Panel (c) of Tables 4-3 to 4-5 (DGPs 3-5) report power functions with non-robust standard errors. With the rolling or recursive schemes, this “dynamic effect” applies to almost all tests, even when the fixed scheme is used if the in-sample window is large and includes the break date; e.g., for 2 under DGPs 3-4 and for 1 2 and 2 under DGP5. This suggests not to use any tests with a rolling or recursive scheme when the forecast model has lagged dependent variables. The results for the dynamic model with the robust tests are presented in panel (d) of Tables 4-3 to 4-5, which highlight all tests with a non-monotonic power function. The results are consistent with the limit of the power losses in Figure 2, which showed that coefficient breaks are transformed into spikes in the loss sequence, getting narrower as the magnitude of the change increases when using the rolling or recursive scheme. 5 Empirical applications We now provide applications to illustrate the ability of the proposed and existing tests to detect changes in forecast accuracy; one related to the equity premium and the other to forecasting inflation. The results show the relevance of the theoretical and simulation results. 15 5.1 Equity premium forecasts The equity premium of the S&P 500 returns is constructed as the difference between the stock yield and the risk free rate, following Jagannathan et al. (2000). For the stock yield, we use the dividend price ratio plus the expected future growth rate of dividends. We use the historical average of the annual growth rates of dividends since 1928:01 to proxy for the latter. For the risk free rate, we use the 10-years treasury constant maturity rate. The data were obtained from Robert Shiller’s web site (http://www.econ.yale.edu/~shiller/data.htm). Figure 3 plots the resulting equity premium from 1980:1 to 2017:12. Jagannathan et al. (2000) and Lettau et al. (2008) noted that it was in an unprecedented low level in the 1990s and the early 2000s. The plot shows a sharp increase around 2007 because of the financial crisis. Afterwards, the equity premium stayed high likely because of the zero interest rate monetary policy. Lettau and Nieuwerburgh (2008) and Xu and Perron (2018) documented the presence of level shifts and structural breaks in excess return prediction regressions. We use a static, regressing + on  and a dynamic model, regressing + on  and  (both with a constant) to produce  -period ahead forecasts. We use the dividend price ratio as the predictor  , as is commonly done; e.g., Campbell and Shiller (1988) and Fama and French (1988). We consider forecast horizons  = 1 3, 6 and 12, and apply the following tests with the quadratic loss function: the  tests ( 1 and  2 for ̄ = 025 and 05), the   and   tests. The truncation  = 01 and the maximum number of breaks is 5 for the    test. We also apply the  tests with  = 331 and 347, which correspond to the outsets of the global financial crisis (2007:7) and the month prior to the initiation of the zero interest rate monetary policy (2008:11). We consider the  tests for 02 ≤  ≤ 08 (1) and for 03 ≤  ≤ 07 (2). The results are presented in Table 5. We first test for the presence of serial correlation in the loss sequence using the LM test of Godfrey (1978) whose results are presented in the columns labelled “1” for  = 331 and “2” for  = 347. They strongly indicate the presence of serial correlation in the loss sequences. Hence, the tests account for serial correlations using a HAC variance estimate. With the static model, there is evidence for a forecast breakdown with any of the proposed tests and the p-values when using the fixed scheme are an order of magnitude smaller compared to using the rolling or recursive scheme. Some -based tests also reject in all cases when the rolling window scheme is used but not with the fixed or recursive one. This is likely due to high power induced by large size distortions as documented above. The -based tests also show less rejections when long horizons are considered under the fixed and recursive schemes. With the dynamic model, we obtain a much clearer contrast. 16 The  tests now show very few rejections and the tests proposed strongly reject the null hypothesis for all horizons with the fixed window. As expected from the theoretical results, the  test fails to reject with the rolling or recursive scheme, while the   and    tests still show rejections, although much weaker in terms of p-values. 5.2 Inflation forecasts using the Phillips curve Forecasting inflation using the Phillips Curve was advanced by, e.g., Stock and Watson (1999) and Atkeson and Ohanian (2001), among many others. We use the model:  + = 0 + 1 () + 2 ()  +  where   is a measure of inflation,  is the unemployment gap (the unemployment rate minus and a measure of the NAIRU). The order of the lag polynomial 1 () was set to  = 1 or  = 3. Since the results are similar, we only report the case with  = 3. For the order of 2 () we report results for both  = 1 and 3, labelled ‘Dynamic 1’ and ‘Dynamic 3’, respectively. We consider two window sizes for the  tests: 1) 1 with  = 241, as in GR, so that the period before 1979 is within the in-sample and the highinflation period of Volker’s Fed Chairmanship is in the out-of-sample; 2) 2 with  = 301 (1984:1), in which case the out-of-sample window covers the Great Moderation. Orphanides and van Norden (2005) find that Phillips curve-based forecasts outperform an autoregressive benchmark prior to 1983 but without improvement for the period after 1984, while Dotsey et al. (2018) find that while the Phillips curve forecasts improve when the economy is weak, the improvement vanishes in the post-1984 period. In either case, the presumption is that a change in forecasting performance occurred. We consider the forecast horizons  = 1 3 6 and 12 months. We use the monthly real-time CPI (consumer price index) and unemployment gap data for the period 1959:1 to 2018:7. The data were obtained from the Federal Reserve Bank of Philadelphia Real-Time Data Set in which the CPI is available only after 1998:11 and the unemployment rate after 1965:11. The earlier data were obtained from the Swanson, van Dijk and Callan dataset (http://econweb.rutgers.edu/nswanson/realtime.htm). Since the data from two sources are very similar in the overlapping period, this merging should not affect the results. Also, as in GR we assume that a time-invariant NAIRU is embodied in the intercept. We construct the annual rate of inflation as   = (1200) ln( −1 ), where  is the CPI at month , whose graph is in Figure 4. The results are presented in Table 6 for the same tests as in Section 5.1. The LM tests for serial correlation in the loss function (Table 6; last two columns) indicate the presence 17 of serial correlation, so the tests are constructed with a HAC correction. For “Dynamic 1”, all proposed tests show strong rejections at any horizon when the fixed window scheme is used. With the rolling or recursive scheme, the  tests fail to reject in most cases, consistent with our theoretical results about the non-monotonic power with a rolling or recursive scheme. The results suggest multiple breaks since the value of   is much larger than   , although both tests reject the null hypothesis at the 1% significance level. More interestingly, all the -based tests, except for the 1 with the fixed and the rolling schemes, have no power. The results are qualitatively the same for “Dynamic 3”. Again, all the -based tests have no power, except for 1 with the fixed and the rolling schemes. In summary, the -based tests find no or very weak evidence of a change in forecast accuracy in the high inflation period of the Volker’s Chairmanship period. They are also not able to detect a change due to the Great Moderation that occurred in the mid-1980s. In contrast, the proposed    and   tests clearly show evidence of changes in forecast accuracy when the fixed scheme is used. 6 Conclusion We considered the issue of forecast failure (or breakdown) and proposed methods to detect changes in the forecasting performance over time. The aim is to assess retrospectively whether a given forecasting model provides forecasts which show evidence of changes (improvements or deterioration) with respect to some loss function. We adapted the classical structural change tests to the forecast failure context. First, we recommend that all tests should be carried with a fixed scheme to have best power. We considered a variety of tests: the GR test (a t-test for a change at some pre-specified date ); maximizing the  test over all values of  within a pre-specified range; a Double sup-Wald test which for each  performs a sup-Wald test for a change in the mean of the out-of-sample losses and takes the maximum of such tests over some range; we also proposed to work directly with the total loss series to define the   and the    tests. The only tests having a monotonic power function for all data-generating processes are the  and    tests, constructed with a fixed forecasting window scheme. The power of the   test is usually higher than that of the  test, hence it is recommended for practical applications. 18