Dynamic Model-Building: A Proposed Variable Selection Algorithm

In this article, we propose an innovative algorithm that is well suited to building dynamic models for credit and market risk metrics, consistent with regulatory requirements around stress testing, forecasting, and IFRS 9. Our method combines economic intuition, statistical rigor, and computational efficiency. We compare the results of the algorithm with standard industry best practice procedures such as stepwise and LASSO and demonstrate its advantages.

Introduction

Modern stress testing exercises require explicit links between market and credit risk metrics with core macroeconomic variables. Such linkages can be established using various econometric techniques that must comply with regulatory requirements, including those of the US Federal Reserve, the UK Prudential Regulation Authority (PRA), and the European Banking Authority (EBA). Such requirements include the timing and magnitude of a response of the modeled metrics to hypothetical macroeconomic shocks.¹

In this article, we propose an innovative algorithm that is well-suited to building dynamic models for credit and market risk metrics, consistent with regulatory requirements around stress testing, forecasting, and International Financial Reporting Standard 9 (IFRS 9). Our method consists of three main steps that combine economic intuition (macro constraints), statistical rigor when selecting model drivers, and computational efficiency to find an optimal parsimonious model with strong in-sample fit and forecasting properties. The user can select a subset of potential macroeconomic drivers and run the customized best-subset selection algorithm to choose the optimal combination (and lags) among these candidates. The selected model is then diagnosed and validated to ensure it is satisfactory for stress testing. This paper also demonstrates the methodology used by Moody’s Analytics to design models in line with the regulatory requirements, and shows its application to selected market risk metrics.²

To set up a model, one needs to find the optimal combination of explanatory variables to forecast a target risk metric. A good model compliant with regulatory demands should include all important drivers, while still being parsimonious, suitable for forecasting, easily interpretable, and in line with economic theory. Practitioners often have to model risk factors for which standard theoretical models abstract from world complexity. Thus, the vector of drivers suggested by economic theory must be extended to include other potential variables that capture market interconnectedness, spillover effects, and other potential determinants. In many cases, historical data and statistical analyses help select what additional variables to keep in models. The variable selection method we describe here is not pure machine learning but is customized to combine economic intuition, the model developer’s experience, and historical data analysis.

The model-building procedure that we often use at Moody’s Analytics consists of three steps. First, we select potential drivers of the target risk metric that are likely to be the most important based on economic intuition, experience with similar models, and historical data analysis. Second, we run a customized variable selection algorithm to choose the optimal combination of drivers. Finally, the subset of likely models is diagnosed and validated to identify a handful of the best-performing forecast models. The model developer can then select an optimal model satisfactory for stress testing.³

One of the key building blocks of this procedure is the best-subset selection algorithm. It first chooses the subset of likely models according to the user-defined correlation and selection criteria. These criteria may include the prior expectations on correlations between the drivers, the estimated signs (based on economic intuition), and a threshold on the statistical significance. The resulting subset of candidate models is then sorted according to ranking criteria that include various measures of in-sample fit and predictive ability.

This paper is structured as follows. In the first section, we discuss key existing variable selection algorithms. The following section shows Moody’s Analytics approach to building a typical model for stress testing. The last section demonstrates the model selection algorithms using the examples of option-adjusted corporate spreads and UK government bond yields.

Variable Selection Algorithms

The most commonly used automatic variable selection algorithms include subset selection, stepwise regression, and shrinkage methods (for example, LASSO).⁴ These methods are based on choosing models with the best in-sample fit as measured by statistical tests (such as F and t-tests), statistical criteria (such as mean-squared error, adjusted R², information criteria, or Mallows' C), and statistical stopping rules (for example, p-values indicate if a variable stays in the model or is excluded).

Each variable selection algorithm has its advantages and disadvantages. Subset selection considers all possible models (good and bad) with various combinations of potential drivers and selects models that satisfy predetermined criteria. This increases the variance of the subset method relative to stepwise and LASSO, which follow certain paths in choosing an optimal model. At the same time, the subset selection is the least biased because it selects a model that globally optimized predetermined criteria. This might not be true for stepwise and LASSO.⁵ However, this advantage comes at a cost – subset selection is a computationally intensive algorithm. As the number of variables increases, the number of models grows exponentially (Hastie et al, 2013).⁶

Three Stages of Model-Building

The variable selection algorithm is only a part of the model-building process for a target risk metric. We believe that models built using pure data-mining techniques, though they may fit the historical data well, are not sufficient for regulatory compliance. Optimal models used for forecasting and stress testing should have a combination of statistical rigor and economic theory. Models built this way enjoy the additional benefit of ease of interpretation.

Our model-building consists of three stages. In the first stage, economic theory, historical correlations and graphical analysis, expert judgment, and experience with similar models guide us in the choice of potential drivers. Our choice is typically restricted to forecasts of drivers provided by the regulators, generated within Moody’s Analytics macroeconomic models, and other available risk metric models. In the second stage, these potential drivers enter the customized best-subset variable selection algorithm which may include the analyst’s expectation on the signs of coefficients, threshold p-values, and correlation between drivers. The output of the second stage is a list of ranked models from which one can select the optimal model. The final stage involves the diagnostics and validation of the optimal model selected in the second stage. We analyze the forecasts, run post-estimation diagnostics, and perform back-testing and sensitivity analysis.

It is crucial to handle the model space properly when using the best-subset selection algorithm, as specifying many potential drivers implies a large number of candidate models. To reduce the computational burden, we use a supporting dataset with the binary representations of the potential models. This significantly reduces the computation time because the potential models do not have to be generated “on the fly.” Also, we can achieve a significant reduction of the model space by setting a ceiling on the maximum number of variables in the resulting models. In Figure 1, we show the number of models as a function of the number of potential drivers m and the number of final drivers k, such that k ≤ m.

Selection and Ranking Criteria

Figure 2 illustrates the customized variable selection algorithm. Given the set of potential models, an analyst can choose to exclude models with highly correlated drivers to avoid the adverse effects of multicollinearity (Greene, 2012).⁷ The selection criteria, which include analysis of the signs of the coefficient estimates and their statistical significance, can be used to eliminate models that do not comply with prior expectations. In this case, once all the potential models are estimated, those with statistically insignificant coefficients and “wrong” signs can be disregarded depending on the problem setup. These criteria help us narrow the model space to include only relevant models that will be analyzed further.⁸

The next step is to rank the remaining models according to the ranking criteria. A large number of criteria exist to evaluate regression models used in the context of variable selection. Traditionally, the ranking of models is based on in-sample fit, which is a measure of the distance of fitted values to actual historical series. In addition, the forecast properties of the models should be considered. The adjusted R² compensates for the fact that the inclusion of more drivers produces larger R² even when the extra variables are irrelevant. In addition to this relative measure of fit, we also calculate an absolute measure, the root mean squared error (RMSE). Since the main purpose of the model is forecasting, the ratio of adjusted R²/RMSE is one of the most important ranking criteria, with higher values indicating better fit. Other measures such as the information criteria impose a heavier penalty on model complexity, applying different punishments for the inclusion of additional variables.⁹

Validation

In addition to measures of in-sample fit, we also consider a model’s predictive performance since a good in-sample fit does not imply good forecast properties. Recent academic work has seen a surge in computationally expensive validation methods. In general, historical data are split into estimation (training) and validation (hold-out) subsamples. To assess the forecast properties of a model, one has to compare its forecast values and actual realizations. For example, the predictive accuracy of a model can be measured by the mean squared error on the hold-out samples. A complementary technique is stability analysis, in which coefficient estimates on hold-out samples are tested for their equality with the estimates obtained on a complete dataset.

One way to split the data for validation is to subsequently remove the most recent observations. However, there are often not enough data for extensive back-testing since the sample size could be too small to be a reliable indicator of future forecasting performance. Alternative versions of defining training/hold-out subsamples are leave-one-out, k-fold cross-validation, and bootstrapping. We typically perform fivefold validation as it is shown empirically to have prediction error rates that suffer neither from excessively high bias nor from very high variance.¹⁰

To perform cross-validation, we first split the historical data into five non-overlapping windows. Each iteration will have a time-ordered training subset and an adjacent validation subset. The variable selection algorithm is run on the training data to select the optimal model. The selected model is used to create a forecast for the validation subset. The cross-validation root mean squared prediction error and its confidence intervals are then computed to assess the predictive ability of the model.

Finally, we perform a sensitivity analysis of the model. We look at the impulse responses of the target risk metric to each driver and standardized coefficient estimates. For impulse response analysis, a shock occurs only in one driver at a time since the shocks in different drivers are independent and the model is used to forecast the dependent variable.

Stationarity and Cointegration

In market risk applications, it is very common to deal with non-stationary variables. Empirical modeling faces difficulties when time series are non-stationary in that the means and variances of outcomes change over time. If unrelated time series are non-stationary due to past shocks of different origins, their correlation is still likely to appear significant, generating nonsensical relations. With wrongly assumed stationarity, the uncertainty is underestimated, affecting accuracy and precision of the forecasts. If appropriately handled, however, non-stationary variables can help to elucidate long-run relationships in the data.

Formally, the Box-Jenkins methodology to model time series requires both dependent and independent variables to be stationary. If this requirement is unmet and the variables are integrated of different orders – for example, one is stationary I(0) and one is non-stationary I(1), or vice versa – this is a case of spurious regression. If both variables are integrated of the same order (typically I(1)) there are two possibilities. If there exists a linear combination of the variables that is I(0), the variables are cointegrated. The cointegrated variables share the same stochastic trends and the ordinary least squares (OLS) coefficient estimates are super-consistent.¹¹ If the two variables are not cointegrated, the regression is spurious.

When specifying a model, a natural approach would be to pre-test the target risk variable and drivers in the regression equation for unit roots. If they are stationary, we can safely apply a classic modeling methodology. If they are non-stationary, we can either transform the data to induce stationarity or check if they are cointegrated. We typically prefer the latter approach, as data transformations are often problematic in terms of interpretation and intuition behind the link between the variables.¹² To test for cointegration, variables in the model need to be of the same order of integration. If they are, then we can further test for cointegration using Johansen and Engle-Granger types of tests.¹³

One should be careful, however, when interpreting the results of stationarity and cointegration tests in finite short samples. These tests have low power in distinguishing highly persistent stationary series from non-stationary processes and they work better for longer time frames. So, 15 years of quarterly data (60 observations) is better than five years of monthly data (60 observations). For time series models, we typically work with a short time frame. Also, the presence of structural breaks makes it difficult to distinguish a non-stationary series from stationary data around a break. Thus, a thorough examination of data followed by a combination of different tests is suggested as a way to eliminate some of these shortcomings.¹⁴

Variable Selection Examples

We illustrate how to apply our model-building procedure to specific problems based on actual client projects. In addition, we evaluate the results of different variable selection algorithms. In our exercise, we focus on stress testing of market risk metrics for regulatory and Moody’s Analytics internal scenarios. The market risk metrics are the rating structure of the US corporate spreads and the term structure of the UK government bond yields.¹⁵ We consider PRA 2016 scenarios (baseline and stress), Comprehensive Capital Analysis and Review (CCAR) 2016 scenarios (baseline and severely adverse), and two Moody’s Analytics scenarios (baseline and protracted slump).

The regulators explicitly prescribe the timing and magnitude of the response of a risk metric to a shock. To satisfy these requirements, the stress testing models must follow the prescribed paths of the exogenous variables provided by the regulators. The inclusion of lag structure in the models might change the timing and magnitude of the shock in the risk metric. For example, ARIMA(p, q) might not deliver sufficient stress if the period of the hypothesized shock is preceded by tranquility in the historical data. To avoid possible inconsistencies with the prescription of the regulators, we typically limit the use of lag structure in the figures that follow, although the algorithm is flexible enough to incorporate it.

The target risk metric is a term or rating structure whereby the variations across various maturities and ratings are observed over time. We first focus on reducing the dimension of the cross-section units to a smaller number of underlying factors. To model the term structure, we follow a modified Nelson-Siegel approach. First, we obtain the forecasts on the level and slope. ¹⁶

Second, the forecasts of the key maturities, such as 1- and 10-year, are built back from the level and slope. Finally, the forecasts of all the remaining maturities are obtained by interpolation between the key maturities based on their historical ratios. This approach properly aligns maturities and is also robust to possible curve inversions in the stress scenarios.

In the first stage of model-building, we select potential drivers guided by economic theory, historical data correlations, and our experience with similar models. The list of potential drivers considered for the three groups of market risk metrics is shown in Figure 3. These drivers have forecasts from the Moody’s Analytics macroeconomic country model or a regulator. In the actual client project our list had more variables, but we have shortened the list to keep the presentation manageable. In the second stage, we run automatic yet flexible and interactive algorithms to select variables for inclusion in the model from the pool of potential drivers.

Global factors are included in the list of potential drivers to maximize the informative content of core macro and financial drivers. They reduce the dimensional space of the explanatory variables, thus achieving more parsimony and flexibility. Principal component analysis is used to extract relevant business cycle information from the sets of macroeconomic variables. The key factors include global measures of growth, equity, and volatility. Each of these factors represents a variation from a wide range of constituent macro and financial variables as well as geographical territories. Global growth factor captures the dynamics of global economic activity and is an aggregate measure of the real GDP growth of key world economies.

In the second stage of model-building, we run automatic yet flexible and interactive algorithms to select variables for inclusion in the model from the pool of potential drivers. We consider best subset selection, forward and backward stepwise, and LASSO to choose a best model for the principal components of the UK bond term structure and the corporate spread rating structure.¹⁷ In the best subset selection algorithm, we restrict our attention to models with a maximum number of drivers allowed in the models to k={max{3,max{k_lasso,k_stepwise}}, where k_lasso and k_stepwise are the number of drivers in the optimal model selected by LASSO and stepwise algorithms, respectively. In actual client projects, we typically use the best subset selection algorithm, while here we compare results of alternative variable selection methods.

Stress Testing the Corporate Spreads Rating Structure

The rating structure consists of seven ratings depicted in Figure 4. The curves are aligned across ratings and the spreads widen during periods of economic downturns and market uncertainty, including the dot-com bubble around 2002 and global financial crisis in 2008. High yield corporate bonds exhibit wider spreads and display more variation than those with investment grade ratings. The rating structure does not have inversions or crossovers.

We first forecast the level and slope components extracted from the rating structure. We then obtain the forecasts of the lower rating C and upper rating AAA based on the two components. Finally, we interpolate the remaining ratings based on the historical distance to lower and upper adjacent ratings.

For the level equation, the optimal model has two drivers, namely the global equity factor (GEF) and the US average A corporate spread based on the ratio adj.R²/RMSE and cross-validation criteria. The model based on the information criteria has three drivers, slightly lower adj.R²/RMSE ratio and cross-validation error, although largely overlapping confidence intervals. The model selected by the stepwise and LASSO algorithms has worse in-sample fit and predictive ability, as shown in Figure 5. For the slope equation (Figure 6), the optimal model has three drivers: the US 10-year government bond yield, GEF, and global equity volatility factor (GEVF). This model is superior to the one selected by the stepwise and LASSO based on both in-sample fit and cross-validation statistics.

Forecasts of the rating structure across the scenarios are shown in Figure 7. They are consistent across ratings and exhibit spreads wider under the CCAR than under PRA or Moody’s Analytics stress scenario.

UK Government Bond Yields

The UK government bond yield term structure consists of 19 maturities, shown in Figure 8. Sharp upswings in the short-term rates reflected the Central Bank controlling tight-money policies. In this expansionary period, the spread between short- and long-term rates is very narrow. Following the peak in 2008, short-term rates fell sharply with economies in recession and policy rate cuts, while the longer-term rates formed a relatively smoother downtrend. This created a wider spread between short- and long-term rates, increasing sharply the slope of the swap rate curves, that is, the difference between the long- and short-term rates. To model this term structure, we first forecast the level and slope components. Second, the key maturities are modeled with level and slope added to the list of potential drivers. Lastly, the rest of the maturities are interpolated in between the key maturities.

Figure 9 and Figure 10 contain the estimation results of the key maturities. For the three-month bond yield, the optimal model includes the level and slope. This model has the best in-sample fit as measured by adj.R²/RMSE and Bayesian information criterion (BIC) as well as the lowest cross-validation RMSE. Notably, despite the fact that the model selected by the stepwise and LASSO shows better in-sample fit and predictive ability than the optimal model, it suffers from collinearity. The mean variance inflation factor (VIF) indicates the presence of multicollinearity. This model was not considered in the best subset selection algorithm since the maximum allowed correlation between drivers is set at 0.85. The optimal equation for the slope includes the level, slope, and volatility.

Forecasts of the term structure of the bond yields are depicted in Figure 11. The longer-term rates increase in stress in PRA scenarios while they decline in Moody’s Analytics and CCAR stress scenarios, since the UK is regarded as a safe haven.

Concluding Remarks

This article presents the methodology we use to build stress testing models for target market and credit risk metrics. This framework is highly flexible to accommodate various increasing regulatory demands and also classical forecasting practices. This is done by customizing the model selection algorithm in line with analysts’ prior expectations and assumptions.

The three-step methodology combines economic intuition and a customized best-subset selection algorithm to find the optimal parsimonious model with good in-sample fit, forecast properties, and consistency with regulatory assumptions. The model developer can be involved in every step of model-building, from selecting the pool of potential drivers to customizing the variable selection algorithm and choosing the optimal model from the list of potential models ranked according to predefined criteria.

Further, this methodology is applied to two types of market risk instruments. We demonstrate that this approach is most suitable for our purposes compared with alternative procedures. In our examples, we find some differences between the optimal combination of drivers selected by the best subset, stepwise, and LASSO approaches, although the selected models have similar predictive ability and in-sample fit. The optimal models selected by stepwise and LASSO are in the range of the 10 best models as ranked by the best subset selection.