The Need for Model Risk Management

Models have long been part of the toolkit used by the financial community to assess, price, and manage the various risks they face. As computing power increases, available data sets expand, and statistical techniques grow in sophistication, these statistical, financial, mathematical, and economic tools have become increasingly central to the operation of individual financial institutions and also to the financial system as a whole. Models and, in many cases, model systems built by taking the outputs of various models and using them as inputs to other models, provide many benefits to firms. Yet, they are also an emerging risk factor as model failure or misuse can seriously damage the finances, reputations, and even the solvency of firms.

Prudent management and, increasingly, regulations have institutions looking closely at the models they employ. This article presents the various components of the model risk management framework institutions use to meet their need to build, manage, and benefit from the models they use.

Models are now critical, if not central, to the success of financial businesses. There are numerous adverse consequences that could result from fundamental model errors, use of erroneous inputs or assumptions, unauthorized model use or changes, or use of a model outside of its developed purpose. Unfavorable model consequences include:

• Ill-formed underwriting
• Underpriced risk
• Incorrect assumptions about market liquidity in times of crisis
• Misguided asset diversification strategies
• Operational gridlock if models are not available or are viewed as unreliable
• Compliance issues from model decisions, particularly around retail lending
• Loss of institutional and market knowledge if models are viewed as just “black boxes”

In addition to individual model failure, many models use as input the output of other models (e.g., a portfolio model uses modeled probability of default values as input) – small errors in one model might be compounded or amplified when their erroneous results are fed into other models.

The post-crisis regulatory environment is looking to mitigate these various model risks by refocusing institutional attention on the models they use. In particular, the Basel Committee publications and, in the United States, the OCC’s Supervisory Guidance on model risk management (OCC 2011-12), require institutions to have a model risk management framework.

Model risk management is the establishment of a framework at an institution that not only provides insight into the use, nature, type, and development of models used at that firm, but is also a mechanism that controls a model’s deployment and range of applications, and (if needed) stops the use of those models.

This model risk management, per the OCC, “should include disciplined and knowledgeable development and implementation processes that are consistent with the situation and goals of the model user and with bank policy.”₁ Attaining regulatory compliance is a key goal of model risk management at institutions; therefore, this OCC guidance serves as a key starting point in the creation of a model management framework. In particular, the following areas are critical in model management frameworks:

• Model development
• Testing and validation
• Implementation
• Oversight and audit
• Governance

These areas are examined in the following sections.

The model development process must always begin with the establishment of clear goals. These goals – such as, efficiency improvements, reducing expected losses, or deploying better pricing – provide the teams developing the models with insight into their ultimate use. The goals should also supply guidance about the tests and criteria on which their models will be judged.

Once goals are established, the model development teams should undertake a methodical survey of the data, resources, and models available both inside the firm and from external sources. Models have been around sufficiently long enough that no firm, however large, has a “monopoly” on the modeling insights or data observations of a particular region or asset class. There are numerous data vendors, dedicated statistical shops, and model suppliers that can potentially meet the modeling needs of a firm. These options should be examined in conjunction with internally available resources, as it is often the case that external tools or data are used with internal resources. For example, third-party data sets covering time periods and regions not available internally are combined with internal data sets to produce a richer modeling data set. Additionally, external models and statistical groups are often used to benchmark or function as challengers to internally-developed models.

The actual model development process must constantly produce documentary evidence to support model choices. Additionally, model development should:

• Strive to avoid oversimplification
• Have rigorous variable selection and variable exclusion criteria
• Evaluate the appropriateness of qualitative overlays and modifications to date
• Employ the best available statistical and analytic rigor to their modeling effort

The result of these efforts, beyond a model the institution can use, should be a clearly documented and theoretically justified package of code and source data sets (with all data modifications and overlays clearly defined and explained) ready for evaluation and testing by groups not connected with the modeling effort.

Testing and validation are critical parts for both the development and ultimate acceptance of a model by the business lines using the model and external regulators reviewing the models, when deployed. It should be viewed as a complementary and integral part of the model development process, as the clearly defined code and thorough data preparation needed for testing and validation greatly aids the initial development and subsequent improvements to the model.

Model testing and validation teams need to be somewhat independent from model development and use, typically through differing reporting lines. The personnel conducting the validation should be able to, based on their skills and organizational standing, challenge model developers on a regular basis. In particular, they should examine and effectively comment on all aspects of a model, including inputs, analytics, reporting, and performance. If needed, they should be able to prevent a model that doesn’t pass their criteria from being utilized by the institution.

Testing and validation focuses on three areas:

1. Testing and validation of the conceptual soundness of the model
2. Identifying potential limitations in the model and in its range of applicability
3. Evaluating model effectiveness, both through back testing and periodic reviews of model results

Model testing and validating utilizes a variety of methods. Unit testing of model variables checks for accuracy, demonstrates model stability and robustness, evaluates the proper fit of variables, and, particularly through sensitivity analysis, assesses model limitations by entering a range of extreme model values. Out-of-sample testing, if possible, includes the use of external data, further tests model stability and performance, and provides a particularly strong challenge to the model under consideration. Finally, regular analytic and statistical model reviews aim to ensure that models perform as expected and within their design parameters, highlight potential model limitations and prescribe corrective actions, and reaffirm the model’s limitations and range of applicability.

The testing and validation areas should regularly employ either specified or statistically determined “stressed” model input variables to evaluate the soundness and performance of models under consideration

Typically, after iterative and rigorous model development, testing, and validation work, the implementation of models is an often overlooked activity at institutions. Effective implementation should not be discounted, as there are numerous ways in which an erroneous deployment will negate the hard work and resources expended to create and validate the model. For example, many deployed models blindly pick up dated financial information, utilize hard-coded estimates of market volatility (i.e., the market estimates that largely define the model’s behavior in times of high stress), and execute queries against input data stores so inefficiently that end users successfully demanded the ability to bypass the use of the model.

To ensure a successful implementation, the model development team should understand the ultimate platform that will host the model, while technology resources should have early opportunities to comment on the model’s overall goals. This enables the team to identify potentially burdensome technical requirements early on and design mitigations. For example, one model’s initial requirement of instant real-time data feeds from tightly-controlled source systems was, on investigation of the nature and criticality of this data, changed to a data snapshot generated daily by the system. The overall stability, security, and speed of the model greatly increased, while the model’s technical complexity decreased. Furthermore, the model’s overall predictive power was not affected.

Increasing the overall visibility of model inputs and outputs should also be a goal of the implementation. Models typically receive automated feeds from many sources, including, for example, interest rate curves, cost of funds estimates, and balance sheet data. Additionally, models likely utilize various infrequently updated or even hardcoded values, such as the firm’s unit costs, leverage targets, and target debt service coverage ratios. This data is often a critical driver of model results, so its misuse or misapplication can produce erroneous model results. During times of stress, for example, one does not want data that assumes market liquidity and an ample supply of buyers and sellers across all risk categories. This would likely greatly underestimate the risks of being in a certain market and potentially precludes management from identifying and stopping the addition of more risky exposures on their balance sheet. In summary, model users and the groups overseeing the model should be aware of this data, be able to quickly find the current values being used, and change this data in a timely fashion.

Models must capture the complexity of the institution and the phenomena they want to simulate. In practice, while the creation and testing of models is somewhat straightforward, ensuring that a particular model “works” (for lack of a better term) is particularly challenging.

Model oversight and audit aims to build confidence in models by subjecting them and their results to a variety of conceptual and quantitative criteria. While testing and validation focuses on building up a rigorous body of evidence to support a particular model, oversight and audit aims to provide an effective challenge to these models by:

• Challenging the modeling teams to establish the conceptual soundness of their models and why their chosen approach should be used over competing approaches
• Establishing or expanding identified limits to a model and then identifying, quantifying, and proposing changes to that model
• Reviewing the history of decisions made by a particular model and then comparing these decisions and their outcomes with both the model development data set and the results predicted at the time of model creation

The oversight and audit area, as the testing and validation area, requires knowledgeable teams that are able to identify, quantify, and propose changes to models. Specifically, organizations need these areas to have the incentive, competence, and influence to effectively understand, audit, and challenge the models.

Model governance determines which models are used, the range of activities they cover, the type and nature of tests they need to be subjected to, and, ultimately, when to stop using a particular model. Governance provides the first and last methods of controlling models, as an institution’s management structure ultimately is responsible for the ways models are used. Key governance activities should include:

• Maintaining a comprehensive inventory of information used in models, data used for model development, test results, the models being used, models recently retired, and proposed models
• Mandating, and strictly using, a revision control system for model code
• Aiming to build up sufficient model knowledge outside of groups that do model development, while recognizing that modelers and decision makers typically come from differing backgrounds and potentially view models differently
• Establishing limits on model use and alert mechanisms when models exceed the limits
• Utilizing strictly-defined roles and responsibilities
• Building the capabilities to regularly monitor model performance through unit testing, operational efficiency checks, performance against model development metrics and similar benchmarks, and out-of-sample testing

These steps greatly increase a firm’s ability to know, monitor, and govern the models used at their firms. Models offer many advantages, yet they must be thoroughly understood. They must be rigorously developed, tested, and monitored, and they must be governed and controlled by the key risk and management areas of an institution. A model risk management framework enables a firm to accomplish all these goals, thereby managing their models instead of being (mis)managed by them.