Do Banks Need Third-Party Models?

“But in consequence of the division of labor, the whole of every man’s attention comes naturally to be directed towards some one very simple object. It is naturally to be expected, therefore, that some one or other of those who are employed in each particular branch of labor should soon find out easier and readier methods of performing their own particular work ...” – Adam Smith

Sometimes, we encounter a perception among banks that regulators expect them to build all their risk management tools in-house and use only internal data. Other times, we find that banks are free to buy external data, mainly when internal supplies are low, but that models estimated using industry-wide databases are unacceptable for use in stress testing, unless they are heavily customized and calibrated to portfolio-specific data.

Such extreme views are at odds with the stated aim of the stress testing experiment. In the wake of the global financial crisis, legislators around the world instigated reforms designed to force large banks to better understand the risks associated with their books. Regulators envisaged that stress tests, when combined with enhanced regulatory scrutiny, could minimize the potential for future government bank bailouts and thus solve the problems of adverse selection of risks and moral hazard.

We describe this process as an “experiment” because, while hopes are high, no one yet knows whether stress testing will actually reduce overall banking system risk. For the experiment to be a success, a significant period of time needs to pass without a bank-failure-induced recession. For the US, a period of 50 years seems appropriate given that the Great Depression, the Savings and Loan Crisis, and the Great Recession all occurred during the past century.

Truly understanding all the risks a bank takes at a given time is a daunting challenge. If analysis of an external data set, or work by a third-party analyst, can help a bank or regulator understand risk more fully, does it matter that the arrangement involves entities and resources external to the bank? We contend that for the stress testing experiment to succeed, regulators should welcome and encourage research and development, as well as data collection and improvement, by anyone who is willing to engage in such activities. This call to arms extends not only to banks, bank employees, and regulators but equally to academics, data collectors, consultants, students, advisors, and freelance analysts. After all, if an amateur astronomer identifies the comet on a collision course, should the analysis fail validation because he or she is not employed by NASA?

One of the main causes of the US subprime crisis was that a number of major institutions had taken long positions in this risky sector. If subprime had instead remained a niche industry with few players, the crisis may never have materialized. This type of behavior is a very common element in historical banking crises. By their very nature, credit-fueled asset price bubbles – the most dangerous phenomena for the survival of banking systems – are characterized by widespread irrational exuberance of many borrowers and many lenders. Banks see their peers making excess profits lending to a certain group of people and rush to join the party. As each new bank enters the market, the risk the initial entrants face rises even if their risk appetite and underwriting standards do not change. A safe, profitable activity for a few banks becomes gravely dangerous when many engage in the same behavior.

The level of risk in a bank’s book depends critically on how the book aligns with those of other banks. In other words, it was important for a hypothetical subprime lender circa 2006 to know and consider the implications of so many other similar lenders being active during the critical time. Furthermore, conservative mortgage lenders that were not engaged in subprime needed to know about and account for the effects of distortions to their industry created by the growth of lending to borrowers at the opposite end of the credit quality spectrum.

To gain a full understanding of risks, therefore, banks must explicitly reference data collected from beyond their own walls. This statement is true for a bank with poor internal data assets, where the external information serves a further purpose of giving modelers something to model. It is also true for banks with abundant internal data at their disposal that can conceivably build any model.

Portfolio alignment across banks seems to be a necessary condition for banking sector stress. Ironically, if lending markets are healthy and thus unlikely to cause problems for large financial institutions, banks probably can safely consider the nature of their portfolios in isolation and gain a largely accurate view of baseline portfolio risk. It is only under stress, when markets are distorted by collective irrational exuberance and its aftermath, that the need for external data becomes truly critical. But it is, after all, the stress events that most interest us here.

Where do banks source the needed external data to accurately gauge stress? Call reports might be one ready source. For some applications, however, these currently public sources may be insufficiently detailed. Regulators could make the data they collect from banks as part of the stress testing process public, though lawmakers or privacy activists might not favor this. In addition, many of the biggest players in the subprime saga were shadow banks and potentially invisible to banking regulators.

For financial institutions to be willing to share their data with their competitors, the data must be suitably anonymized and aggregated. Private-sector companies have historically provided a conduit through which banks can happily share information without giving up any sensitive trade secrets. Data gathering start-ups may already be collecting the data that holds the key to identifying the next crisis. They should be encouraged to continue the search and, when successful, charge an appropriate price for their products.

Having established the case for the use of external data, the next question concerns who should model it. The concept of stress testing a bank’s book, especially against a pre-specified, exogenous macroeconomic scenario, is a relatively young discipline compared to other risk management practices. The reality is that in universities, NGOs, regulatory offices, banks, and consulting companies, dedicated professionals are busy trying to better understand stress testing methodology to make it easier for banks to implement and make it more accurate for users.

Regulators and academics will presumably continue to engage in considerable innovative effort. Academics will likely pursue stress testing because it is consequential, yields large amounts of interesting data, and is intellectually stimulating. Regulators, meanwhile, will seek to innovate out of pure necessity. These organizations must seek the best available stress testing tools to confront rogue banks and stay ahead of the next banking crisis.

Banks have a strong incentive to maintain at least a minimum standard of stress test model performance. Shareholders expect banks to pay dividends, and if a failed stress test results in a reduction or suspension of such payments, the incumbent CEO could lose the support of the shareholders. Nevertheless, among the bank holding companies that have not suffered a qualitative failure, it is unclear whether those institutions that took the stress test most seriously were rewarded for their efforts, compared to those that merely did enough to fall over the line. Because a bank’s management team represents the interests of shareholders, it is more likely to invest in activities that increase shareholder value than in those that minimize the FDIC’s losses should the bank happen to fail.

Regulators, of course, have called on banks to stitch the stress test into their day-to-day operations. For this to become a reality, however, stress test models must yield insights that enable business managers to lower risk for a given return or clearly increase the profitability associated with running the portfolio. If using the model does not yield such insights, banks may pretend to take the models seriously when under the regulatory spotlight but make no actual changes in their banking behavior or operations.

In terms of downside risk management, the incentive for banks to innovate may therefore be thin. True, they must take the stress test seriously enough that the probability of a failure is sufficiently low, but there is little incentive for them to do any more. If the stress test process is improved to the point where managers can rely on the models to make money, the innovation floodgates will open and banks will be motivated to invest in research that will give them an edge over their competitors, on both the upside and downside.

Vendors, meanwhile, invest in innovation with the hope of realizing a financial return. One way that a vendor can be rewarded for innovation is by becoming the standard source for a particular analytical tool like a credit score, an asset price forecast, a probability of default, or a rating. They can then charge a premium over an upstart market entrant. Once a vendor has been established as the source and the tool is being used productively for business decisions, it will be in the vendor’s best interest to ensure the quality of the analysis and carefully maintain the infrastructure used to produce the information. If the provider is motivated by something other than profits, continuity of service may be illusive. Consequently, market participants may be reluctant to invest in adopting tools that are not produced by forprofit entities.

A recurring footnote in the Federal Reserve Supervisory's Dodd-Frank Act Stress Test (DFAST) results provides a sample of approximately 25 vendors whose analytics the Federal Reserve Supervisory used to conduct their DFAST analysis.¹ Of these, only three are not-for-profit organizations; four are financial institutions. The rest are for-profit vendor companies that rely on a combination of financing from investors and revenue from the sales of analytics to fund both their operating costs and whatever research and development they conduct.

Although many of these firms are likely to do some bespoke consulting work, we find it interesting that well-known management consulting firms (such as the Big Four) are not on the list. These firms help banks both build internal models using bank data and validate their use of vendor models. As such, they may be reluctant to sell analytics as it would conflict with their core business.

If a bank develops a promising new technique that it uses to beat the market, it will likely be reluctant to sell such analytical tools to similar institutions; for competitive reasons, other financial institutions may also be reluctant to buy them. However, if a vendor does achieve a breakthrough, it will expect to be well compensated for its success, but this will be achieved through propagation of its innovation throughout the industry.

While vendors will naturally seek to protect their intellectual property, the propagation of soft knowledge in the industry will likely be greater than if the technology is locked in a specific bank’s intellectual vault. While all scientific progress is welcome, information externalities are arguably greater if a vendor, as opposed to a bank, is responsible for the breakthrough.

Home-grown analytics, those produced within a bank, have the advantage that bank managers and executives retain complete control. Vendor analytics, in contrast, often reflect the experiences of many market participants, offer more features and documentation, and are less expensive to implement.

Because the incremental costs of making analytics available to additional clients declines as the number of clients using the analytics grows, it is generally efficient for one party to produce them and then to share them with multiple parties. Our interpretation of what regulators have written about the use of vendor models is that they expect financial institutions to take ownership of whatever analytics and data they use, but this does not imply that they should necessarily build their own analytics with their own data in all cases.

In the analytics business, producing new products and supporting existing products is work, but the work product is scalable across many users. New products frequently necessitate that a firm invest in collecting data and developing analytics for several years prior to the sale of the product to the first client. There are no guarantees that the new product will be successful.

Supporting data and models requires documentation, validation, and periodic model updates. Users also require guidance from vendors on the use of the model. Much of this effort is reusable: The needs of one client will overlap heavily with the needs of other clients. Nevertheless, because every financial institution is different, there will always be a customization aspect to the provided support. Consequently, the marginal costs of providing analytics will decline as the number of users grows, but the marginal costs of providing support services always remain positive.

Having a set of firms producing data analytics for many banks is more efficient than every bank attempting to replicate all of these products on their own. Further, the more heavily used the product, the higher the quality of the product. Suppose there is an issue with a particular model. If 10 banks are using the product, the issue is likely to be discovered sooner than if the client base consists of only one institution. If one bank discovers an issue that affects nine others, beneficial externalities accrue to all banks as a result of the actions of the observant institution. Such externalities are not present in a system that relies only on internal modeling.

One issue that is often mentioned in the context of scalable analytics is that it can foster potentially dangerous concentration risks. Suppose a particular model becomes an industry standard, to the point where all banks must use the model’s predictions to be viewed as competitive by financial markets. If the model has a structural flaw that causes it to under-predict losses in the industry, this could conceivably destabilize every institution using the model instead of just one.

Assume that the vendor model under consideration produces accurate insight into the riskiness of a portfolio that simply cannot be gleaned from any other source. We are not saying that the vendor model produces a complete picture of risk, just that it shades a particular color in a way that cannot otherwise be captured by risk managers. Forcing banks to exclude the use of such a vendor model will result in an incorrect rendering of the risk picture. Thus, system risk could decline if all banks adopted the use of the vendor model, as only the model’s users would know that the fig leaf is, in reality, poison ivy.

Analytical concentration risk therefore depends critically on what exactly the concentration is. How the information is used by banks is also critical. If the vendor model contains unique, accurate, and pertinent information, it is not necessarily a bad thing if all banks adopt the model. If the model is flawed, a feature common to every model ever built, the onus shifts to the bank’s risk managers to ensure that the information is correctly harnessed in assessing portfolio risk. We would never advocate blind acceptance of one of our models or, indeed, of any model built by any mortal. This classification certainly extends to any and all of our current and future competitors. Downside model concentration risks tend to be realized only when banks confuse a model’s predictions with gospel truth and take actions based on that “truth.”

The best defense for this issue is the concept of “effective challenge,”² which is a regulatory expectation for all models that have a material impact on business decisions. The Federal Reserve Supervisory defines an effective challenge as a “critical analysis by informed parties that can identify model limitations and assumptions and produce appropriate changes.” For the challengers to be effective, they must be independent from the model builders, have the appropriate degree of expertise, and enough influence so that their challenges will be appropriately addressed. A well-built third-party model can certainly play this role in the validation process.

Analytics can mitigate issues that result from misaligned incentives owing to principal-agent issues, asymmetric information, and the moral hazard problem. The analytics for this purpose should be valid and unbiased, and use objective and verifiable inputs. An analytic produced by a third party is more likely to fit these purposes than one produced by a financial institution or regulator.

To give one example, after graduate school, one of the authors paid a significant commission to a real estate agent to help him lease a rentstabilized apartment in New York City. The agent used his credit score to verify that he was a person likely to fulfill his financial obligations to potential property owners.

This situation is a very common one; it is instructive to consider the motivations of the parties involved and why the analytical second opinion was sought from a third party. The lessee felt he would be a good tenant but had no way of quickly making his case. The realtor’s position was more tenuous in the sense that he knew little of the potential tenant, but wanted to make the commission and move on to the next deal. The owner of the property, meanwhile, could have made time to interview the potential lessee, check references, and verify income, though this would have provided uncertain signals and would have been relatively expensive and time-consuming to procure.

Though the credit score does not measure tenant soundness per se – it gives no indication of tidiness or proclivity for playing loud music – it is cheap to procure, has no horse in the race, and is sound enough to provide a useful signal to all relevant parties to the transaction. In this case, a third-party model mitigated the principal-agent problem while also helping to overcome significant informational asymmetries the parties to the transaction faced. Financial institutions use models in very similar ways. For example, credit risk buyers will often ask sellers to use a specific vendor model to indicate the likely future performance of the portfolio. In this case, the third-party model partially mitigates the issue of asymmetric information, in that sellers know more than buyers about the underwriting conditions applied in originating the loan. This situation arises with considerable frequency. In the mortgage industry, banks will often corroborate home appraisals using AVMs – auto valuation models – that are owned and operated by third parties. In auto leasing, residual prices will be set using analytical forecasting tools that are not owned by any of the parties to the transaction. Credit ratings from reputable companies will often be required before institutional investors take positions in certain risky assets. In all of these cases, there are sound reasons for why analytics simply must be undertaken by external entities.

But back to stress testing. As we mentioned, regulators want banks to stitch stress testing tools into a business’ day-to-day operations. This can be achieved by either incorporating models that are already in place for day-to-day operations into the Comprehensive Capital Analysis and Review (CCAR) stress testing or using CCAR stress testing models for day-today operations either in addition to or in place of existing practices. In stress testing models, a portfolio’s initial risk level is a key determinant of the expected losses. Using internal models to determine initial risk levels will benefit banks with more aggressive models. Such a policy could lead to a moral hazard problem, because banks with more aggressive models would have an incentive to make more aggressive loans. An industry model that can be applied to all banks can serve as a check on the banks with more aggressive models. This type of industry model could be developed by regulators – provided that they have the required data. Still, even if the regulators’ data is comparable to that of the third party, the third-party model may be more credible if the banks push back on the regulator for being too conservative.

In addition to economies of scale and scope, third-party analytics – because they are produced by third parties – can mitigate incentive issues associated with the principalagent problem, asymmetric information, and moral hazard. In each context, the two parties that are sharing risk can agree to use the thirdparty analytics to make risk more transparent.

We are now seven years gone since the financial crisis triggered the Great Recession. The first stress test (the 2009 Supervisory Capital Assessment Program) did help restore the market's confidence in the US banking system, and there is little doubt that the US banking system is now better capitalized than it was in August of 2008. But it is equally true that the new regulatory environment has yet to be tested by a new banking crisis or, for that matter, a recession of any flavor. Before making a proper assessment of how robust the new system actually is, we would want to see it perform under real stress.

We also would like to see banks use their stress testing infrastructures for their day-to-day business decisions. These infrastructures can be used for tactical decisions – e.g., how to manage a specific exposure – as well as strategic decisions – e.g., whether to expand/contract exposure to an industry, region, or asset class. In either case, the infrastructure would presumably enhance shareholder value.

If banks choose the analytics with the most attractive balance of costs and benefits, they will happily invest shareholder funds in stress testing models, in the knowledge that doing so will increase share prices. If the decision of which analytic to use is constrained, however, a bank is likely to use the analytic only to ensure that it meets regulatory requirements; the bank is unlikely to use the analytic to make business decisions.

For model risk management, the concept of an effective challenge plays a key role. For an effective challenge to be credible, a bank should look to all possible sources of information and knowledge. For a bank to only look internally for answers to these critical questions is simply anathema to the goals of regulatory stress tests.