Systemic vulnerabilities are an important, if often overlooked, aspect of a financial system’s stress testing regime. This article looks back at the Asian financial crisis of 1997-1998 and applies new methods of measuring systemic risk and pinpointing weaknesses, which can be used by today’s financial institutions and regulators.
Assessing systemic vulnerabilities: East versus West
The ability of a country’s financial system to withstand a severe negative shock has important implications for its general economic and social well-being. Following the global financial crisis and the sovereign debt crisis in Europe, stress testing has become one of the primary techniques for gauging the robustness of individual financial institutions and the financial system as a whole. Although officially part of stress testing mandates, such as the Federal Reserve’s Comprehensive Capital Analysis and Review (CCAR), measuring systemic vulnerabilities has not been emphasized.
In the West, that is. In Southeast Asia, measuring and understanding the potential impact of systemic risk became imperative following the Asian financial crisis of 1997-1998. The crisis began in Thailand in July 1997 and quickly spread to Malaysia, Indonesia, Korea, and the Philippines. Singapore, a regional financial hub with an open economy, was also affected.
The impact of the crisis on these countries was staggering: in one year’s time, a decade of extremely strong economic growth, the “East Asian miracle,” risked being erased. Between June 1997 and March 1998, GDP contracted by nearly 6% in Korea, 9% in Thailand, and 14% in Indonesia. Equity valuations plummeted by 50% or more in most of the affected countries.1
Assessing systemic risk has been a key part of financial supervision in the region ever since. In addition to regulators’ and central banks’ increased focus on systemic risk in the wake of the crisis,2 the International Monetary Fund and the World Bank jointly initiated the Financial Sector Assessment Program (FSAP) in 1999 to assess financial stability and perform stress testing of countries’ financial sectors. These initiatives have been credited with helping Southeast Asia weather the worst of the global financial crisis (GFC) and avoid a repeat of the economic devastation caused by the Asian financial crisis.
“Contagion” is the word typically used to describe how the crisis spread so virulently throughout the region. But contagion is just another way of saying that these countries’ economies were highly interconnected, and thus that systemic risk in Southeast Asia was high. Since the Asian financial crisis, new tools and techniques have been developed to better measure the multiple dimensions of systemic risk.
In this paper, we describe a method for measuring the interconnectedness of financial institutions and apply it to the ASEAN-5 group of countries.3 Our data allow us to go back to before the Asian financial crisis and to compare how the different shocks to the global financial system since then have impacted the systemic risk of financial institutions in the ASEAN-5 countries.
Interconnectedness as a measure of systemic risk
Systemic risk refers to a shock that results in a broad-based failure of the financial system, which in turn threatens to jeopardize the economy. The initial shock(s) can be exogenous (an oil price shock, for example) or endogenous (the bankruptcy of a systemically important firm such as PT Bank Century, for example). Whatever the source of the shock, a high degree of systemic risk implies the potential for a cascade of distress or failure among financial institutions.
The word “potential” is important in this context. A high degree of connectivity among financial institutions is a necessary but not a sufficient condition for a systemic crisis. Indeed, the probability of a systemic crisis is another matter entirely. However, when the number and strength of the connections between financial institutions in an economy is high, contagion risk across firms will also be high. A market shock affecting one firm can quickly spread to others through sharp drops in market valuations and the mark-to-market impact on financial institutions’ balance sheets. A credit event affecting one firm can spill over through on- and off-balance sheet exposures among banks and financial counterparties. When the tinder is piled sufficiently high, a small spark can ignite a conflagration.
The lines connecting Thailand, Singapore, and Malaysia also tend to be blue, meaning that the relationship between financial institutions in these countries is positive: An increase in credit risk among financial institutions in one of these countries has a high propensity to cause an increase in credit risk in the others.
Interconnectedness has traditionally been gauged using various quantitative measures such as the size of non-bank deposits, the size of domestic interbank borrowing, and the importance to the domestic payments system. The Monetary Authority of Singapore, for example, used these measures of systemic risk when it participated in the 2002 FSAP stress tests.4
Although these traditional, descriptive measures of systemic risk are useful and important, size measures do not necessarily uncover the risks resulting from a high degree of interconnectedness. Size is an imperfect measure of systemic risk. It is vital to know which firms occupy important nodes in the financial network – for example, those that may be a near-monopolist in market making for a particular asset class, regardless of size. Hence, measuring too-connected-to-fail is as important as measuring too-big-too-fail.5
Measuring financial firms’ interconnectedness
In this paper, we analyze the interconnectedness and potential for contagion among financial institutions using Moody’s Analytics Expected Default Frequency (EDF™) measures. EDF measures are probabilities of default derived from a contingent claims model of credit risk.6 Our systemic risk framework is built on a network analysis perspective. Using firm-level EDFs and the determinants of those PDs – market leverage and asset volatility – we measure dynamic linkages by estimating Granger causal connections among all pairs of large financial institutions in the ASEAN-5 countries.7
A time-series x is said to Granger-cause a time-series y if past values of x provide statistically significant information about future values of y. Figure 1 illustrates the concept, plotting the time-series of PD for two hypothetical firms, x and y. The default probabilities for these two firms look very similar, but they are out of phase; changes in the PD for firm x precede changes in firm y’s PD. Firms x and y are temporally interconnected – changes in the values of x are closely followed by changes in firm y. In this particular example, knowing past values of firm x’s PD would lead to good predictions for y. The Granger causal link is positive and strong.
Figure 1. Movements in the PD for firm x Granger-cause changes in the PD for firm y
Source: Moody's Analytics
Formally, Granger causality means that the β and/or γ coefficients in the following bivariate vector autoregression (VAR) are statistically significant:
If the relevant F-test of the γ coefficients is significant at the 5% level, x is said to Granger-cause y; whereas if the equivalent F-test of the β coefficients is significant at the 5% level, y Granger-causes x. If both sets of coefficients are significant, there is mutual influence between firms x and y.
Our approach to measuring systemic risk, which is described in more detail in Hughes and Malone, is an extension of similar approaches taken in the systemic risk literature.8 Gray and Malone9 and Gray, Jobst and Malone,10 for instance, adapt and apply contingent claims-based methods to the measurement of systemic risk, although they do not take a network approach to estimating the dynamic linkages across financial institutions. Billio et al. estimate systemic risk measures based on Granger causality networks derived from linkages identified on the basis of bivariate VAR models for pairs of entities in the financial system under consideration.11 Their unit of study was equity returns, however, rather than PD measures derived from a structural credit risk model and historical data on default events such as EDF measures.
The paper most closely related to the approach taken here is Merton et al., which applies the Granger causality network technique of Billio et al. to the expected loss ratio (ELR)12 of firm debt for major sovereigns and global financial institutions.13 Our analysis can be seen as complementing theirs, in the sense that we study network linkages among expected default frequency measures, as well as asset volatilities and leverage ratios, which drive EDFs (and ELRs) in asset value-style credit risk models.
One of the advantages of the network approach to measuring systemic risk is that we can estimate both the direction and strength of the connectedness between financial institutions.14 The direction of connectedness is determined by the statistical significance of the VAR coefficients, as described previously. The strength of the linkages between financial institutions is measured by calculating the degree of Granger causality (DGC) described in Billio et al. DGC is simply the fraction of statistically significant Granger-causality relationships among all N(N-1)/2 pairs of financial institutions at any given point in time. Each unique pair of financial institutions can have zero, one, or two Granger-causal connections, thus implying a maximum of N(N-1) possible connections that can be active in the system. The DGC measure, therefore, lies between zero and one. The higher the ratio, the higher the systemic risk.
The DGC measure captures both upstream and downstream Granger causal linkages in the system. In order to “share down” the DGC to individual institutions, we follow Billio et al. in computing what is known as the “Out measure” for each institution. The Out measure is equal to the percentage of the rest of the other N-1 institutions in the network that are Granger-caused by the institution in question. We can think of the Out measure as capturing an institution’s contribution to systemic risk in the form of dynamic downstream linkages to other institutions.
Size is an imperfect measure of systemic risk. It is vital to know which firms occupy important nodes in the macro-financial network – for example, those that may be a near-monopolist in market making for a particular asset class, regardless of size. Hence, measuring too-connected-to-fail is as important as measuring too-big-too-fail.
There are many advantages to using Expected Default Frequency metrics and their drivers as the basis for calculating the DGC and out measures. EDF measures are forward-looking PDs; in this paper, we use EDFs with a one-year time horizon. That allows us to kill two analytical birds with one stone: we can measure the forward-looking likelihood of the default of a firm (or of a financial system by aggregating EDFs across firms), and calculate the level of systemic risk using the EDF-based DGC measure.
EDF measures have also demonstrated they exhibit superior power, compared with equity returns, for predicting future credit events such as defaults and restructurings.15 We can also calculate DGC measures on market leverage and asset volatility, the two primary drivers of the EDF model, to gain insight into whether credit risk contagion is being driven by volatility or leverage spillovers.
The results of our empirical analysis are based on a dataset of financial institutions (SIC code between 6,000 and 6,799) domiciled in the ASEAN-5 group of countries: Indonesia, Malaysia, the Philippines, Singapore, and Thailand. We limit our dataset to financial institutions with at least US$1 billion in book assets observed at some point over their available histories. The only other selection constraint placed on our data is that financial institutions are required to have traded equity and public financial statements with which to calculate Expected Default Frequency measures over the time interval of our study, which begins in 1995 and runs through October 2014. Our study includes 201 unique financial institutions in the ASEAN-5 countries: 36 in Indonesia, 49 in Malaysia, 30 in the Philippines, 46 in Singapore, and 40 in Thailand.
Figure 2 shows the top 10 financial institutions with the highest Out measures as of October 2014. TMB Bank Public Co. Limited, based in Thailand, exhibits the highest Out measure of the ASEAN-5 firms. The Out measure indicates that the bank’s EDF movements Granger-cause EDF movements in 30.6% of the other financial institutions in the network. Notably, the statistics shown in Figure 2 suggest that systemic risk (measured by Out) bears little correlation with either the probability of default or with firm size on average. The EDF-Out correlation exhibits significant and informative time variation, however, as Figure 5 shows.
The EDF rank column shows the firm’s EDF rank (sorted in increasing order) out of the 122 firms present in the network in October 2014; book-asset rank is measured in descending order. Half of the top 10 firms with the highest systemic risk measures as of October 2014 are based in Thailand and are about average with respect to their EDF levels and book-asset size. Most of the firms in the top 10 list are banks, but the rest are in the broker-dealer, real estate, infrastructure, and insurance sectors.
Figure 2. Top 10 firms ranked by Out measure as of October 2014, with EDF level and firm size
Source: Moody's Analytics
The results shown in Figure 3 bring the impact of the 1997-1998 Asian financial crisis into sharp focus. The graph on the left side shows the weighted average EDF level for the ASEAN-5 countries over time. We weight the historical EDF values using book assets (size) and by Out (systemic influence).16 By either measure, the risk of default reached a historic peak during the Asian financial crisis. It is also notable that the peak in the systemically weighted EDF measure occurs one to two years prior to the peak in the size weighted EDF measure. The average risk of default dropped sharply after 1998, but trended higher during the early 2000s as the dot-com bubble burst, resulting in a recession in the United States, and Argentina defaulted on its foreign debt.
The global financial crisis, as severe as it was in the West, is a relatively minor blip in the time-series for the ASEAN-5 nations. These results suggest a potentially useful and powerful way of monitoring the future likelihood of systemic crises.
Figure 3. Aggregate EDF and DGC measures for ASEAN-5 financial institutions, 1995-3Q2014
Source: Moody's Analytics
The left side of Figure 3 shows the 12-month moving average of the DGC measure for the network at each point in time. The right side of Figure 3 shows the DGC measure for the network at each point in time. In this one graph, we get a panoramic view of how systemic risk has evolved for ASEAN-5 financial institutions over the past 20 years. The strength of interconnectedness among financial institutions and the high risk of contagion that characterized the Asian financial crisis is captured by the peak 0.31 DGC measure.
The graph also shows that it took at least four years for systemic risk to subside to levels that prevailed before the Asian financial crisis. Although economic growth in the countries most affected by the crisis bounced back strongly after 1998, our results on systemic risk corroborate other macro-financial indicators that show that their financial systems and economies took a number of years to fully heal.
The DGC time-series in Figure 3 attests to the fact that the risk of credit risk spillovers arising from the global financial crisis was virtually a non-event for the ASEAN-5 group of financial institutions. Although registering a brief spike, the DGC measure continued to fluctuate around the 0.18 average that prevailed after the Asian financial crisis. In contrast, the DGC measure for large US financial institutions reached a peak of 0.61 at the height of the global financial crisis.17 Intriguingly, systemic risk as measured by the DGC reached its highest level since the Asian financial crisis in July 2013. However, systemic risk has subsided considerably since that date, falling to its lowest level in 20 years.
Figure 4 reinforces our historical understanding of the role of leverage as one of the key causes of the Asian financial crisis. Here, leverage is defined as the ratio of a firm’s default point to its market value of assets.18 As in Figure 3, we calculated two weighted measures of leverage: using book assets (size) and the out measure (systemic influence).
Size-weighted leverage is nearly always higher than systemic influence-weighted leverage, and by a considerable margin in some time periods. The implication is that larger financial institutions lever up more, a finding consistent with data for US financial institutions. A second, and perhaps more important, implication is that a firm’s size is not perfectly correlated with the spillover dimension of systemic risk contribution.
Figure 4. Weighted average leverage and volatitlity, 1995-3Q2014
Source: Moody's Analytics
The graph on the right side of Figure 4 shows size- and systemic influence-weighted average asset volatility over time. Unlike average EDF levels and leverage values, the weighted volatility measures rise throughout but peak well after the Asian financial crisis, around the time of Argentina’s default. Systemic influence-weighted volatility is everywhere above size-weighted volatility: firms that exhibit a relatively high Out ratio, and therefore have a high potential for contagion, also exhibit higher asset volatility.
The global financial crisis exerts a stronger effect on leverage than on volatility for ASEAN-5 financial institutions. Weighted volatility rises and persists through the European sovereign debt crisis, but the magnitude of the increase is relatively small. Weighted leverage spikes to levels that persisted during the early 2000s but then falls sharply to pre-global financial crisis levels.
Tracking cross-sectional correlations over time can yield additional insights into system dynamics. Figure 5 displays two Spearman (rank) correlations: the EDF-Out measure correlation and the leverage-volatility correlation. At each time point, the correlations shown are computed using only the cross section available at that time point for the system. EDF levels and systemic influence correlations tend to be negative during calm periods and positive during crisis periods. Leverage and volatility correlations are always negative.
During times of crisis, the EDF-Out correlations increase, as do the leverage-volatility correlations. The interpretation is that riskier (that is, higher default probability) financial institutions increasingly drive the system during crises and that the negative relationship between leverage and volatility underlying optimal leverage theories in corporate finance (for example, the tradeoff theory of the capital structure) becomes weaker in the cross section of firms during crises. Figure 5 also shows that the EDF-Out correlation tends to spike at the beginnings of crisis episodes, a pattern that is also apparent in data for US financial institutions around 2007-2009. 19
Figure 5. Cross-sectional correlations over time: EDF-out and leverage-volatility
Source: Moody's Analytics
Thailand was the epicenter of the Asian financial crisis in 1997. The devaluation of the baht set off a cascade of financial distress throughout the ASEAN countries. Our study of Granger causal connections among EDF measures reveals that financial institutions in Thailand still represent a concentration of systemic risk in the ASEAN-5 network. Financial institutions in Singapore and Malaysia also stand out as having a high concentration of positive (that is, forcing) Granger causal relationships.
Figure 6 shows the complete network map of Granger causal connections as of October 2014. Circles represent financial institutions, and are color coded by country of domicile. This graph displays linkages based on the coefficients at lag 1 in the VAR models using EDF measures (equation 1 above). Red lines correspond to negative coefficients (damping effects), and blue lines correspond to positive coefficients (forcing effects).
Figure 6. Network map based on Granger causal connections as of October 2014
Source: Moody's Analytics
The sets of lines connecting financial institutions in Thailand (green), Singapore (yellow) and Malaysia (red) are numerous, giving the graph a very dense appearance on the right side. The lines connecting Thailand, Singapore, and Malaysia also tend to be blue, meaning that the relationship between financial institutions in these countries is positive: An increase in credit risk among financial institutions in one of these countries has a high propensity to cause an increase in credit risk in the others.
The experience of the Asian financial crisis sparked an intense interest in measuring systemic risk among regulators in Southeast Asia, with the aim of developing policy tools to mitigate its reoccurrence. That interest has been further reinforced by the global financial crisis, which, while not having a serious effect on Southeast Asia, served as a salutary reminder that a systemic crisis can arise in one part of the world and spread to others.
Going forward, we expect that regulators will require financial institutions subject to supervisory stress tests to pay greater explicit attention to systemic risks. In order to do so, they must be able to quantify systemic risk in real time. Our empirical results showed that Granger causal measures captured the contagion risk of the Asian financial crisis to the greater region extremely well. Financial institutions in the ASEAN-5 nations experienced historic levels of interconnectedness and default risk during the Asian financial crisis. They were, however, relatively immune to the effects of the global financial crisis.
The tools we have explored in this research note can be of indispensable use to both financial institutions and regulators for estimating the current and future level of systemic risk and for identifying the sources of its changes. Regulators can, at a glance, obtain tangible signals indicating which institutions are most strongly connected in the network and thus pinpoint weaknesses in the broader financial system.
Managers of financial institutions, meanwhile, can assess their counterparty risks more fully via consideration of joint and conditional default likelihoods calculated using network-based simulations. In an environment where financial institutions are less likely to be bailed out, managers must take steps to guard against failures caused merely by being in the wrong place at the wrong time.
We described a useful measure of interconnectedness using a network approach whose implementation is straightforward and whose outputs are intuitive and easily interpretable. Granger causality networks address one of the key aspects of systemic risk: the extent of dynamic spillovers between institutions. By using expected default frequency measures as the fundamental unit of observation, we are able to relate statements about the likelihood of default for particular institutions to measures of systemic risk exposure and contribution.
1 Andrew Berg, International Monetary Fund Working Paper, WP/99/138, The Asian Crisis: Causes, Policy Responses, Outcomes, 1999.
2 Masahiro Kawai and Peter J. Morgan, Asian Development Bank Institute working paper, No. 377, Central Banking for Financial Stability in Asia, August 2012.
3 The Association of Southeast Asian Nations is an association for regional social and economic cooperation consisting of ten Southeast Asian countries. ASEAN was formed in 1967 by Indonesia, Malaysia, the Philippines, Singapore, and Thailand (the ASEAN-5 countries). Five additional countries joined later.
4 Chan Lily and Lim Phang Hong, The Monetary Authority of Singapore, Staff Paper No. 34, FSAP Stress Testing: Singapore’s Experience, August 2004.
5 This fact has been recognized by the central banks in the ASEAN-5 nations. Much interesting research on systemic risk from a too-connected-to-fail perspective has been generated by the central banks of Indonesia, Malaysia and Thailand, including: Ayomi and Hermanto (2013), Bank Negara (2013), Hwa (2013), Nacaskul (2010). Sheng (2010) also studies systemic risk using a network approach.
» Sri Ayomi and Bambang Hermanto, Bank of Indonesia Bulletin of Monetary, Economics and Banking, Systemic Risk and Financial Linkages Measurement in the Indonesian Banking System, October 2013.
» Bank Negara Malaysia Financial Stability and Payment Systems Report (2013): 46-51, Risk Developments and Assessment of Financial Stability in 2013, External Connectivity and Risk of Contagion to the Malaysian Banking System, 2013.
» Tng Boon Hwa, Bank Negara Malaysia working papers, WP1, External Risks and Macro-Financial Linkages in the ASEAN-5 Economies, 2013.
» Poomjai Nacaskul, Bank of Thailand working paper, Toward a Framework for Macroprudential Regulation and Supervision of Systemically Important Financial Institutions, December 24, 2010.
» Andrew Sheng, World Bank working paper, No. 67, Financial Crisis and Global Governance: A Network Analysis, 2010.
6 Zhao Sun, David Munves, and David T. Hamilton, Moody’s Analytics Model Methodology, Public Firm Expected Default Frequency (EDF™) Credit Measures: Methodology, Performance, and Model Extensions, June 2012.
7 Clive C. W. Granger, Econometrica Vol. 37, No. 3, 424-438, Investigating causal relations by econometric models and cross-spectral methods, July 1967.
8 Tony Hughes and Samuel W. Malone, Moody’s Analytics white paper, CCA Financial Networks and Systemic Risk: Concepts and Outputs, October 2014.
9 Dale Gray and Samuel W. Malone, Chichester, England: John Wiley & Sons, Macrofinancial Risk Analysis, 2008. Dale Gray and Samuel W. Malone, Annual Review of Financial Economics Vol. 4, No. 1: 297-312, Sovereign and Financial Sector Risk: Measurement and Interactions, 2012.
10 Dale Gray, Andreas Jobst, and Samuel W. Malone, Journal of Investment Management Vol. 8, No. 2: 90-110, Quantifying Systemic Risk and Re-conceptualizing the Role of Finance for Economic Growth, 2010.
11 Monica Billio, M. Getmansky, A. Lo and L. Pelizzon, Journal of Financial Economics Vol. 104, No. 3: 535-559, Econometric Measures of Connectedness and Systemic Risk in the Finance and Insurance Sectors, June 2012.
12 ELR is defined as the expected loss, or implicit put option, component of the debt divided by its promised (or risk-free) value.
13 Robert C. Merton, Monica Billio, Mila Getmansky, Dale Gray, Andrew W. Lo, and Loriana Pelizzon, Financial Analysts Journal 69 (2): 22-33, On a New Approach for Analyzing and Managing Macrofinancial Risks, 2103.
14 Although we do not discuss it in this paper, Granger-causality networks also allow us to identify whether the relationship between financial institutions is positive (“forcing”) or negative (“dampening”), depending on the signs of the β and γ coefficients in equations (1). Hughes and Malone (2015) estimate the forcing and damping effects for U.S. financial institutions.
15 Zhao Sun, Moody’s Analytics ViewPoints paper, An Empirical Examination of the Power of Equity Returns vs. EDFs for Corporate Default Prediction, January 2010.
16 To be precise, the systemic influence weights are an equally weighted average of weights based on Out, Out.plus, and Inverse Closeness; the latter two measures are described in Hughes and Malone (2015).
17 Hughes and Malone, 2014
18 In the expected default frequency model, the default point is defined as the notional value of liabilities that would trigger a credit event. For corporates, it is calculated as short-term debt plus half of long-term debt. For financial institutions, it is calculated as 75% of reported total liabilities.
19 Tony Hughes and Samuel W. Malone, Moody’s Analytics ViewPoints paper, Systemic Risk Monitor 1.0: A Network Approach, forthcoming, 2015.
A well-recognized researcher in the field; offers many years of experience in the real estate ﬁnance industry, and leads research efforts in expanding credit risk analytics to commercial real estate.
Focuses on helping financial institutions improve their data management practices and capabilities for enhanced risk management, business value, and regulatory compliance.
A new study from Moody's Analytics uses a quantitative Expected Default Frequency (EDF) model to assess the impact of the pandemic on corporate credit risk in Southeast Asia.
En este seminario web, utilizaremos las métricas EDF de Moody’s Analytics para evaluar el impacto que COVID-19 ha tenido hasta ahora en el riesgo de crédito.
As COVID-19 spreads globally, fear and uncertainty are rising, roiling financial markets and pushing the global economy towards recession. This report uses Moody’s Analytics CreditEdge™ public-firm EDF™ (Expected Default Frequency) metrics to assess the impact that the coronavirus has had so far on credit risk.
In this webinar, we will use Moody’s Analytics EDF metrics to assess the impact COVID-19 has had so far on corporate credit risk.
Highly Commended for Best Fixed Income Paper in the 2019 Savvy Investor Awards, this paper tests the early warning power of the CreditEdge Deterioration Probability (DP) metric for Fallen Angel downgrades.
Moody's Analytics Webinar: Environmental, Social & Governance and Credit Risk Ideas for Fixed Income Investors
Fixed income and equity investors face growing pressure to measure and monitor the ESG risk of their portfolios.
We look at climate risk and consider how a heating planet might impact a bank's performance
Expanding Roles of Artificial Intelligence and Machine Learning in Lending and Credit Risk Management
With ever-expanding and improving AI and Machine Learning available, we explore how a lending officer can make good decisions faster and cheaper through AI. Will AI/ML refine existing processes? Or lead to completely new approaches? Or Both? What is the promise? And what is the risk?
When banks manage risk, conservatism is a virtue. We, as citizens, want banks to hold slightly more capital than strictly necessary and to make, at the margin, more provisions for potential loan losses. Moreover, we want them to be generally cautious in their underwriting. But what is the best way to arrive at these conservative calculations?
The traditional build-and-validate modeling approach is expensive and taxing. A more positive and productive validation experience entails competing models developed by independent teams.