Featured Product

    BIS Bulletin Examines Cognitive Limits of Large Language Models

    January 25, 2024

    The use cases of generative AI in the banking sector are evolving fast, with many institutions adopting the technology to enhance customer service and operational efficiency. However, this adoption has been accompanied by increasing regulatory scrutiny, as regulators seek to address the complexities and risks associated with generative AI. These models’ potential for biased decision-making is one of the key concerns regulators have and thus is also increasingly an area of research. A recent Bulletin from the Bank of International Settlements (BIS) discusses the cognitive limits of large language models (LLMs) such as Generative Pre-trained Transformer (GPT). The authors, Fernando Perez-Cruz and Hyun Song Shin, explore whether these models truly understand the content they generate or merely replicate the text encountered during training.

    The authors tested GPT-4 with a logic puzzle known as Cheryl’s birthday puzzle. The authors highlight the impressive capabilities of LLMs, including generating computer code, images, and solving complex mathematical problems. The LLM solved the puzzle flawlessly when presented with the original wording but failed when small incidental details, such as the names of the people, the number of dates, or the order of the clues, were changed. This suggests that while LLMs can perform tasks flawlessly when presented with familiar wording, they struggle with tasks that require rigorous reasoning and understanding when the wording is changed. The authors argue that this failure indicates a lack of true understanding of the underlying logic and a reliance on familiar wording. They also highlight the model's lack of self-awareness of its own ignorance.

    The authors note that central banks’ activities are well-suited for the application of machine learning and artificial intelligence, reflecting the ample availability of structured and unstructured data, coupled with the need for sophisticated analyses to support policy. Historically, central banks had been early adopters of machine learning methods in statistics, macroeconomic analysis, and regulation/supervision. The authors also highlight that these findings do not detract from the tangible and rapid progress being made in these areas, as well as in scientific applications of artificial intelligence that have seen rapid progress. However, the authors conclude that while LLMs have made considerable progress in applications such as data management, macro analysis, and regulation/supervision, caution should be exercised when deploying them in contexts that demand rigorous reasoning in economic analysis. While the economic impact of AI could be significant, the current generation of LLMs falls short of achieving artificial general intelligence and cannot substitute for the rigorous reasoning abilities necessary for some core analytical activities. However, the ability of LLMs to engage in rigorous reasoning will determine which tasks and business processes will be impacted by their widespread deployment.

    Visit Moody’s website to find out how Moody’s is incorporating cutting-edge technologies, such as artificial intelligence, to help banks meet their existing challenges more effectively.

     

    Related Link: BIS Bulletin

    Keywords: International, Banking, BIS, RegTech, Generative AI, SupTech

    Related Articles
    News

    EFRAG Proposes XBRL Taxonomy and Standard for Listed SMEs Under ESRS

    The European Financial Reporting Advisory Group (EFRAG), which plays a crucial role in shaping corporate reporting standards in European Union (EU), is seeking comments, until May 21, 2024, on the Exposure Draft ESRS for listed SMEs.

    February 23, 2024 WebPage Regulatory News
    News

    ECB to Expand Climate Change Work in 2024-2025

    Banking regulators worldwide are increasingly focusing on addressing, monitoring, and supervising the institutions' exposure to climate and environmental risks.

    February 23, 2024 WebPage Regulatory News
    News

    ECB is Conducting First Cyber Risk Stress Test for Banks

    As part of the increasing regulatory focus on operational resilience, cyber risk stress testing is also becoming a crucial aspect of ensuring bank resilience in the face of cyber threats.

    January 24, 2024 WebPage Regulatory News
    News

    EBA Continues Momentum Toward Strengthening Prudential Rules for Banks

    A few years down the road from the last global financial crisis, regulators are still issuing rules and monitoring banks to ensure that they comply with the regulations.

    January 24, 2024 WebPage Regulatory News
    News

    EU and UK Agencies Issue Updates on Final Basel III Rules

    The European Commission (EC) recently issued an update informing that the European Council and the Parliament have endorsed the Banking Package implementing the final elements of Basel III standards

    December 19, 2023 WebPage Regulatory News
    News

    Industry Agency Expects Considerable Uptake for Swiss Climate Scores

    The Swiss Federal Council recently decided to further develop the Swiss Climate Scores, which it had first launched in June 2022.

    December 18, 2023 WebPage Regulatory News
    News

    BCBS Consults on Disclosure of Climate Risks, Issues Other Updates

    The Basel Committee on Banking Supervision (BCBS) launched consultation on a Pillar 3 disclosure framework for climate-related financial risks, with the comment period ending on February 29, 2024.

    December 18, 2023 WebPage Regulatory News
    News

    US Government Moves to Regulate Development and Use of AI Models

    The U.S. President Joe Biden signed an Executive Order, dated October 30, 2023, to ensure safe, secure, and trustworthy development and use of artificial intelligence (AI).

    December 18, 2023 WebPage Regulatory News
    News

    MAS Launches Gprnt Digital Platform for ESG Reporting for SMEs

    The Monetary Authority of Singapore (MAS) launched an integrated digital platform, Gprnt, also known as “Greenprint.”

    November 29, 2023 WebPage Regulatory News
    News

    EBA Finalizes Templates for One-Off Climate Risk Scenario Analysis

    The European Banking Authority (EBA) has published the final templates, and the associated guidance, for collecting climate-related data for the one-off Fit-for-55 climate risk scenario analysis.

    November 28, 2023 WebPage Regulatory News
    RESULTS 1 - 10 OF 8949