Featured Product

    BIS Bulletin Examines Cognitive Limits of Large Language Models

    January 25, 2024

    The use cases of generative AI in the banking sector are evolving fast, with many institutions adopting the technology to enhance customer service and operational efficiency. However, this adoption has been accompanied by increasing regulatory scrutiny, as regulators seek to address the complexities and risks associated with generative AI. These models’ potential for biased decision-making is one of the key concerns regulators have and thus is also increasingly an area of research. A recent Bulletin from the Bank of International Settlements (BIS) discusses the cognitive limits of large language models (LLMs) such as Generative Pre-trained Transformer (GPT). The authors, Fernando Perez-Cruz and Hyun Song Shin, explore whether these models truly understand the content they generate or merely replicate the text encountered during training.

    The authors tested GPT-4 with a logic puzzle known as Cheryl’s birthday puzzle. The authors highlight the impressive capabilities of LLMs, including generating computer code, images, and solving complex mathematical problems. The LLM solved the puzzle flawlessly when presented with the original wording but failed when small incidental details, such as the names of the people, the number of dates, or the order of the clues, were changed. This suggests that while LLMs can perform tasks flawlessly when presented with familiar wording, they struggle with tasks that require rigorous reasoning and understanding when the wording is changed. The authors argue that this failure indicates a lack of true understanding of the underlying logic and a reliance on familiar wording. They also highlight the model's lack of self-awareness of its own ignorance.

    The authors note that central banks’ activities are well-suited for the application of machine learning and artificial intelligence, reflecting the ample availability of structured and unstructured data, coupled with the need for sophisticated analyses to support policy. Historically, central banks had been early adopters of machine learning methods in statistics, macroeconomic analysis, and regulation/supervision. The authors also highlight that these findings do not detract from the tangible and rapid progress being made in these areas, as well as in scientific applications of artificial intelligence that have seen rapid progress. However, the authors conclude that while LLMs have made considerable progress in applications such as data management, macro analysis, and regulation/supervision, caution should be exercised when deploying them in contexts that demand rigorous reasoning in economic analysis. While the economic impact of AI could be significant, the current generation of LLMs falls short of achieving artificial general intelligence and cannot substitute for the rigorous reasoning abilities necessary for some core analytical activities. However, the ability of LLMs to engage in rigorous reasoning will determine which tasks and business processes will be impacted by their widespread deployment.

    Visit Moody’s website to find out how Moody’s is incorporating cutting-edge technologies, such as artificial intelligence, to help banks meet their existing challenges more effectively.

     

    Related Link: BIS Bulletin

    Keywords: International, Banking, BIS, RegTech, Generative AI, SupTech

    Related Articles
    News

    BCBS Consults on Guidelines for Counterparty Credit Risk Management

    The Basel Committee on Banking Supervision (BCBS) is seeking comments, until August 28, 2024, on guidelines for counterparty credit risk management of banks.

    May 21, 2024 WebPage Regulatory News
    News

    ISSB Releases Digital Sustainability Disclosures Taxonomy

    The themes of the harmonization and interoperability of sustainability disclosure standards among various jurisdictions remain at the top-of-mind for international standard-setting bodies.

    May 21, 2024 WebPage Regulatory News
    News

    BCBS Report Studies Implications of Digitalization of Finance

    The Basel Committee on Banking Supervision (BCBS) published a report that examines the implications of the digitalization of finance for banks and supervisors.

    May 21, 2024 WebPage Regulatory News
    News

    HKMA Publishes Hong Kong Taxonomy for Sustainable Finance

    The Hong Kong Monetary Authority (HKMA) published the Hong Kong Taxonomy for Sustainable Finance.

    May 21, 2024 WebPage Regulatory News
    News

    EU Taking Steps to Set Out and Enforce AI Regulations

    Many believe that the transformative power of generative artificial intelligence (GenAI) has potential to reshape the financial sector in the time to come.

    May 21, 2024 WebPage Regulatory News
    News

    BIS Paper Outlines Vision for Future Financial System

    In a recent paper, the General Manager of Bank for International Settlements (BIS) and the Indian entrepreneur (Infosys co-founder) Nandan Nilekani have laid out a vision for the Finternet, which is proposed to be a network of multiple financial ecosystems, much like the internet.

    April 29, 2024 WebPage Regulatory News
    News

    NGFS Outlines Options for Supervisory Review of Transition Plans

    The Network for Greening the Financial System (NGFS) recently published three reports on the use of transition plans to boost sustainable finance and manage climate-related financial risks.

    April 29, 2024 WebPage Regulatory News
    News

    BCBS Issues Discussion Paper on Climate Scenario Analysis

    The Basel Committee on Banking Supervision (BCBS) issued a discussion paper on the use of climate scenario analysis to strengthen the management and supervision of climate-related financial risks.

    April 29, 2024 WebPage Regulatory News
    News

    OSFI Issues Phase2 Consultation on Climate Scenario Exercise for Banks

    The Office of the Superintendent of Financial Institutions (OSFI) recently announced a consultation on the second phase of the Standardized Climate Scenario Exercise (SCSE) for banks and other financial institutions it regulates in Canada.

    April 25, 2024 WebPage Regulatory News
    News

    CFIT to Chair Open Finance Taskforce Announced by UK Government

    The UK government announced the formation of an industry-led Open Finance Taskforce, chaired by the Center for Finance, Innovation, and Technology (CFIT).

    April 25, 2024 WebPage Regulatory News
    RESULTS 1 - 10 OF 8967