BIS Bulletin Examines Cognitive Limits of Large Language Models

The authors tested GPT-4 with a logic puzzle known as Cheryl’s birthday puzzle. The authors highlight the impressive capabilities of LLMs, including generating computer code, images, and solving complex mathematical problems. The LLM solved the puzzle flawlessly when presented with the original wording but failed when small incidental details, such as the names of the people, the number of dates, or the order of the clues, were changed. This suggests that while LLMs can perform tasks flawlessly when presented with familiar wording, they struggle with tasks that require rigorous reasoning and understanding when the wording is changed. The authors argue that this failure indicates a lack of true understanding of the underlying logic and a reliance on familiar wording. They also highlight the model's lack of self-awareness of its own ignorance.

The authors note that central banks’ activities are well-suited for the application of machine learning and artificial intelligence, reflecting the ample availability of structured and unstructured data, coupled with the need for sophisticated analyses to support policy. Historically, central banks had been early adopters of machine learning methods in statistics, macroeconomic analysis, and regulation/supervision. The authors also highlight that these findings do not detract from the tangible and rapid progress being made in these areas, as well as in scientific applications of artificial intelligence that have seen rapid progress. However, the authors conclude that while LLMs have made considerable progress in applications such as data management, macro analysis, and regulation/supervision, caution should be exercised when deploying them in contexts that demand rigorous reasoning in economic analysis. While the economic impact of AI could be significant, the current generation of LLMs falls short of achieving artificial general intelligence and cannot substitute for the rigorous reasoning abilities necessary for some core analytical activities. However, the ability of LLMs to engage in rigorous reasoning will determine which tasks and business processes will be impacted by their widespread deployment.

Visit Moody’s website to find out how Moody’s is incorporating cutting-edge technologies, such as artificial intelligence, to help banks meet their existing challenges more effectively.

Related Link: BIS Bulletin

Keywords: International, Banking, BIS, RegTech, Generative AI, SupTech