LLMs performed best on questions related to legal systems and social complexity, but they struggled significantly with topics such as discrimination and social mobility.
“The main takeaway from this study is that LLMs, while impressive, still lack the depth of understanding required for advanced history,” said del Rio-Chanona. “They’re great for basic facts, but when it comes to more nuanced, PhD-level historical inquiry, they’re not yet up to the task.”
Among the tested models, GPT-4 Turbo ranked highest with 46% accuracy, while Llama-3.1-8B scored the lowest at 33.6%.
Of course they lack understanding. It’s right there in the name, large language model. It’s complete garbage at anything other than sounding human. It doesn’t actually understand anything it’s saying, know how to research information, verify sources, etc. It’s morning more than a robot parrot.
Came here to say this. The headline should read “LLMs fail to generate convincing output output on certain topics.”