‘Significant inaccuracies’ found in AI-generated news summaries: BBC

The BBC issued a February report that found “significant inaccuracies” with news summaries generated from artificial intelligence (AI) engines including OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini, and Perplexity AI.

“The answers produced by the AI assistants contained significant inaccuracies and distorted content from the BBC,” the outlet wrote in its report.

It found 51 percent of all AI answers to questions about the news were judged to have significant issues, including a failure to differentiate between fact and opinion. 

Nineteen percent of AI answers that cited BBC content introduced factual errors — incorrect factual statements, numbers and dates — while 13 percent of the quotes sourced from BBC articles were either altered from the original source or not present in the article cited.

“This matters because it is essential that audiences can trust the news to be accurate, whether on TV, radio, digital platforms, or via an AI assistant,” the BBC declared in the report. 

“It matters because society functions on a shared understanding of facts, and inaccuracy and distortion can lead to real harm. Inaccuracies from AI assistants can be easily amplified when shared on social networks.”

The outlet found that Perplexity AI altered statements from a source quoted in an article while Copilot used a 2022 article as its sole source for a news summary in addition to other glaring errors. 

Apple heeded the BBC’s warning by temporarily pausing an AI feature that summarizes news notifications, after the outlet alerted the company to serious issues.

The publication is hoping to enact similar change by proposing three next steps to cope with a growing global industry. Those steps include focusing on regular evaluations, holding constructive conversations with AI companies and increasing regulations for large language models.

Some political figures have warned against imposing too many regulations for AI.

Vice President Vance attended the Artificial Intelligence Action Summit in Paris and argued against “excessive regulation.”

“We believe that excessive regulation of the AI sector could kill a transformative industry just as it’s taking off,” Vance said Tuesday in Paris. “And I’d like to see that deregulatory flavor making a lot of the conversations this conference.”

Deborah Turness, CEO of BBC News and Current Affairs, argued government officials, tech CEOs and the media must come together to solve a rapidly evolving problem.

“We’d like other tech companies to hear our concerns, just as Apple did. It’s time for us to work together – the news industry, tech companies – and of course government too has a big role to play here,” Turness wrote in a Tuesday blog post. 

“There is a wider conversation to be had around regulation to ensure that in this new version of our online world, consumers can still find clarity through accurate news and information from sources they know they can trust.”

She said earning the trust of readers is her top priority as CEO.

“And this new phenomenon of distortion – an unwelcome sibling to disinformation – threatens to undermine people’s ability to trust any information whatsoever,” Turness added. 

“So I’ll end with a question: how can we work urgently together to ensure that this nascent technology is designed to help people find trusted information, rather than add to the chaos and confusion? We at the BBC are ready to host the conversation.”

“Microsoft’s approach to generative AI helps to steer the model with proper context from the person’s input and to ground the model in relevant and up-to-date data. For example, we ground Copilot’s model with Bing search data to help deliver more accurate and relevant responses and provide citation links that allow users to look up and verify information,” a Microsoft spokesperson told The Hill. 

“With any AI system we encourage people to verify the accuracy of content, and we remain committed to listening to feedback to improve our AI technologies.”

The Hill reached out to OpenAI, Google and Perplexity AI for comment.

Updated at 3:55 p.m. EST.