OSINT Pulse: September 2025 | Navigating AI’s limits while building better OSINT resources

Hyderabad: September has been a busy month in the world of open-source intelligence.

From new research exposing the limits of chatbots in verifying images, to audits showing language models are repeating false claims more often, AI remains at the centre of debates about reliability and trust.

At the same time, fresh resources are emerging to support investigators, including GIJN’s new reporting guide on detecting AI-generated content and the first anniversary of Bellingcat’s Toolkit.

Here is your monthly roundup of key updates, tools and insights shaping the OSINT landscape.

Chatbots offer clues, not certainty, in visual investigations

Chatbots are increasingly being used for casual fact-checking of visuals, with questions such as “hey @Grok, is this image real?” now appearing frequently online.

A recent investigation by Klaudia Jaźwińska and Aisvarya Chandrasekar for the Tow Centre for Digital Journalism tested how large language model chatbots perform when asked to verify images. The findings revealed that even the most advanced models still fail at basic verification tasks.

The researchers asked seven different AI systems to identify the location, date and source of photographs taken by professional photojournalists. Out of 280 questions, the models managed to get only 14 completely right. While the tools can sometimes provide useful starting points for OSINT practitioners, their reasoning was often opaque, misleading or simply inaccurate.

This makes them a risky choice for fact-checking, particularly for those without specialised training to sift useful clues from unreliable information. The study highlights that although chatbots have potential value in open-source investigations, they cannot yet be relied upon for verifying visuals.

From silence to certainty: How LLMs echo false claims

NewsGuard’s latest audit shows that leading large language models are now repeating false information 35 per cent of the time, up from 18 per cent in August 2024. The rise comes as chatbots shifted to using real-time web searches.

Their refusal rates fell from 31 per cent last year to almost none in August 2025, but this change has made them more likely to echo unreliable material.

Instead of pointing to data cut-offs or avoiding sensitive questions, the systems now draw directly from the messy online information ecosystem, giving confident answers even when there are no trustworthy sources.

In August, false claims about Moldova’s parliamentary elections became a recurring problem. Several models were still pulling from the pro-Kremlin Pravda/Portal Kombat network, despite the EU sanctioning its developer, Yevgeny Shevchenko. One chatbot that had previously stopped citing Pravda domains was once again using its content, this time through posts on VK. The episode shows how the network’s material keeps resurfacing in new ways.

After a year of audits, it is clear these are not isolated slip-ups. They point to deeper weaknesses in how AI systems deal with breaking news, assess sources and handle information operations. The models are not only repeating falsehoods more often but are also struggling with gaps in reliable data, fast-changing stories and languages outside English, the very spaces where disinformation is strongest.

New GIJN Reporting Guide on detecting AI-generated content

Journalists are facing increasing hurdles in spotting AI-generated text and visuals. GIJN’s latest reporting guide, written by Henk van Ess, introduces a new detection tool along with seven advanced techniques to help identify content that is likely machine-made.

The guide is designed as a practical resource for reporters and editors who want to strengthen their ability to separate authentic material from synthetic output and to safeguard accuracy and credibility in their work.

Bellingcat Toolkit turns one

Bellingcat’s revamped Toolkit has just marked its first anniversary. The platform serves as a one-stop hub for discovering open-source tools, with use cases, guidance and candid reviews contributed by a global network of volunteers.

The toolkit continues to grow as part of the Global Authentication Project and Bellingcat is inviting experts in specific categories of open-source research tools to contribute. Those interested can reach out to Johanna Wild at toolkit@bellingcat.com

For anyone who missed it, Johanna previously joined Bellingcat’s Discord community to explain the process behind the relaunch and why regular updates are vital. That conversation can still be accessed here.