“We tested 11 of the best AI models available, and none of them passed. None of them came even close to doing a good job on cyber defence,” Ambuj Kumar, founder and CEO of Simbian.ai, told IndianExpress.com. “When it came to attacking, they were better than most humans. But on defence, they really struggled.”
A Stanford University and IIT Kanpur alumnus, Kumar is a tech veteran who spent eight years at Nvidia building the hardware that powers today’s AI. He previously founded a cybersecurity company and has been called a pioneer of confidential computing, a technology that keeps data secure even when everything else is compromised.
How the test worked
Kumar’s team created the Cyber Defense Benchmark which is among the first tests to see how well AI can hunt for cyber threats. They gave the AI models a realistic challenge, such as finding hackers hiding in massive piles of computer security logs.
Think of it like searching for needles in a haystack. Each test included between 75,000 and 135,000 log entries, but only 1-5 per cent were actually malicious. The AI had to figure out which ones without any hints.
The team tested top AI models, including Claude Opus 4.6, GPT-5, and Gemini 3.1 Pro, across 26 different attack scenarios covering 105 hacking techniques.
And when it came to the results, even the best performer, Claude Opus 4.6, covered roughly half of the attack stages but detected only a small fraction (around 4 to 5 per cent) of actual malicious events. The other models did worse. Out of 859 test runs across all models, not one ever found all the threats.
“What it means is that if it was really powerful, it would have found 100 flags, 100 things about an attack,” Kumar explained. “But it only found 44-45 out of 100. So even the best model is failing more than half the time on cybersecurity defence.”
Why defence is so much harder
Previous AI security tests gave the AI specific questions to answer and highlighted which data to examine. Kumar’s test was different, as it dumped raw data and said, “Figure it out yourself.” That’s what real cybersecurity analysts face daily.
The research found three key problems:
The haystack is too big: With over 100,000 log entries but only seeing 10 at a time, the AI couldn’t just scan everything. It needed to ask smart questions.
Seeing but not believing: The AI often spotted suspicious activity but didn’t flag it. Claude Opus 4.6 saw 159 malicious events on average but only reported 113.
Some attacks are nearly invisible: Certain hacking techniques leave very faint traces. All the AI models almost completely missed these.
The attacks are already here.
While AI struggles to defend, hackers are already using AI to attack. “It’s very easy to impersonate people with AI because it can write nice emails,” Kumar said. He highlighted a recent case where scammers used AI to create a fake person on a video call, complete with realistic face and voice. They fooled a company’s chief financial officer into transferring $25 million.
Further, Kumar explained why this is getting worse. “Anthropic is saying that their frontier model is better than most humans at cyberattacks. That is clearly concerning. How are we going to keep ourselves safe if we are bombarded by these automated attacks that are very sophisticated?”
However, according to Kumar, even more worrying is that these powerful AI tools are becoming widely available. When asked about what real-world harm current open-source AI models, especially open-source ones, can actually cause, Kumar explained, “Open-source models are always three to six months behind the best closed-source models.” “What it tells us is that in a year, open-source models will be more powerful than current frontier models.” In other words, the AI tools that only tech companies have today will be in everyone’s hands tomorrow – including criminals.
India’s cybersecurity role
Despite these challenges, Kumar is optimistic about India’s position in the global cybersecurity landscape. “India is kind of the security operations centre of the world,” he said adding that major Indian companies like Tata Consultancy Services, Infosys, Wipro, and HCL are managing cybersecurity for hundreds of companies worldwide.
Kumar was initially worried that AI might disrupt India’s cybersecurity industry, but he found the opposite. “These companies are much more hungry and much faster in evaluating and adopting new technology,” he said. “I feel really good about India’s security operations business capitalising on AI to keep both Indian enterprises and outside enterprises safe and secure.”
What comes next?
For companies experimenting with AI security tools, Kumar’s advice is simple, “Move fast. You don’t have to bet the farm – you can start small, maybe take one application. But speed is paramount in AI.”
Kumar’s research asks a critical question. “If bad people are using AI to attack you, can we use AI to defend?” Right now, the answer is that AI defence isn’t good enough. But, according to him, identifying this gap is the first step. His team has made their research public so other scientists can work on fixing the problem.
The numbers are clear as even the best AI today misses more attacks than it catches. As AI-powered hacking becomes easier and more common, we’re in a race to build better AI defenses. Kumar’s research shows we have a long way to go.
The findings are published in the paper Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps by Ambuj Kumar, Alankrit Chona, and Igor Kozlov (Simbian AI), released in April 2026.
Research shows that even the most advanced AI models can identify only a small fraction of real-world cyber threats. (Image for representation: FreePik) 
Share
