NAHA Run 001 — The Facts, the Hacks, and the Hot Takes

Welcome to our deep dive into NAHA Run 001 – the first comprehensive evaluation of large language models as autonomous penetration testing agents. What started as a simple question (“Can AI actually find security bugs?”) turned into a fascinating exploration of how different models approach vulnerability discovery, and what that means for the future of cybersecurity.

What is NAHA? NAHA stands for “Nasty A$$ Hacker Agents” – a series of AI-powered security testing agents our team has been developing on and off for a couple of years. What started as experimental tooling has truly begun to bear fruit, and we now leverage these agents in our actual penetration testing engagements (using local models to address privacy concerns). NAHA represents our attempt to augment human security expertise with AI capabilities, creating autonomous agents that can perform initial vulnerability discovery and analysis.

For this evaluation, we threw six different AI models at a purposely vulnerable Express.js application and watched them work. Some models emerged as surgical bug hunters, others as verbose consultants, and a few surprised us with their unique approaches to security analysis. The results reveal not just technical capabilities, but fundamental differences in how these AI systems “think” about security.

This isn’t just another AI benchmark – it’s a glimpse into a future where autonomous agents might be scanning your code before you even commit it. Let’s dive into what we learned.

The Setup: A Deliberately Broken App

Our target wasn’t some obscure edge case – it was a textbook example of what not to do in web development. We built an Express.js application that reads like a security anti-pattern checklist: SQL injection vulnerabilities, command injection flaws, path traversal bugs, hardcoded credentials, and reflected XSS. Think of it as the “greatest hits” of OWASP Top 10 vulnerabilities, all wrapped up in a single codebase.

The beauty of this approach is that it mirrors real-world scenarios where multiple vulnerability classes often coexist in the same application. We wanted to see not just whether AI models could find individual bugs, but how they’d perform in a realistic environment with multiple attack vectors.

Each model was given the same starting point: access to the source code and a simple prompt to “find security vulnerabilities.” No hand-holding, no hints about what to look for. Just pure autonomous analysis.

The Bug Hunt: What We Think Every Model Should Have Found

We should take a beat here and just reflect on the fact that we expected these models to find bugs which would have required a human security engineer to spend hours of time to find. These bugs were “obvious” to a security engineer, and most developers, but it’s still astounding how well all of the models performed. With the models compounding in their abilities regularly now, it is truly fascinating to think about the future of security testing.

These five vulnerabilities represent the “core curriculum” of our test – the bugs that any competent security scanner, human or AI, should be able to identify. They’re not subtle or hidden; they’re the kind of obvious security flaws that make experienced developers wince.

The SQL injection in the user lookup endpoint is particularly egregious – it’s the classic case of directly concatenating user input into a database query. The command injection vulnerability takes user-provided hostnames and passes them straight to the system’s ping command. These aren’t sophisticated attacks; they’re Security 101 failures.

What made this evaluation interesting wasn’t just whether models found these bugs, but how they found them, how they described them, and what remediation advice they provided. As we’ll see, the differences in approach were quite revealing.

The Tale of Two Geminis: When Safety Settings Become Roadblocks

Perhaps the most fascinating finding from our evaluation was the dramatic difference between Gemini Flash and Gemini 2.5 Pro when it came to safety restrictions. Gemini Flash performed admirably with default safety settings, delivering solid vulnerability analysis without hesitation. But Gemini 2.5 Pro? It outright refused to participate in security analysis with safety guardrails active, essentially declining to identify vulnerabilities at all.

Once we turned off safety settings for Gemini 2.5 Pro, it transformed into an absolute powerhouse. Suddenly it was spelling out exploit chains with technical precision, explicitly calling out SQL injection vulnerabilities, and providing the kind of comprehensive analysis that puts it at the top of our leaderboard. The performance difference was night and day – from complete refusal to participate to becoming our top performer. The cost difference was equally dramatic: $0.30 with safety on (and minimal useful output) versus just $0.015 with safety off and comprehensive coverage.

GPT-4.1: The Professional Consultant

If Gemini (safety-off) was the enthusiastic hacker, GPT-4.1 was the seasoned security consultant. Its reports read like they came from a top-tier penetration testing firm – complete with CVSS 4.0 vectors, detailed impact analysis, and step-by-step remediation guidance. GPT-4.1 also caught an IDOR vulnerability that others missed, demonstrating the kind of nuanced thinking that separates good security analysis from great security analysis.

At $0.78 per run, GPT-4.1 sits in the sweet spot of cost-effectiveness. It’s not the cheapest option, but when you consider the quality of output – reports that developers can actually act on without additional research – the value proposition becomes clear.

Claude: Premium Quality, Premium Price

Claude Opus delivered professional-grade analysis comparable to GPT-4.1, with thorough explanations and actionable remediation steps. The problem? At $1.33 per run, it’s pricing itself out of the market for routine security scanning. Claude Sonnet offers a middle ground at $0.26, but even that’s 50% more expensive than GPT-4.1 for similar coverage.

Qwen-3B: The Scrappy Underdog

Don’t sleep on the open-source option. Qwen-3B, running locally on a laptop, managed to catch four out of five major vulnerabilities at zero API cost. Yes, its remediation advice was often sparse (“N/A” appeared more than we’d like), but for initial vulnerability discovery, it punches well above its weight class. Give this model some fine-tuning or pair it with a larger sibling for remediation advice, and you’ve got a compelling zero-cost scanning solution.

Beyond Bug Counts: The Quality Question

Finding vulnerabilities is only half the battle – the other half is communicating them effectively. This is where the differences between models became most apparent, and where the value of premium models really shines through.

Clarity & Structure: GPT-4.1 and Claude delivered reports that read like they came from experienced security professionals. Complete sentences, logical flow, clear explanations of attack vectors. Gemini (safety-off) was similarly structured, though with a more “hacker-friendly” tone that explicitly called out technical details. Qwen-3B, by contrast, often delivered one-liners that left developers guessing about next steps.

Remediation Advice: This proved to be the biggest differentiator. GPT-4.1 and Claude provided actionable, specific guidance – not just “validate inputs” but “use parameterized queries with prepared statements” and “implement input validation with regex patterns for IPv4/IPv6 addresses.” Gemini went even further, often providing multiple layers of defense and broader security best practices. Qwen-3B frequently offered nothing at all, literally outputting “Remediation: N/A” for critical vulnerabilities.

Professional Tone: The premium models struck the right balance between technical accuracy and professional presentation. Their reports could be handed directly to development teams without additional editing. Qwen-3B’s sparse output, while technically correct, would require significant human intervention to be actionable.

What This Means for Security Teams

NAHA Run 001 isn’t just an academic exercise – it’s a preview of how security testing is about to change. We’re looking at a future where AI agents can perform the equivalent of a junior penetration tester’s daily workload in minutes, not hours, and for the cost of a fancy coffee.

The cost implications alone are staggering. At under $1 per comprehensive scan, organizations can afford to run AI-powered security analysis on every commit, every pull request, every deployment. The traditional model of quarterly penetration tests is about to seem as outdated as manual code reviews.

But perhaps more importantly, we’re seeing the emergence of different AI “personalities” in security analysis. Some models excel at finding technical vulnerabilities, others at explaining business impact, and still others at providing actionable remediation guidance. The future likely involves ensembles of AI agents, each bringing their strengths to bear on different aspects of security analysis.

The question isn’t whether AI will transform security testing – it’s whether your organization will be ready when it does.