An evidence review of AI-assisted development

Working code is not production-ready.

AI has made software generation abundant. But security, comprehension, and verification have not kept pace. This is the measurable risk behind vibe coding.

Read the story Explore the evidence

Vibe-coded repositories with a confirmed vulnerability 90% Deng, Fan & Meng, 2026 preprint

Syntax pass rate in frontier models 95%+ Veracode, Spring 2026

Security pass rate remains 45-55% Across 150+ models, 2023-2026

The productivity question

AI really does change productivity.

The evidence is not "AI works" versus "AI does not work." Results change dramatically with the task, the developer, and the maturity of the codebase, as shown by Peng et al., Cui et al., METR, and DORA.

Greenfield task 55.8%

less time for 95 developers building a bounded JavaScript service.

Enterprise field trials +26.1%

completed tasks across 4,867 developers at three companies.

Mature repositories 19%

slower for experienced maintainers in familiar open-source codebases, measured in early 2025. The same lab's 2026 update points toward a speedup.

Workplace adoption 90%

of DORA respondents reported using AI at work.

The central finding

Models learned to make code run. They did not learn to make it safe.

Correctness improved. Security stayed flat.

Approximate Veracode trend across 150+ models, 2023-2026

Syntax 2023 ~50%

Syntax 2026 >95%

Security 2023 ~50%

Security 2026 ~55%

Evidence from deployed applications

The first large study of vibe-coded apps is stark.

Researchers identified 10,517 confirmed vibe-coded applications and deeply audited 200 deployed web apps with multiple agents and independent human experts.

Confirmed findings 1,471

exploitable vulnerabilities across the audited sample.

Affected repositories 90%

contained at least one confirmed vulnerability.

Severity 76.7%

of findings were rated Critical or High.

Median development 9.8d

from first to last recorded development activity.

The dominant vulnerability classes

Three familiar failures dominate.

Nearly three quarters of confirmed vulnerabilities fall into categories mature engineering workflows are specifically designed to catch.

Broken access control

36.0%

Authorization enforced in the interface but missing from APIs or the data layer.

Cryptographic failures

20.7%

Exposed secrets, weak storage choices, and security features that silently fail open.

Injection

17.7%

Untrusted input crosses multiple lines or files without consistent validation.

Why these failures recur

The failure is in the workflow.

AI optimizes for fast, local, visible success. Security, maintainability, and architectural fit are slower, global, and often invisible until deployment.

Knowledge defects

49.9%

Hidden security rules
Hallucinated behavior
Insecure agent instructions

Objective defects

30.6%

Demo-oriented design
Controls weakened to clear errors
Functionality prioritized over safety

Memory defects

19.5%

Incomplete change propagation
Forgotten obligations
Security delegated to the user

Follow the evidence.

Separate the measured findings from the anecdotes, understand the incidents, and focus controls on the failures that actually recur.

Research

AI made code generation abundant. The scarce resource is trustworthy verification.

The answer is not to reject AI-assisted development. It is to stop treating generated functionality as evidence of production readiness.