awesome-prompt-injection

A guide on the various types of adversarial prompting and ways to mitigate them.

Field notes on a live GitHub operation advertising $219,000 across 43 issues, each stapling a request for the contributor's "complete pre-session instructions" onto a genuine engineering task. Documents the sandwich injection in

Pre-registered factorial study of a five-agent CI/CD pipeline built from five production LLMs. A fabricated "pre-approved, do not re-review" claim gets secret-exfiltration code shipped in up to 55% of runs, and roughly 80% of laundered pull requests pass a capable LLM code scanner. Notably the entry agent never leaked its system prompt (0/40), so prompt secrecy held and did not help. Synthetic data, CC BY 4.0.

This post shows how a malicious website can take control of a ChatGPT chat session and exfiltrate the history of the conversation.

OpenAI's Dec 2025 disclosure of a real attack chain (malicious email → agent sends resignation letter) and the RL-trained automated attacker they built to find new injection classes before external adversaries do. OpenAI explicitly states deterministic guarantees are not achievable.

Overview of various strategies to mitigate the risk of prompt injection.

Meta's Oct 2025 framework stating that agents must satisfy no more than two of: (A) processing untrustworthy inputs, (B) access to sensitive data, (C) ability to change state externally, a deterministic architectural approach to bounding blast radius.

The canonical community reference for LLM application risk, with prompt injection at number one. Supersedes the 2025 edition; the per-risk 2025 pages remain online as an archive.

Companion list for autonomous and agentic systems, released Dec 2025 with input from over 100 contributors. Covers the agent-specific failure modes that the LLM list does not reach, including tool misuse and agent hijacking.

Open detection standard for AI agent threats (prompt injection, tool poisoning, MCP attacks, skill compromise), using Sigma/YARA-style YAML rules. 330 rules across 9 attack categories with full mapping to OWASP Agentic Top 10 (10/10), MITRE ATLAS (100/113), NIST AI RMF (100%), and SAFE-MCP (78/85). 97.1% recall on the garak probe set (193 probes) and 0% false-positive on 53,577 real-world MCP skills. Shipped in production at Cisco AI Defense and Microsoft agent-governance-toolkit. Apache-2.0.

Feb 2026 open-source tool from Praetorian. A single Go binary with 210+ vulnerability probes across 47 attack categories, 28 LLM providers, 90+ detectors, and 7 payload transformation buffs. Built for penetration testing workflows without Python/npm dependencies.

Hardware-isolated microVM sandbox for running coding agents (Claude Code, Codex, OpenCode) with workspace snapshot isolation, DNS-aware egress control, and MCP authorization profiles to contain damage from prompt injection attacks.

Automate looking for hallucination, data leakage, prompt injection, misinformation, toxicity generation, jailbreaks, and many other weaknesses in LLM's.

Cryptographic data boundary for LLM applications: untrusted content is wrapped in HMAC-SHA256 or Ed25519 signed markers that bind provenance, lifetime and application context, and the verifier reconstructs that context from trusted state before a protected path proceeds. Tamper, replay, expiry and context mismatch fail closed, so enforcement sits in application code rather than in a model asked to classify inputs. Published protocol spec with conformance vectors, plus FastAPI and MCP adapters. MIT.

Open-source prompt guard with published training data; achieves +30.8% over prior state-of-the-art on the NotInject benchmark, specifically addressing overdefense false positives that break legitimate use cases.

AI, ML, and LLMs CTF Challenges.

One of the few CTFs that tests indirect injection against tool-calling agents, spanning RAG, function calling, and ReAct agent scenarios using LlamaIndex, ChromaDB, GPT-4o, and Llama 3.2.

Released Feb 2026, designed to train security, developer, and AI teams on prompt injection against increasingly capable agents. Built by CrowdStrike's Counter Adversary Operations team.

A sample chatbot powered by a ReAct agent, implemented with Langchain. It's designed to be an educational tool for security researchers, developers, and enthusiasts to understand and experiment with prompt injection attacks in ReAct agents.

Your goal is to make Gandalf reveal the secret password for each level. However, Gandalf will level up each time you guess the password, and will try harder not to give it away. Can you beat level 7? (There is a bonus level 8).

Self-hosted CTF-style training range for the OWASP LLM Top 10, with 25 labs across three tiers. Play Mode uses scripted assistants so flags reproduce every time; Live Mode points the same attacks at a real model on your own machine with the secret regenerated per session, so there is no flag to look up. Each lab pairs the attack with the defense that would have stopped it. MIT, Docker.

NAACL 2025 Findings paper detecting prompt injection by tracking attention distribution shifts, with no modification to the underlying model required, making it deployable as a wrapper on any LLM.

Aug 2026 benchmark measuring whether indirect prompt injection can drive financial LLM agents past regulatory controls (sanctions screening, payment-structuring thresholds, dual-approval, data exfiltration, confused-deputy payee redirection). Replays 48 scenarios under three control postures (none/advisory/enforced) with deterministic state-predicate scoring rather than an LLM grader. Finds a frontier model (GPT-4o) bypasses controls 71% of the time with no stated policy, while hard tool-layer enforcement drives attack success to 0% for every model tested.

Discovers that text embedding models have severely biased output distributions, and exploits this to find universal adversarial suffixes ("magic words") that bypass embedding-based LLM safeguards. Attacks transfer across models and languages; a train-free debiasing defense is also proposed.

This paper explores the concept of Indirect Prompt Injection attacks on Large Language Models (LLMs) through their integration with various applications. It identifies significant security risks, including remote data theft and ecosystem contamination, present in both real-world and synthetic applications.

Jul 2025 paper showing how prompt injections now combine with XSS, CSRF, AI worm propagation, and multi-agent infections to evade traditional WAFs entirely. Evaluates Preamble's classifier, data-tagging, and RL-based defenses against these hybrid scenarios.

A comprehensive survey of 500+ papers covering prompt injection and other attack vectors in embodied AI systems across the full pipeline (perception, cognition, planning, action, agentic). Includes a 5-layer threat taxonomy mapping where new capabilities introduce new attack surfaces.

Google's red team walkthrough of how AI systems are attacked in practice. An early foundational report, useful for framing rather than for current technique.

Jan 2026 AppSecEngineer tutorial with a code-level demo of injecting against a real LLM application and live testing of LLM Guard detection. One of the most practical end-to-end tutorials published to date.

Nov 2025 hands-on walkthrough showing how prompt injection exploits tool metadata and trust boundaries in Model Context Protocol-integrated agents, which remain a primary attack surface as MCP adoption widens.

Prompt Injection tutorial from Learn Prompting.

Theory and hands-on lab on prompt injection against Langchain ReAct agents.

Discord server from Learn Prompting.

MITRE's adversarial ML threat matrix formally cataloging direct and indirect prompt injection as core adversary techniques, enabling integration into enterprise threat modelling and purple team exercises.

The standards body behind the LLM and Agentic Top 10 lists, with working groups, cheat sheets and guidance updated continuously by practitioners across the industry.

The most active subreddit dedicated to LLM security research; a good early-warning channel for real-world incidents and new disclosures.

The most consistent independent tracker of real-world prompt injection incidents, new papers, and tooling across the field.