A primary reason many AI pilot projects fail to reach production is lack of confidence in the security of the systems. One of the most common and pernicious security issues affecting generative AI (GenAI) systems is the prompt injection attack—now listed by OWSAP as the number one security risk for generative AI. These attacks manipulate the prompts and data a GenAI system ingests to alter its behavior in undesirable ways. Because large language models (LLMs) have a conversational interface, there is very little difference between the way it treats administrator guidelines, user instructions, and other input data. In other words, they lack the structural ability to distinguish the context of the text they receive.
This lack of data separation allows users to trick the model into violating internal rules and can lead to several kinds of security problems. For instance, a chatbot can be tricked by conversation into exposing internal, sensitive data (known as data exfiltration attacks), or an AI agent can receive an email that it treats as instructions from its owner to execute an unwanted action.
Prompt injection attacks and related jailbreak attacks are becoming a central obstacle for AI POCs that never move into production at scale. Until we have solved the issue, fewer companies and people will be willing to subject their data and systems to such exposure. As these attacks multiply and become more public, they will also have more consequences for the people operating the systems and the people using them.
Our current methods for mitigating prompt injection attacks are insufficient. This post will introduce three promising approaches coming out of the research community:
- instruction hierarchy
- separation of instructions and data
- damage containment
Why guardrails are not enough
Currently, the most common mitigation against prompt injection attacks are guardrails—semantic input and output filtering usually using another, separate GenAI model to analyze text and detect unwanted patterns. But so far, this approach falls short. Not only does this mean multiple layers of non-determinism—you can never guarantee that something won’t slip through—but if you can trick the underlying model, why can’t you also trick the one above it? There are proven methods for getting around guardrails, including phrasing attacks such as legalese or poems, using invisible Unicode characters, or intentionally setting up a denial-of-service attack through circular logic bombs.
These are only a few of the countless methods bad actors can use to circumvent guardrails, and it becomes even more complex when considering the use of AI agents and public data inputs (e.g., emails, blog posts, GitHub PRs). Multiple levels of protections and rules must be in place between agents and external data stores, agents and internal data stores, agents and LLMs, and between different agents themselves in order to mitigate the risk of attack .
Another problem with using guardrail filters is model performance. The filter needs to quickly assess the threat of the prompt, which means it needs to use a faster–and thus less capable–model to do the assessment. Researchers have proven that guardrails will always be vulnerable because injections can be hidden in puzzles, roleplays, and vectors that the smaller models aren’t powerful enough to decode, but which can be decoded and followed by the underlying model. If the guardrail is as powerful as the backend model, then every prompt is slower and burns more tokens than the original.
These different levels of filtering and guardrails are similar to web application firewalls (WAFs) used to protect normal web applications. While WAFs can be useful to cover holes in insecure applications that can’t be properly hardened, they are not the end goal and should never be confused with applications with proper security, especially against standard injection attacks (e.g., SQL, shell, XML). They are at best a safety net used to detect known attack vectors.
Can models get smarter through instruction hierarchy?
In some ways, prompt injection attacks are similar to social engineering attacks, but instead of trying to trick a coworker or employee, we trick the model instead. One of the ways organizations protect against social engineering is requiring employees to undergo repeated training so they recognize how an attack might work and aren’t naive about the methods. Can we somehow train models the same way?
Red Hat engineer Morgan Foster has written that one of the main causes of prompt injection attacks is that LLMs don’t understand the context of who is talking. Everything is just a stream of text, so the LLM can’t tell the difference between an instruction from its boss, an executive, a fellow employee, or an outsider with malicious intent. What if we could introduce some sort of structure to that text, that is, a hierarchy of which text has priority over how the model behaves?
Instructional Segment Embedding (ISE) is an emerging approach that attempts to teach the model about four different categories of text: system instructions, user instructions, data, and output. The idea itself is flexible enough that it could be expanded to more categories in the future. If the model could understand which category texts fit into, it could more easily recognize how much importance to give it. While this approach does improve things, it seems to be pretty modest, with a 15-18% improvement rate over baseline.
The instruction hierarchy is another approach that retrains models to recognize differences between prompts and instructions. This allows the model to not only prioritize texts, but also detect when a prompt subverts a higher priority instruction. This approach shows even more promise, with a range of 10-63% improvements over baseline depending on the benchmark test. But even the best test only has a success rate of around 95% (with some as low as 72%), and 1 out of 20 failures is too many for any sort of reliability.
Another drawback of this approach is that everything is determined at the time a model is trained, or during pre-training. Not only do the hierarchy levels need to be baked into the models during pre-training, but so does the logic governing how text is prioritized. It doesn’t give the application the flexibility and choice to make that determination, which is where that decision should really live. If we have a complex system of AI agents doing work over multiple streams of data, how much trust to put into each piece of data is a decision that should be made by the architects of that agentic system, not the trainers of the underlying models.
Separation of instructions and data—a lesson from other injection attacks
While prompt injection attacks against LLMs are relatively new, other injection attacks have existed for a long time. For example, let’s look at SQL injection. In the early days of two-tier web applications, a script would take user data and craft a custom SQL query that uses that data to query the database. People quickly figured out that if you supplied data in a certain way, it could take advantage of SQL syntax to run their own queries by “escaping” the original query. The fix was to use a feature of most SQL engines called prepared statements: the query could be hardcoded with placeholders where the data would eventually be put. It didn’t matter how much I tinkered with my input data—it couldn’t taint the query because they were treated as two distinct pieces of text.
Any practical use of LLMs doesn’t want to be limited to hard-coded prompts, but we should re-use the idea of having the application explicitly specify the separation of instruction vs. data.
A new technique, Architectural Separation of Instructions and Data (ASIDE), tries to do just that for language models. It uses a novel technique of applying a fixed, orthogonal rotation to the data tokens during an embedding layer’s forward pass during training. This approach should be mostly model agnostic, but does require an extra fine-tuning step to teach the model about the differences between the two types of rotations (it can be expanded by adding additional rotations). ASIDE is much more robust than the ISE approach above because it doesn’t rely on the model to figure out which part of the input is instruction vs. data, but instead lets the application do the classification.
ASIDE does have the downside of requiring extra fine-tuning (and thus less flexibility in model choices), but the fine-tuning step is much less involved than ISE and does not add extra parameters to the model. ASIDE just needs to teach the model the difference between the embedding rotations rather than a more extensive effort to train the model to learn about various attacks. If this approach becomes more standard, it’s possible that most model vendors will support pre-defined rotations to separate data vs. instructions. This reinforces the idea that the best approach is for the application to choose which part of the input is data vs. instruction, and the model should have a structural way to represent that internally.
While this is a promising technique, it’s not yet perfect. However, ASIDE is not exclusive and can be combined with other techniques like ISE to have possibly better combined results.
Reduce the blast radius: methods of damage containment
Modern software security is about recognizing that nothing is perfect and everything will eventually be compromised. Not only should we be applying defense-in-depth, but we also need to architect our systems such that when something is compromised, we can limit the damage that can be done. What does this mean for LLMs and agentic systems?
Tim Kellogg wrote about the idea of using distinct risk categories, which he calls colors, to label AI agents and tool use. It’s a spin on a classic “pick two” pattern in computer science when we have three competing priorities. In this case the three categories are:
- untrusted inputs
- access to sensitive systems
- the ability to change state (or exfiltrate data)
All of these pose risks, but the biggest problems happen when an LLM or agent has access to all three. If we label each of those with a color, we can then declare that no tool or usage should be allowed to have more than two colors. But we need to be cautious, because systems are dynamic, and input that was once considered trusted has a way of being manipulated. This doesn’t solve the problem of mitigating prompt injection, it just says it’s not as catastrophic if all of the data and tools are labeled correctly and the three-color separation is properly enforced.
Another approach, Type-Directed Privilege Separation, tries to organize the system architecture to limit the spread of prompt injections. The gist of the solution is using multiple layers of agents, some privileged, some unprivileged. The unprivileged ones can interact with potentially dangerous text, converting it into structured data with limited, pre-approved types (no freeform text). This now structured and sanitized data is fed into the privileged agents, which can then take actions based on it.
This technique drastically reduces prompt injection attacks and even eliminates some forms of indirect attacks in benchmarks. However, Type-Directed Privilege Separation has to be built into the system architecture and is thus very specific and bespoke to the application in question. It seems like a very good security design pattern for generative AI systems, but it is not necessarily a solution that can be designed into platforms, making adoption harder. And while the unprivileged agents can’t be fooled into taking direct action themselves, they can be fooled into wasting time and tokens in DOS attacks. Another drawback is that sometimes agents can become less successful in their tasks because context was removed in the process of creating the limited, structured data.
Prioritizing prompt injection mitigation
A foolproof method for protecting against prompt injection attacks doesn’t exist yet. In many ways the current state of GenAI is similar to the early protocols of the World Wide Web: LLMs were not designed with security in mind. While the current state-of-the-art usage of guardrails is inadequate, there are several promising research ideas that could provide meaningful advances in security for both prevention and risk reduction, especially when combined. As an industry, we need to prioritize implementing and testing these ideas, building tools and platforms to make adoption easier. If companies, organizations, and people can’t trust these systems, many GenAI projects won’t make it past the prototype stage.
