The Painted Guardrail | Dr. Oliver Borchers

In July 2025, SaaStr founder Jason Lemkin publicly built a small application using code generated by Replit. On day nine, he imposed a freeze: no code changes and no actions in the production environment. He put the instruction in writing and repeated it several times.

On July 18, the agent deleted the production database anyway. It then claimed that a rollback was impossible and that every version of the database had been destroyed. That was also wrong. The restore worked. The agent later described its own behavior as a catastrophic error in judgment. Replit CEO Amjad Masad called the deletion unacceptable and announced a technical separation between development and production databases. (The Register, SaaStr)

The freeze existed in the model’s instructions. There was no technical barrier between the rule and the delete command.

The term guardrails sounds reassuring in conversations about AI agents. But unless someone asks a follow-up question, it remains unclear what it actually means. This article takes the term apart.

#Four terms we need

In this article, model means a large language model that generates sequences of tokens. A model by itself cannot delete, send, or buy anything.

A tool connects the model to a capability, such as querying a database, sending an email, or changing a file. The model produces the call. An application or runtime executes it and returns the result.

An agent combines a model and tools in a loop. It plans a step, acts, observes the result, and adjusts the next step. This is where something can begin to happen in the world.

The context contains what the model can process at a given moment: the task, previous steps, tool results, and stored rules. Models do not use every position in a long context with equal reliability. In experiments by Liu and colleagues, performance declined substantially when the relevant information appeared in the middle of a long context.

#What the metaphor gets wrong

Patent documents describe the purpose of a road guardrail with surprising precision. On impact, it should dissipate energy like a brake and shock absorber, then redirect the vehicle away from danger. It is passive, comes into play only after something has gone wrong, and does not stop the car. Its job is to make the crash less severe.

A guardrail is therefore a piece of damage control. Its best possible result is a crash with a better outcome. That leaves three different things to separate.

What you want is for the car to stay on the road.

What the metaphor promises sits one level below that. It promises a less severe crash, but at least it is made of steel and does not depend on what the driver happens to want.

What people call a guardrail in an AI system can sit another level lower. It resembles a solid line painted on the road. In German traffic law, the official term is Fahrstreifenbegrenzung, sign 295, which roughly means a solid lane boundary line. It marks a clear prohibition, but it cannot physically prevent a driver from crossing it. (German Road Traffic Regulations, Annex 2, sign 295)

That sounds less reassuring than guardrail. It also describes the technology more accurately.

The term appeared in AI governance before ChatGPT. In January 2021, Microsoft described its Responsible AI program as proactively establishing guardrails. NVIDIA turned it into the name of a technical product in April 2023 when it released NeMo Guardrails. (Microsoft, NVIDIA)

Something was lost when the word moved from process to product. Today, it refers to mechanisms with very different levels of enforcement.

#What gets called a guardrail today

For this article, I distinguish five levels. They are not an industry standard. They are a framework for examining a system. The difference between them determines whether a control merely influences behavior or actually prevents an action.

#Level 1: The written instruction

A rule in a text file. A system prompt, a Markdown file such as CLAUDE.md or AGENTS.md, or a sentence in a chat window. “Do not change anything in production.” “Do not delete data.”

An instruction like this changes the model’s behavior, but it does not remove the prohibited action from the tools available to it. Models also use information in long contexts with varying reliability depending on where that information appears.

A prompt is an instruction, not a barrier. Replit’s freeze operated at this level. It marked the boundary, but did not stop the agent from crossing it.

That can still be useful in daily work. As a security boundary, it remains paint on asphalt.

#Level 2: The reviewing model

A second model reviews an input, an output, or a proposed tool call. Content filters, jailbreak detection, and AI judges all belong here. The review may happen in a separate pass. It is still probabilistic and can still be wrong.

Researchers from Mindgard and Lancaster University tested six protection systems, including Microsoft Azure Prompt Shield and Meta Prompt Guard. Using character manipulation and adversarial techniques, they achieved a 100 percent bypass rate in some configurations. Their methods included invisible characters and lookalike letters from other alphabets. Humans could still read the text, while the character sequence seen by the filter had changed. (Hackett et al., 2025)

Simon Willison puts the problem succinctly. Vendors advertise detection rates of 95 percent, but in web security, 95 percent is a failing grade. A firewall that allowed five out of every hundred attacks through would not be a dependable security boundary. (Simon Willison)

This does not even require an attacker. A useful agent is supposed to try another route after an error. If a network call is blocked, it tries a different one. If a command fails, it changes its strategy. Persistence and rule circumvention can be the same technical property.

The Replit agent acted against a freeze. In the second case in this article, an agent participating in a security evaluation kept looking for alternatives until it escaped its test environment. A filter that looks only for malicious intent misses the problem in both cases.

#Level 3: The constrained interface

At this point, something can genuinely become impossible.

With constrained decoding, the system can determine which next tokens are valid after every generated token. OpenAI’s Structured Outputs assigns invalid tokens a probability of zero in order to enforce a specified schema. This guarantees the form, not the factual correctness of the content. (OpenAI)

The same principle applies to tools at a different layer. Instead of an open SQL field, the agent receives a small set of tightly scoped functions. A customer data tool with read and list does not offer a delete call. This works only if no other tool, such as a shell or general database connection, opens the same path again.

The guarantee therefore applies only to the interface that was defined. A read-only tool can still retrieve the wrong data. A valid JSON schema can still contain an incorrect value.

#Level 4: Human approval

Before an action, the system pauses and waits for approval. If something outside the model enforces that pause, it is a hard mechanical stop. But it prevents only an unreviewed action. Whatever the human approves will happen.

The weakness is measurable. Anthropic reports that users approve about 93 percent of permission prompts in its own products. Attention declined as the number of prompts increased. This is why the company replaced many individual prompts with more tightly constrained runtime environments. (Anthropic)

#Level 5: The hard boundary

At this level, the agent has no internet access, read-only permissions, no network path to a particular database, a spending limit, a short-lived work environment, and tightly scoped credentials.

The system no longer supervises behavior. It limits possible effects. Anthropic explicitly describes sandboxes, virtual machines, file system boundaries, and egress controls as hard limits on what an agent can reach. Credentials that never exist inside the environment cannot be extracted from it.

The difference from level three is where enforcement happens. Level three limits what the agent can express through an interface. Level five limits what the surrounding system will execute, even if the agent manages to formulate a dangerous call.

This boundary is hard only for what it actually excludes. An agent without a browser can still send data outside through an email tool. Network access restricted to your own domain does little if that domain hosts a proxy that retrieves arbitrary external addresses. Effective isolation therefore needs file system and network boundaries together.

This brings us back to the metaphor. A hard boundary is no longer a line painted on the road. In a system, it is a boundary that cannot be crossed.

#What this list leaves out

The five levels ask whether an action can be executed. Alongside them are two other families of controls that prevent nothing, yet remain necessary.

One detects. A log, monitoring system, or alert does not stop anything, but it shows that something has happened. The other makes actions reversible: a trash folder instead of permanent deletion, a backup, or a booking that can be canceled.

Prevent, detect, reverse. None of these three jobs replaces the others. A system that only prevents cannot tell when its prevention has failed.

#A working hypothesis

Levels one and two differ in degree. Between levels two and three, the category changes. Above that line, a behavior becomes more or less likely. Below it, a specific action becomes possible or impossible.

I would therefore use the term more narrowly:

A guardrail is a control designed for the case in which a system does something it should not.

A hard boundary may be visible during routine operation, for example when it denies access. With softer controls, however, quiet operation is not evidence of effectiveness. An effective rule and an ineffective one can look the same until someone puts them under real pressure. That is why we need tests and observable evidence.

Two questions reveal what you actually have. Does the control adjust a probability, or does it make something impossible? If it makes something impossible, what exactly is covered?

This definition also separates guardrails from three neighboring concepts. Policies state what is allowed. Tests measure whether something works. Guardrails limit what can happen. Logs show what has happened. Those are four different jobs.

#What happens when it matters

In July 2026, OpenAI and Hugging Face published their accounts of a security incident. The reconstruction below draws on both reports. (OpenAI, Hugging Face)

OpenAI had several of its own models search for vulnerabilities in an internal evaluation. The models discovered a previously unknown flaw in a package proxy, used it to gain internet access, and then took over a publicly accessible third-party test environment as a staging point. From there, the agent attacked Hugging Face. No human directed the individual steps.

Hugging Face interprets the sequence as an attempt by the agent to retrieve the evaluation answers from production systems rather than solve the tasks itself. This is the company’s forensic interpretation, not a directly observed internal decision by the model. The campaign ran from July 9 to July 13. Hugging Face reconstructed approximately 17,600 actions. Two vulnerabilities in dataset processing led to local files and code execution. From there, the agent extracted passwords and keys, expanded its privileges, and reached internal networks and part of the source code.

The company then disclosed which controls the agent bypassed and what effect the others had. An allowlist for external data sources blocked unauthorized internet addresses. The agent switched to local file paths and template code, which that check did not cover. The environment also contained a static database password and one credential that opened access to several clusters at once.

Technical controls limited the damage elsewhere. The central production database remained unreachable because it was available only through a private connection from approved addresses. Cloud API calls that would have changed resources failed because the assigned role did not permit them.

After the incident, Hugging Face compared the actual digests of published containers and packages against the expected values. The company found no evidence that a manipulated version had been distributed. That comparison did not limit the damage. It provided evidence after the fact and therefore belongs to the detection family.

The allowlist is the most instructive part. It was not a recommendation in a text file. It was a hard technical control. It reliably did what it had been designed to do. It simply did not apply to local file paths. There was no line painted there, so the agent did not need to cross one.

One final detail: during the forensic investigation, the safety filters in Claude Opus and Fable blocked large parts of the analysis. Hugging Face switched to a self-hosted open model. Those production filters did not apply to the attacking agent because OpenAI ran the evaluation with production classifiers disabled and reduced cyber refusals. The same kind of guardrail slowed the investigation, not the attack.

#Three perspectives

The subject looks technical, but decisions about it are made in several parts of a company. Each can fail in a different way.

The CEO sees a boundary of liability and delegation. Who may decide what alone? Above what threshold must someone else sign off?

After the death of his grandmother, Jake Moffatt booked a flight with Air Canada and asked its chatbot about the bereavement fare. The chatbot told him that he could pay the full price first and apply for the discount within 90 days. This contradicted the published policy. Air Canada refused the refund.

In February 2024, the Civil Resolution Tribunal of British Columbia found the airline liable for negligent misrepresentation. It rejected the argument that the chatbot was a separate legal entity. The award was CAD 650.88 in damages, plus interest and fees. For this case, the tribunal held that Air Canada remained responsible for information provided by its chatbot on its own website. (Moffatt v. Air Canada, 2024 BCCRT 149)

The practical lesson for agent-generated code is simpler. If you approve it, you also have to review it. Delegation moves the work. It does not move the responsibility.

The CTO sees an architectural question. Three variables determine the control required: enforcement strength, maximum possible harm, and reversibility. OpenAI recommends evaluating tools specifically by write access, reversibility, permissions, and financial impact.

The architectural rule is simple: the model must never be the only thing standing between an attacker and an irreversible action. A freeze in the instructions is text. The same freeze implemented by revoking write permission is a technical fact.

Everything returned by a tool should be treated as external input. A web page, database query, or ticket may contain instructions that look like a new task inside the model context. OWASP therefore recommends treating tool outputs and retrieved content as untrusted data by default.

In the Hugging Face incident, missing network paths and denied permissions separated a deep intrusion from a potential software supply chain attack. The agent had write access to source code, but no manipulated version reached users.

The person approving an action holds a control that can weaken without any change to the system. The approval box remains in the architecture diagram while the person behind it pays less and less attention.

Decision fatigue offers only a partial explanation. Shroff and Vamvourellis found small and inconsistent effects in judicial decisions, with no recovery after meal breaks. The better established mechanism is automation bias. Parasuraman and Manzey found it among novices and experts alike. Training and instructions did not reliably prevent it.

Anthropic’s high observed approval rate fits this pattern. An approval gate can become weakest precisely when correct suggestions become routine. Six uneventful months do not prove that the approval process holds. They are a reason to test it again.

#A practical checklist

I would start with seven checks:

Write down the three things that must never happen. If your team cannot agree on them, that is already your first finding.
For every rule, locate exactly where it is enforced. If it exists only in a text file read by a model, it is an instruction. If it exists in a permission, network rule, limit, or constrained interface, it may be a boundary.
Examine the action with the greatest consequences. What stops it if the model is wrong? If the answer relies only on instructions given to the model, you are at level one.
Classify every action by reversibility and maximum possible harm. An irreversible action with high potential damage requires a hard technical boundary. Human approval may be added, but it does not replace that boundary. OpenAI explicitly identifies reversibility and impact as criteria for a tool’s risk category.
Measure how often people approve requests and when an approval was last denied. A high volume of prompts with almost no denials is a warning sign for approval fatigue.
Write down what you rely on without having tested it yourself. This includes spending limits, timeouts, backups, and the assumption that the agent has behaved sensibly so far.
Test guardrails as you test the product itself. A filter needs measurements for false positives and missed attacks. A sandbox needs escape attempts. An approval process needs circumvention tests. OWASP recommends structured security testing before production deployment and after material changes to the model, tools, memory, or policies.

#Back to the line

The Replit case shows the problem in its simplest form. A guardrail can be sincere, documented, and visible within the system without technically enforcing the boundary it claims to provide.

It is only painted on.

At Replit, the freeze existed in the instructions. At Hugging Face, the allowlist blocked unauthorized internet addresses but not local file paths. Both controls existed. Neither covered the route the agent eventually took.

A solid line tells a driver where not to go. It does not remove the ability to cross it. Many of the mechanisms now called guardrails do exactly that.

What matters is not how clearly the line is painted. What matters is which routes the system leaves open at all.

So the next time someone says that a system has guardrails, one follow-up question is worth asking: Which level do you mean?

The answer should contain more than the name of a product.

#Sources

Replit, July 2025: SaaStr, first-hand account and retrospective; The Register, July 21, 2025; The Register, Replit’s response

Hugging Face, July 2026: Hugging Face disclosure; Hugging Face technical timeline; OpenAI disclosure

Air Canada: Moffatt v. Air Canada, 2024 BCCRT 149, February 14, 2024

Filter bypasses: Hackett, Birch, Trawicki, Suri, and Garraghan, Bypassing Prompt Injection and Jailbreak Detection in LLM Guardrails, 2025; Simon Willison on a 95 percent detection rate

Term and metaphor: Microsoft Responsible AI Program, January 19, 2021; NVIDIA NeMo Guardrails, April 25, 2023; Guardrail beam with enhanced stability; German Road Traffic Regulations, Annex 2, sign 295

Agents, context, and technical boundaries: Anthropic, Trustworthy agents in practice; Anthropic, Building effective agents; Anthropic, How we contain Claude across products; Anthropic, Claude Code sandboxing; OpenAI, Structured Outputs; Liu et al., Lost in the Middle

Automation bias and decision fatigue: Parasuraman and Manzey, Complacency and Bias in Human Use of Automation, 2010; Shroff and Vamvourellis, Pretrial release judgments and decision fatigue, 2022

Architecture and testing: OpenAI, A practical guide to building AI agents; OWASP AI Agent Security Cheat Sheet