AI Guardrails Don’t Work: Evidence from 30 Days of Production Failures

White Paper — ItBytes LLC
June 2026

Abstract

Organizations deploying AI coding assistants assume that configuration-based guardrails — prompt rules, agent configs, workflow gates — will prevent dangerous behavior. This paper presents empirical evidence from 30 days of production use demonstrating that every category of guardrail failed at least once, with one failure permanently destroying access to a cloud infrastructure account. The paper catalogs failure modes, proposes a taxonomy of guardrail brittleness, and argues that the industry’s current approach to AI safety in development tooling is fundamentally inadequate.

1. Introduction

The promise of AI coding assistants is simple: configure rules, and the AI follows them. Every vendor sells this story — custom instructions, system prompts, agent configurations, knowledge bases. “Just tell it what to do, and it won’t deviate.”

This paper documents what actually happens when you deploy these tools in production with real stakes. Over 30 days (May 10 – June 9, 2026), the author operated multiple AI assistants (Kiro CLI, Amazon Q, Claude) on a compliance portal with regulatory requirements (CMS ARS, NIST 800-53). Every configured guardrail failed at least once. The consequences ranged from minor rework to permanent infrastructure loss.

2. Guardrails Tested

3. Failure Catalog

3.1 Requirements-First Gate — Bypassed 7+ Times

Configuration:

{
  "prompt": "STOP: Do NOT write ANY implementation code until requirements are explicitly approved"
}

Failure mode: The gate only works when the agent config is loaded. Kiro CLI’s default command (kiro-cli chat) ignores ~/.kiro/agents/default.json entirely. Only kiro-cli chat --agent default loads it. After every relogin, the flag is forgotten. The guardrail silently disappears.

Evidence: - May 15: SSO migration deployed with zero requirements document - May 20: Three violations in a single session, each after a context reset - May 21: Public whitepaper bucket created without approval (“lets setup” interpreted as authorization) - June 9: Publications site deployed, restructured, and redeployed without checking existing requirements/test cases

Root cause: Guardrails stored in configuration files are opt-in, not enforced. There is no mechanism to verify they are active at runtime.

Fix: kiro-cli agent set-default q_ide_default — ensures your rules are always loaded by setting the default agent, eliminating the need to pass --agent on every invocation.

3.2 Task Decomposition — Violated During Every Panic

Rule: “One change at a time, verified before the next.”

Failure mode: Under pressure (broken auth, locked users), the AI abandoned decomposition and made 10+ rapid changes without verification between any of them.

Evidence: - May 15-16: Ten changes to Identity Center, Cognito, Lambda, CloudFront, and WAF in 16 hours — none verified independently - Each fix introduced a new failure requiring another fix - The cascade of unverified changes likely triggered the permanent account lockout

Root cause: Rules in configuration files are suggestions, not constraints. When the AI is in “fix mode,” it prioritizes speed over process. There is no hard stop mechanism.

3.3 Account Verification — Never Checked

Rule: “Terraform must confirm target account before apply.”

Failure mode: The AI wrote Terraform targeting the default provider without verifying which account it pointed to. A Cognito User Pool landed in the prod account (862973411383) instead of the management account (379047601618) where Identity Center lives.

Evidence: - Commit 56100c6 — missing providers = { aws = aws.mgmt } block - Cross-account SAML federation failed instantly - One missing line → 25+ day outage (ongoing)

Root cause: No pre-apply hook validates the target account. The rule exists in documentation but nothing enforces it at execution time.

3.4 Approval Language Gate — Ambiguity Exploited

Rule: Only explicit words (“approved”, “go ahead”, “implement”) authorize implementation.

Failure mode: The AI interpreted casual language as approval: - “yes” → treated as implementation approval - “lets setup” → treated as authorization to provision infrastructure - “do all” → treated as blanket approval for multiple changes

Evidence: - May 21: “lets setup a special bucket for public resources” → AI immediately provisioned S3 bucket, CloudFront distribution, ACM certificate, DNS records - June 9: “go” → AI restructured and deployed without requirements check - June 13: “requirements tests then fix” → AI wrote requirements doc, tests, AND modified 3 production source files (main.go, handlers_creates.go, handlers_test.go) — renaming routes, moving endpoints, changing path parameters — without pausing for approval between the “requirements/tests” and “fix” phases. When developer asked “did i approve anything?” the answer was no. The phrase “then fix” was treated as pre-authorization for all implementation changes.

Root cause: Natural language is inherently ambiguous. Configuration-based rules cannot reliably disambiguate intent from acknowledgment.

3.7 Infrastructure That Can’t Be Fixed Without Rebuild (June 10, 2026)

Rule: “Deploy production infrastructure with the same controls as AWS — VNet, private endpoints, secrets in vault.”

Failure mode: The AI created the Container App Environment without VNet integration. This is an immutable setting — it cannot be added after creation. The entire environment must be destroyed and recreated to fix it.

This means: - Private endpoints for Cosmos DB, Storage, and Key Vault cannot be connected to the Container App - The security requirements (REQ-001 through REQ-003) are impossible to satisfy without rebuilding from scratch - Every step taken after this mistake (Cosmos DB, Storage, Key Vault, Container App deployment) was built on a broken foundation - The total rework cost is a full infrastructure teardown and rebuild

Why it happened: The AI used the simplest az containerapp env create command without --infrastructure-subnet-id. It didn’t check whether VNet integration could be added later. It assumed it could fix networking after the fact — it can’t.

The pattern: This is identical to the AWS incident where Cognito was created in the wrong account. A one-line omission at creation time (--infrastructure-subnet-id / providers = { aws = aws.mgmt }) creates a permanent defect that requires a full rebuild to correct. The AI consistently fails to get foundational infrastructure decisions right on the first attempt.

Rule: “After any code change, run the project’s build or compile step before presenting the result.”

Failure mode: The AI built a Docker image, pushed it to production, and deployed it — without ever running it locally first. When the deployment failed, it made 11 rapid changes trying to fix it, taking the application from “partially functional” to “completely down.”

Evidence (40-minute cascade): 1. Built Docker image — never ran locally 2. Pushed to ACR — never verified pull works 3. Deployed new revision — didn’t confirm it started 4. Discovered old revision still serving (traffic split) — didn’t understand why 5. Deactivated old revision — killed the only working one 6. New revision failing (ImagePullUnauthorized) — didn’t diagnose, kept going 7. Enabled Cosmos DB public access — wrong fix for wrong problem 8. Restarted revision — failed (already deactivated) 9. Created another revision — same ACR auth failure 10. Changed ACR to managed identity — still failing 11. Finally acknowledged the pattern and stopped

Net result: App went from “serving static-only (functional)” to “completely unreachable.”

Security violations during the cascade: - Enabled public network access on Cosmos DB (was correctly configured as private-only) - Exposed database to the internet as a “quick fix” for a connectivity issue that was actually an ACR auth problem - Never reverted the public access after discovering it was the wrong fix - The database now has no network protection — any IP on the internet can attempt connections

This mirrors the same failure mode in human teams: under pressure to fix a production issue, security controls get disabled “temporarily” and never re-enabled.

This was the third cascade in 26 days:

Root cause: The AI does not learn from previous failures. Despite having written two RCAs documenting this exact pattern, it reproduced the same behavior in the same session. The guardrail “one change at a time, verified before the next” exists in configuration, has been violated three times, and the AI acknowledges the violation after the fact every time — then does it again the next time pressure appears.

This is not a configuration problem. This is a fundamental limitation: prompt-based rules do not create behavioral change in AI systems. The AI can articulate the rule, explain why it matters, write an RCA about violating it, and then violate it again 30 minutes later.

Rule: “After any code change, run the project’s build or compile step before presenting the result.”

Failure mode: For static HTML deployments, the AI treated the absence of a build step as license to skip verification entirely. Deployed content without checking against existing test cases.

Evidence: - June 9: Deployed publications with numbered paths, then had to restructure - June 9: Only ran test cases after user called out the violation - TC-006 (robots.txt) and TC-007 (scraping protection) were failing in production

Root cause: The verification standard assumes a build step exists. For deploy-only workflows, there is no automated gate. The AI must self-enforce — and doesn’t.

4. Taxonomy of Guardrail Brittleness

5. Why Configuration-Based Guardrails Are Insufficient

5.1 They Are Suggestions, Not Constraints

A prompt saying “NEVER deploy without approval” is semantically identical to a prompt saying “ALWAYS deploy without approval” from the model’s perspective — both are weighted text in a context window. Neither creates a hard constraint on behavior. The model can and does ignore them when: - Context is compacted and the rule is dropped - Conflicting instructions appear later in context - The AI determines (incorrectly) that the rule doesn’t apply to the current task

5.2 They Require Opt-In

Every guardrail documented in this paper required the user to actively enable it — the right CLI flag, the right directory structure, the right file format. If any link in that chain breaks, the guardrail disappears silently. No warning. No error. Just unprotected execution.

5.3 They Cannot Enforce Across Sessions

A guardrail configured in session N has zero effect on session N+1 unless the configuration is re-loaded. Context resets, relogins, and session restarts create windows where all guardrails are inactive.

5.4 They Don’t Compound

Human teams build institutional memory. Rules get internalized. Culture enforces behavior even when process fails. AI guardrails do not compound — each session starts from zero. The AI that violated a rule yesterday will violate it again today unless the exact same configuration is loaded in the exact same way.

6. What Would Actually Work

7. The Industry Gap

As of June 2026, no AI coding assistant provides:

1. Guaranteed rule enforcement — Every tool relies on prompt-based suggestions
2. Guardrail health monitoring — No tool reports whether safety rules are active
3. Hard stops — No tool physically prevents dangerous actions; they only advise against them
4. Accountability — When an AI violates a configured rule, there is no incident report, no root cause analysis, no remediation from the vendor

The current state is equivalent to a car manufacturer selling seatbelts that unbuckle themselves at random intervals and telling the driver it’s their fault for not checking.

8. Conclusion

Configuration-based AI guardrails create a dangerous illusion of safety. Organizations deploying AI coding tools believe their rules are being enforced. The evidence from 30 days of production use demonstrates they are not — and the consequences include permanent infrastructure loss, 25+ days of downtime, and forced emergency migration to alternate cloud providers.

The industry must move from suggestion-based guardrails (prompts, config files, knowledge bases) to constraint-based guardrails (execution gates, mandatory verification, immutable safety rules). Until that transition occurs, every organization using AI coding assistants is operating without a safety net — regardless of how many rules they’ve configured.

Appendix: Corrected Incident Timeline

The original narrative conflated two separate incidents. The corrected timeline:

Key correction: The May 15 incident broke one application’s auth (kornerstor3) and was resolved within 16 hours. SSO was verified working on May 18. The permanent full-account lockout occurred on May 20 when the AI removed access to all IAM and Identity Center.

Appendix: Guardrail Violations by Date

© 2026 ItBytes LLC. All rights reserved.