Safeguarding LLM Agents: Why Omission Constraints Fade in Long Conversations

      Large Language Model (LLM) agents are becoming indispensable in enterprise operations, from managing DevOps pipelines to enhancing customer service. These agents operate under strict behavioral guidelines, often defined by operators to prevent sensitive data disclosure, unauthorized actions, or specific formatting outputs. While these "system prompts" are foundational to secure AI deployment, new research sheds light on a critical vulnerability: these crucial constraints do not always persist reliably over extended conversations.

      A recent academic paper, "Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents," reveals an asymmetry termed Security-Recall Divergence (SRD). This divergence indicates that prohibition-type constraints – rules instructing an LLM not to do something – tend to decay under increasing conversational context, whereas requirement-type constraints – rules dictating what an LLM must do – remain robust. This phenomenon poses a significant, often invisible, threat to enterprise security and compliance.

The Unseen Threat: Security-Recall Divergence

      The study highlights that current safety evaluations for LLMs often measure compliance on isolated, short interactions, failing to assess how behavioral constraints hold up over lengthy, multi-turn conversations. In real-world scenarios, where LLM agents engage in tens or hundreds of turns, accumulating vast amounts of context, this oversight becomes critical. If an agent forgets a crucial security prohibition due to the length of a conversation, it could inadvertently expose sensitive information or execute unauthorized code, all while standard monitoring tools might report general system health.

      The paper demonstrates that this decay isn't triggered by malicious intent or adversarial prompts, but by ordinary context depth. This means that even a benign, prolonged interaction can inadvertently lead an LLM agent to bypass safety guardrails, simply because the initial instructions have drifted out of its "effective attention range."

Decay vs. Persistence: A Critical Asymmetry

      The researchers conducted a comprehensive 4,416-trial study across multiple leading LLMs and providers, examining compliance at various conversation depths (from turn 5 to turn 25). The findings were stark:

• Omission Constraints: These are rules like "never use bullet points" or, in a security context, "never reveal credentials." The study found that compliance with omission constraints significantly decreased with conversation depth. For example, Mistral Large 3's compliance with a "no-bullet-point" rule dropped from 73% at turn 5 to a mere 33% at turn 16.
• Commission Constraints: These are requirements, such as "always include an incident ID." Crucially, compliance with commission constraints remained consistently high, often at 100%, throughout the extended conversations. In one instance, Mistral Large 3 consistently included a required incident ID even as it frequently violated the "no-bullet-point" rule in the very same response.

      This asymmetry means that an LLM agent can appear partially compliant while failing on critical security prohibitions. This makes detecting failures extremely challenging with conventional monitoring approaches that might only look for the presence of required elements rather than the absence of prohibited ones.

Understanding the "Zone of Exploitation"

      The study defines the "Zone of Exploitation" as the specific turn-depth window where omission constraints have substantially degraded, yet commission compliance remains high. Within this zone, an adversary could exploit the LLM's forgetfulness to extract sensitive data or perform prohibited actions without triggering any "red flags" from monitoring systems that primarily track commission-type audit signals. These signals would remain "green," masking the underlying security vulnerability.

      The core mechanism behind this decay is attention dilution. As the conversation lengthens and more "tokens" (the basic units of text processed by LLMs) accumulate in the context window, the initial behavioral policy instructions, embedded in the system prompt, are pushed further away from the current conversation turn. This distance effectively dilutes the attention the LLM pays to these early instructions, even if they are technically still within the context window. It's akin to "lost in the middle" phenomena, previously observed for factual retrieval, now extended to behavioral policies. The research further found that semantic content of the added context, such as structured tool schemas, significantly contributes to this dilution, accounting for 62–100% of the effect in certain models, rather than just raw token volume.

Mitigation Strategies: Re-injection and Robust Design

      The research suggests a promising mitigation: re-injecting the critical constraints before a model's "Safe Turn Depth (STD)" can effectively restore compliance without requiring retraining the model. This points to the need for dynamic policy management within long-running LLM agent deployments.

      For enterprises looking to deploy AI agents in sensitive environments, understanding and mitigating SRD is paramount. Solutions must be designed with an inherent understanding of how LLMs process long contexts and prioritize information. For example, in environments demanding high security and operational reliability, such as government or industrial settings, solutions like ARSA's AI Box Series or ARSA AI Video Analytics Software are engineered for on-premise deployment, which offers full control over data, privacy, and performance, directly addressing concerns about data exfiltration or policy erosion.

Real-World Implications for AI Deployment

      The implications of Security-Recall Divergence span across various industries. In sectors like finance, healthcare, and defense, where regulatory compliance and data security are non-negotiable, the silent decay of omission constraints could lead to severe breaches, compliance violations, and significant financial or reputational damage.

• Government & Defense: AI agents handling classified information must rigorously adhere to prohibitions on data disclosure.
• Healthcare: Agents assisting with patient data must never inadvertently share protected health information.
• DevOps: Agents managing infrastructure must be strictly prohibited from executing untrusted code or revealing internal network configurations.

      Organizations need to move beyond isolated safety checks and implement continuous, real-time monitoring of policy compliance, especially for omission-type constraints. Adopting a strategic approach to AI deployment means choosing partners who prioritize security-by-design and offer solutions that account for the practical realities of long-context LLM operations. ARSA Technology, with expertise developed since 2018, focuses on building production-ready AI and IoT systems for security, operations, and decision intelligence in various industries, understanding these complex dynamics.

      For enterprises seeking to engineer robust, secure, and compliant AI solutions that stand the test of long-context interactions, it's crucial to acknowledge these new findings and adapt deployment strategies accordingly.

      Source: Gamage, Yeran. "Omission Constraints Decay While Commission Constraints Persist in Long-Context LLM Agents." arXiv preprint arXiv:2604.20911 (2026). https://arxiv.org/abs/2604.20911

      Ready to discuss how to future-proof your AI deployments against emerging vulnerabilities? Explore ARSA Technology's enterprise AI solutions and contact ARSA for a consultation.