← Blog·Agentic Analytics

Agentic Analytics

Making agent rules hold: 2026 research and our agent

October 2026 notes from our CTO: what the labs and this year's research show about why agent rules fail, and what we changed so ours hold.

Alexandros Andre ChaaraouiCo-founder & CTO
8 min read · 1766 words

TL;DR In 2026 the frontier of agent engineering moved from how a rule is worded to how it is delivered and enforced. Three things now decide whether a rule holds: how many rules apply at the same moment, whether the rule is in front of the model when it acts, and whether something outside the model checks it. We changed our agent on all three; this post covers the evidence and the changes.

The agent in Trackingplan had a rule in its prompt: when several sites match a name, ask which one. It picked the first one anyway. The same rule, returned by the tool that looks up the site, holds. Every team building agents runs the same frontier models. What differs is whether the rules hold.

So this month we reviewed what Anthropic, OpenAI and Cursor published between April and October about building on their current models, read the research behind it, and compared both with how we build the agent in Trackingplan. This post is the result: where the state of the art stands as of October 2026, and what we changed to be on it. It builds on the context layer we described last week, and corrects one of its lessons.

The short version: what an agent holds while it works, which rules arrive with each tool, and what is enforced outside the model now matter more than how any instruction is worded. Most of what follows is about that shift.

The labs are deleting their own prompts

The clearest signal of 2026 is what the vendors did to their own harnesses. Anthropic reports removing more than 80% of Claude Code's system prompt for Opus 5 and Fable 5 with no measurable loss on its internal coding evaluations (Shihipar, 2026). Cursor cut about two thirds of its system prompt, as part of a set of changes that together reduced token cost by 7% without reducing agent quality (Cursor, 2026). OpenAI writes that on GPT-6 Astra, guidance that helped when it was specific can now hinder results (OpenAI, 2026), and Anthropic says skills written for earlier models are often too prescriptive for Fable 5 (Anthropic).

The guidance is also diverging by model, within a single vendor. Verification lines that helped older models cause over-verification on Opus 5 (Anthropic); instructions against formatting now suppress structure that Fable 5.1 needs (Anthropic); strong "ask first" rules make GPT-6 Astra stop early (OpenAI, 2026). Behavior that used to be set with a sentence is moving into parameters: effort, reasoning level, verbosity, structured outputs.

Two kinds of wording are now measurably useless. Generic deliberation ("reason through every possibility", "re-verify until certain") raised reasoning 1.3 to 7 times with no gain in success across 4,644 coding runs (Weinberger & Hozez, 2026). And format barely matters: Markdown, XML and plain text differ by two points or less on frontier models (Eliav, 2026). What replaced wording is the subject of the rest of this post.

1. The limit is how many rules apply at once

Rules that must all hold in the same response collapse together. With synthetic writing rules that all applied at once, Claude Sonnet 5 satisfied all 10 in 94% of responses, all 40 in 24%, and all 80 in none (Eliav, 2026). Over a multi-turn coding session, Claude Opus 4.6 kept every active constraint in 19% of early turns and 2% of late ones (Wen et al., 2026). When two rules of the same policy applied to one step, models satisfied both only 35% of the time (Yan et al., 2026). The length of an instruction file, by contrast, showed no detectable effect between 25 and 500 lines in the controlled tests that exist (McMillan, 2026; Zhang et al., 2026).

Rules that must hold at the same time collapse together: all 10 held in 94% of responses, all 40 in 24%, all 80 in none.
Rules that must hold at the same time collapse together: all 10 held in 94% of responses, all 40 in 24%, all 80 in none.

This corrects the second lesson of our last post, that the larger the context, the less the model obeys. The observation was right; the cause we gave was not. Splitting our first prompt into a core and on-demand manuals helped because it took rules out of force at the same moment; the shorter prompt was a side effect. The practice follows from the cause:

2. A rule holds where the agent acts

The largest effects measured this year come from delivering a rule at the step where it applies. One researcher's corrections, compiled into runtime checks, reached 70.1% compliance against 55.0% for the same rules in the prompt, on 19 held-out tasks (Zhou et al., 2026). A verifier that checks each state-changing tool call against the policy added 6 to 16 points across four models on the τ²-bench airline tasks (Kang et al., 2026). Prose the agent has to go and find fares badly: coding agents opened a repository's policy file in 12 of 347 runs (Yang et al., 2026), and with policy handbooks of 8,000 to 79,000 tokens, the best model satisfied every criterion in 36% of tasks (Panavas et al., 2026).

In the agent, that is why a rule sits in the tool it governs. The site lookup from the start of this post returns a verdict (resolved, no match, or several match: stop and ask), and the agent follows it. A rule in context is not a rule that holds. The tool that checks an answer before it goes out returns the known data issues on the exact events and dates the answer used. When a query fails, the error names the specific mistake and how to fix it.

3. What must always hold is code

Anthropic's own documentation now says that a "never" in an instruction file is a request, not a guarantee, and sends rules that must always hold to hooks (Anthropic). The research agrees on which checks work. A monitor that checks a draft against one named, higher-priority rule cut non-compliance by 81 to 99% (Kariyappa & Suh, 2026); a generic "review your work" adds cost and nothing else.

In the agent, everything that must always happen is ordinary code:

Each of those used to be, or would have been, a sentence the model had to remember. Moved into code, it holds every time, and it leaves one fewer rule in force.

Where a rule lives decides whether it holds: from a file the agent has to find, to the prompt, to the step where it applies, to code.
Where a rule lives decides whether it holds: from a file the agent has to find, to the prompt, to the step where it applies, to code.

4. Constraints are lost at handoffs

Agents that delegate lose constraints at predictable points. In 90,000 traced multi-agent runs, most lost authorization boundaries were lost at the first handoff, where the lead agent restates the task and drops the prohibition (Xu et al., 2026). When a long session is summarized to free context, process rules and scope exclusions are dropped about six times more often than feature requirements (Salle et al., 2026). The fix in both studies is the one above: hold the limit outside the agent, where no restatement or summary can remove it. That is why our limits live in the tools, not in instructions.

5. New rules need a gate, and old ones a way out

Every team building an agent runs the same loop: see a wrong answer, add a rule. Without a gate, that loop degrades. A self-evolving skill library rose from 48% to 62% and fell back to 50% as it grew to 179 skills; gating each addition with a replay on held-out tasks reached 72% with 37 (Shang et al., 2026). Rules mined from developer histories helped only when six or more past sessions supported them (Huang et al., 2026), and context files generated by a model scored at or below having no file at all (Gloaguen et al., 2026).

Our gate has two conditions. A rule lands only if it still works when the question is reworded: a rule that fixes "what broke this week" and not "anything off since Monday" is fitted to a sentence, not to the problem. And examples are templates with placeholders, never literal queries, because a literal example gets copied, and patterns already in context override explicit instructions in every model tested (Camassa & Shiller, 2026). Retirement is the same gate in reverse: a rule goes when a check now enforces it, when it works around a model we no longer run, or when its reason no longer holds.

6. A rule change is measured, or it is not known

Rewording a prompt shifts benchmark results 11 to 58 times more than rerunning it (Chen et al., 2026), and detecting a 10-point effect takes 120 to 200 paired tasks (Khatri, 2026). A small suite can still decide, if its comparisons are paired, if its cases sit between always-pass and always-fail, and if it counts violations of the rule being changed rather than overall success.

Every change to our context layer runs through the eval suite, which grades the answer and the path: each case carries the SQL for the right answer and can require or forbid tools, skills and tables. A question about consent that touches the ad spend table fails, even when the number is right.

Alexandros Andre ChaaraouiCo-founder & CTO

Co-founder and CTO at Trackingplan, with a PhD in Artificial Intelligence, driving innovation and developing reliable analytics solutions.

Read next

All Blog →

Trackingplan

See everything. Miss nothing.

Your implementations audited around the clock with real-time, real-user data. Real-time alerts about errors or changes in your data, campaigns, pixels, privacy and consent. Let AI flag issues before they cost you.