Agent harness for marketing data: how we built ours

TL;DR An agent harness is everything around the model: the data it can reach, what it knows about that data, the tools it can call, the rules that decide when it may call something a conclusion, and the tests that say whether it got better. Ours is built for one kind of data, marketing and analytics tracking, and that shaped every decision. Independent benchmarks show why that matters: even the best model gets four quantitative questions in ten wrong at least once in five tries. The rest of the lesson is subtraction: less context in every prompt, fewer tools, rules placed where the decision happens, and a test suite that can say no to a model upgrade.

In September we launched the new Trackingplan, your digital analytics agent. The launch post describes what it does. This one is about the agent harness underneath, and the harness engineering behind it: why a harness for marketing data looks different from a general one, where we started, what broke, and what we would tell another team building an agent for a narrow, high-stakes job.

What an agent harness is

An agent harness is the software around a language model that turns it into an agent: the data it can reach, the context it reads, the tools it can call, the rules that decide when it may call something a conclusion, and the tests that tell you it got better. In short, agent = model + harness.

We all use the same frontier models, and they reason well enough. Whether an agent gets the answer right depends on what surrounds the model:

• The data it can reach, and how clean and joinable that data is.
• The context: what it knows about that data before the first query, and what it reads only when it needs it.
• The tools it can call, and the limits on each one.
• The rules that decide when a finding becomes a conclusion.
• The tests that tell you a change made it better, not just different.

That is the harness. The model is the reasoning engine; the harness decides what it can see, what it can touch and how it proves things. You can swap the model. You cannot swap the harness without rebuilding the product.

Harness or framework? An agent framework supplies building blocks: a loop, tool calling, memory. A harness is the assembled, running system for one agent, with its data, its context, its tools, its rules and its tests. Two agents on the same model and the same framework can behave very differently, and the difference is the harness. Harness engineering is the work of designing, testing and improving that system, and it is where most of our time went.

The model alone gets the analysis wrong

How often a capable model gets data analysis right is now measured independently. Artificial Analysis runs AA-AnalystAgent: 80 quantitative questions across 14 business and scientific domains, each with its folder of spreadsheets and documents, each run five times. The score counts a question only if the answer is right all five times, because an analyst agent is only useful if its answers hold up without re-checking.

On October 25, 2026, the best model scored 60%. At launch in August it was 54%, and one leading model solved 81% of the tasks at least once but only 41% every time. The most common failure, in 57% of them, was anchoring on a wrong early hypothesis: the model commits to a source or an interpretation early and defends it to the end.

Two things in that result matter for marketing data. The benchmark already gives the model a capable general-purpose harness and curated files, and the best one still misses four questions in ten at least once. And it assumes the files are right. In marketing analytics they often are not: a purchase event that lost its transaction ID on Tuesday, a consent banner that blocks one pixel and not the next, a campaign whose UTMs changed mid-flight. A model will compute a confident conversion rate from that, and nothing in a general harness tells it the number is broken.

That is why we built a harness for this domain and put a data quality layer underneath it.

A harness built only for marketing data

General-purpose agent harnesses are built to work on any data, so they know nothing about any of it. Ours is built for one domain, and four decisions follow from that.

The industry comes preloaded. The agent starts every conversation knowing how the tools of the trade work: how GA4, Adobe Analytics, the Meta pixel and Conversions API, consent management platforms and tag managers send, name and drop data, and what correct tracking looks like for each. Years of digital analytics work are written down as rules, metric definitions and methods, so the agent does not have to rediscover what an experienced analyst already knows.

The semantic layer is there from day one. Every destination names the same order, page and campaign differently. Our semantic layer reconciles GA4, Adobe Analytics, Meta, TikTok and the rest into one vocabulary and one schema of around 30 tables, with metrics defined once as the exact SQL the agent must use. A team can analyse its marketing data the day it connects, without modelling anything first.

Correct data comes first. Before it was an agent, Trackingplan was a data quality platform, and that technology is the foundation. Monitoring keeps watching every event, property, dataLayer push and consent state, and the agent reads what it catches: before an answer goes out, it checks whether a known issue distorts the number. Everything sits in our own analytical database, where hits, sessions, the dataLayer, consent, tag manager releases and ad spend share the same keys. That is what makes correlation cheap, and correlation is what error detection and root cause analysis run on, in real time.

The tools are marketing tools. The agent drives a real browser that captures every network request, console message and JavaScript error, and measures page performance, so it can reproduce a broken funnel on a live site. It reads your tag manager: Google Tag Manager containers are decoded release by release, so it can say which change broke an event. And it reads the ad platforms (Google, Meta, LinkedIn, TikTok, Microsoft and more): spend, campaigns and their UTM tagging, joined with the traffic those campaigns actually drive.

Where we started: a prototype on nao

Before we wrote our own harness, we built a prototype on nao, an open-source analytics agent. Pointing it at our own data showed us what this technology could already do with hits, schemas and alerts, and it ruled out a few ideas we had been sure of before we had written a line of our own.

That prototype was the starting point, not the product. What runs in Trackingplan today is the harness this post describes: the context, the rules, the marketing tools and the tests are ours, and that is where the agent’s answers are won or lost. Our thanks to the nao team for the head start.

Context engineering: one big prompt stops working

The first version put everything in the prompt: every rule, every table manual, every metric definition. It was easy to reason about, and it grew to 58,750 tokens before the first question was asked.

We split it in two. A short spine goes into every prompt: the rules every answer follows, the glossary and a one-line index of every column. Everything else (one manual per table, one metrics file per area, tested example queries) is read on demand, before the agent first queries that table. The spine fell to 29,026 tokens, the suite used 22% fewer tokens at an equivalent pass rate, and answers got slower, because every extra read is one more step.

Then we measured splitting further, and stopped. Each extra read re-sends the whole conversation, so moving a section out of the spine only pays when few questions need it. The rule we kept: move something out only if fewer than about one question in 25 needs it.

Rules in prose get skipped; rules at the decision point don’t

A rule in the system prompt is a suggestion the model weighs against everything else it has read. When a user had several sites with the same name, the agent quietly picked the first one, despite a clear rule saying it must ask.

The fix was not a stronger rule. It was moving the rule to the moment of the decision: the tool that finds a site now returns, alongside the results, a verdict on whether the choice is ambiguous and what to do about it. Since then, when the agent ignores context, we audit the system prompt and the tool descriptions before we add another sentence.

Tool design: every tool is a promise

At one point the agent could create dashboards and monitors. It also started suggesting them, unprompted, in answers that had nothing to do with either. Holding a tool was enough for the model to treat it as part of every answer.

We retired both. A tool earns its place only if the agent should reach for it when the question calls for it, and we read every tool description as part of the prompt, because it is.

Agent memory belongs to the site, not the person

The first memory learned per user. If one analyst told the agent that the Meta pixel was retired, their colleague never benefited. We replaced it with plan context: facts and standing rules about one site or app, shared by everyone who works on it and reviewable in settings. The extractor keeps facts about the plan, never about the person, and a test checks that a personal remark is not stored.

Evidence before conclusions, and no false alarms

A language model handed a symptom will always find a plausible story, and independent benchmarks name the same failure: anchoring on a wrong early hypothesis is the most common way models get analysis wrong. So the default posture is hypothesis-first: a cause becomes a conclusion only when at least two independent pieces of evidence converge on the same mechanism, such as a container change just before the problem started plus traffic or a live check confirming how it broke.

The same discipline runs the other way. The data quality note on an answer appears only when a known issue actually moves the number. The first requirement we wrote for it was "never overfire": a caveat on every number is noise, and readers learn to skip it.

Agent evals: grade the number and the path

The suite started with analyst questions and known answers and grew to 250+ evals. Three things make it worth trusting:

• The answer is checked against the data, not against text. Each case carries the SQL that produces the right number, re-executed when the test runs.
• The path is graded too. Cases assert which tools, skills and tables the agent used, and which it must not use. A right answer reached by guessing still fails.
• Reasoning is judged where it matters. Root-cause conclusions, refusals and prose are graded by a separate judge against written criteria, including questions that have no answer.

We had to test the tests as well: some cases passed silently because of how they were written, and now fail when loaded. Every improvement from user feedback lands with a new case, so the same mistake cannot quietly come back.

Model selection: the suite cannot pick your model

We expected the evals to rank models. They could not: across our three tiers, pass rates were statistically tied. What the suite missed, production showed. A newer default model kept the pass rate level and more than doubled answer times, because it reasoned far longer on the same questions. We rolled it back two days later, and every model change now ships with a full suite run and a comparison of answer times.

To decide which model answers what, a blind pairwise judge did what the suite could not: it preferred the strongest model for root-cause and planning questions and found no difference elsewhere. That became Auto mode, which routes questions about why something happened, what to do or a broad review to the strongest tier, and shows which model answered.

The unglamorous parts

• Python. A simple built-in interpreter failed on more than half of the real calls the agent made. It now runs in an isolated executor with the usual data libraries and no network.
• Long answers. Our first hosting dropped long agent streams, so the agent moved to infrastructure built for long-lived connections.
• Runaway queries. After a timeout, the agent retried the same heavy query again and again. A turn now stops querying after two timeouts and says what it could not compute.
• Untrusted text. Everything a tool returns, from a web page to a query result, is framed as data, never as an instruction, and a separate model reviews any action that would leave the system.

What we would tell another team

• Build the harness for your domain. The loop is a commodity; the data, the semantic layer, the context and the tools are not.
• Prototype on an open agent first, then spend your time on what only you know.
• Keep the always-on context small, and measure every split.
• Put a rule where the decision happens, and give the agent only the tools you want it to use.
• Grade the path, not only the answer, and test your tests.
• Let production latency veto a model upgrade.

Models will keep improving, and we will keep switching when the evidence says so. The agent harness is the part that stays, and for marketing data it is the part that decides whether an AI analytics agent gives you a number you can act on.

Frequently asked questions

What is an agent harness?

An agent harness is the software around a language model that turns it into an agent: the data it can reach, the context it reads, the tools it can call, the rules that decide when it may draw a conclusion, and the tests that show whether it improved. Agent = model + harness.

Is an agent the same as a harness?

No. The model is the reasoning engine and the harness is everything around it. The agent is the two together, which is why two agents on the same model can give very different answers.

What is harness engineering?

Harness engineering is the work of designing, testing and improving the harness: choosing what context the model always sees and what it reads on demand, which tools it can call, which rules gate its conclusions, and which evals prove a change made it better.

Which is the best agent harness?

There is no best harness in general. The right one is built for the data and the questions it has to answer and is measured on them. For marketing analytics, that means the raw tracking data, a semantic layer for the industry’s tools, a data quality layer and evals made of analyst questions.

Why does marketing data need its own agent harness?

Because the data itself is often wrong. Events lose parameters, consent blocks some tags and not others, and campaign tagging drifts. A general harness has no way to know, so the model computes a confident number from broken data. A harness built for marketing data checks the data before it answers.