How to Build Marketing Agents With Evals That Teams Trust
The short answer
- A marketing agent is ready for production when it passes a fixed golden set of real examples, scored the same way every time, before a person approves anything it ships.
- The three parts that make agents trustworthy are a golden set, an automated judge with a written rubric, and a human review gate that cannot be skipped.
- In one production setup, eight agents built this way cut drafting from 45 to 8 minutes and lifted first-pass approval from 55% to 88%.
What is an eval for a marketing agent?
An eval is a repeatable test that tells you whether an agent's output is good enough to send. For a marketing agent the output is copy, a brief, an email or an ad variant, so the test has to check what a marketer would check: is it on brand, is it accurate about the product, does it follow the channel's rules, and would a reviewer approve it without edits.
The mistake most teams make is testing an agent by trying it a few times and deciding it feels right. That works for a demo and fails in month three, when nobody is reading every output any more. An eval replaces the feeling with a number you can track every time a prompt, a model or a skill changes.
A useful way to think about it: the agent is the junior writer, the eval is the style guide made testable, and the reviewer is the editor. Each part on its own is weak. Together they let a team ship more without lowering the bar.
What goes into a golden set?
A golden set is a fixed collection of real inputs with the outputs your best reviewer would accept. Start with 30 to 50 examples taken from work that already shipped: briefs that turned into approved ads, emails that were sent, landing-page sections that passed legal. Include the hard cases on purpose: a product claim that is easy to overstate, a channel with a strict character limit, a tone that is easy to get wrong.
Keep the golden set in the same repository as the agent's prompts and skills, so every change to the agent runs against the same examples. When a reviewer rejects a real output, add that case to the set. Over time the set becomes a record of every mistake the agent is no longer allowed to make.
- Real inputs from shipped work, not invented ones
- The approved output, plus a note on why it was approved
- Edge cases: claims, character limits, tone, compliance wording
- A version history, so a score can be compared with last week's
How does an LLM judge score the output?
An LLM judge is a second model that reads the agent's output next to the golden example and scores it against a written rubric. The rubric matters more than the model. Write it the way you would brief a new editor: brand voice in three rules, the product facts that must be exact, the channel limits, and what counts as a fail.
Research on this method found that a strong judge model can agree with human preferences at over 80%, about the same rate at which people agree with each other, while also showing biases such as preferring longer answers. Treat the judge as a fast first reader, not the final word. It tells you which outputs to look at, and its agreement with your human reviewers is itself something to measure.
Why does a human review gate still matter?
Velocity is the easy part of agents. Quality drift is the hard part, and it arrives quietly: one marketer on Reddit described month three as the point where you stop reading every output, which is exactly when something subtly wrong ships. A human gate is the control that stops that from reaching a customer.
The gate works best when it is narrow and fast. The agent drafts, the judge scores, and a named person approves or edits before anything is published or sent. Every edit the reviewer makes is a signal: if the same correction keeps appearing, it belongs in the rubric or the golden set, not in the reviewer's head.
Which marketing tasks should get an agent first?
Pick tasks that are frequent, rule-bound and quick to judge. Ad variants inside character limits, email subject lines, product-page sections written from a fixed fact sheet and weekly performance summaries all qualify: the rules fit on a page and a reviewer can approve the output in under a minute.
Leave strategy, positioning and anything with legal exposure for later. Those tasks are rare, hard to score and expensive to get wrong, so the eval would be weak and the time saved small. The goal of the first agent is a clear before-and-after number, not the most impressive demo.
- High volume: the task happens many times a week
- Clear rules: brand, facts and limits can be written down
- Fast review: a person can approve in under a minute
- Measurable: time per asset and approval rate can be tracked
What changed when one team ran agents this way?
At Imagera AI, Shalini Verma built eight production marketing agents in Claude Code. Agent Skills hold the brand rules, MCP connects Meta Ads, GA4 and Ahrefs, a golden set scored by an LLM judge checks every change, and a person approves before anything ships. Each agent has an owner, an eval set and a rollback path.
The measured result: drafting time fell from 45 to 8 minutes, first-pass approval rose from 55% to 88% on the golden set, and creative output grew four times. A weekly golden-set review keeps first-pass quality above 85%, so the gains do not erode as prompts and models change.
| Measure | Before | After |
|---|---|---|
| Minutes per draft | 45 | 8 |
| First-pass approval | 55% | 88% |
| Creative output | 1× | 4× |
How do you roll this out without slowing the team?
Start with one agent on one high-volume task, such as ad variants or email subject lines, and build its golden set from the last month of approved work. Run the agent and the judge side by side with your current process for two weeks and compare the approval rate before you let it publish anything.
Give every agent an owner, a version number and a rollback path. When a model or prompt changes, the eval runs first; if the score drops, the old version stays live. Then add the next agent. Teams that hire for this work now ask for it directly: a MarTech review of 60 live marketing job posts that mention AI found that 45 list designing or shipping AI agents and automated workflows as a requirement.
Questions
How many examples should a marketing golden set have?
Start with 30 to 50 real examples from work that already shipped, including the hard cases: exact product claims, strict character limits and tone. Add every output a reviewer rejects. The set grows into a record of mistakes the agent may not repeat, and it is rerun every time a prompt, skill or model changes.
Can an LLM judge replace a human reviewer?
No. A strong judge agrees with people about as often as people agree with each other, but it has known biases, such as favouring longer answers. Use it to score every output quickly and to flag which ones need attention, and keep a named person approving anything that is published or sent.
What metric shows a marketing agent is working?
First-pass approval rate on a fixed golden set is the clearest single measure, because it combines quality and reviewer trust. Track it with time per asset and output volume. If approval falls after a change, roll back to the previous version before the change reaches customers.
Which tools do you need to build marketing agents with evals?
An agent environment such as Claude Code, a way to store prompts, skills and the golden set together (a Git repository works well), a judge prompt with a written rubric, and connections to your data through MCP or APIs. The human review step can live wherever the team already works, such as Slack.
Sources
Try it
