LLM-as-Judge for Marketing Copy: Rubric and Golden Set

What is an LLM judge, and why use one for copy?

An LLM judge is a model asked to grade another model's output. For marketing copy, it reads an ad, email or page section next to your rubric and an approved example, then returns a score and a reason.

The point is scale and consistency. A person can review twenty variants carefully; a judge can score two hundred the same way every time, and flag the handful a person should look at. That turns copy quality into a metric you can watch as prompts and models change.

A judge is especially useful for the checks people find tedious: counting characters, spotting a banned claim, confirming a required element is present. Those are exactly the checks that slip when a reviewer is tired, and exactly the ones a model does reliably.

How reliable is an LLM judge?

The most cited study, from the team behind MT-Bench and Chatbot Arena, found that strong judges such as GPT-4 matched human preferences at over 80% agreement, the same level at which people agree with each other. The same paper documents the biases to plan for: favouring the first answer shown, favouring longer answers, and favouring output from the judge's own model family.

So the judge is a fast, cheap first reader, not a final authority. Measure how often it agrees with your own reviewers on a sample each month, and treat a drop in agreement as a reason to fix the rubric.

What should the rubric contain?

Write the rubric the way you would brief a new editor, and keep it to things a reviewer can check without guessing. Score each criterion separately, pass or fail, before an overall grade; separate scores tell you what to fix.

Keep the rubric in the same repository as the prompts and the golden set, and change it through review like any other part of the system. A rubric edit changes what counts as good, so it deserves the same care as a prompt edit.

• Brand voice: three rules, each with one example of right and wrong
• Accuracy: the product facts and claims that must be exact, and the ones never allowed
• Channel rules: character limits, required elements, banned phrases
• Clarity: one idea per line, the benefit stated before the feature
• Fail conditions: anything that makes the copy unusable regardless of the rest

What does a judge prompt look like?

Keep the judge prompt short and structured. Give it the rubric, the approved example for the same input, and the candidate copy. Ask it to score each criterion pass or fail with one sentence of reason, then give an overall verdict, and to output in a fixed format your workflow can read.

Two details make a large difference. First, ask for the scores before the explanation, so the explanation cannot talk the score up. Second, tell the judge what a fail looks like with a real example from your rejected drafts; a concrete bad example teaches more than an abstract rule.

• Inputs: rubric, approved example, candidate
• Output: per-criterion pass or fail, one reason each
• Verdict last, in a fixed, machine-readable format
• One real rejected example per common failure

How do you build the golden set?

A golden set is the fixed exam every version of the agent sits. Take 30 to 50 inputs from real, shipped work, with the copy your best reviewer approved and a note on why. Add the hard cases deliberately: a claim that is easy to overstate, a 30-character headline limit, a tone that slides into hype.

Every time a reviewer rejects a live output, add that input to the set. The set then grows to cover exactly the mistakes your team cares about, and a new model or prompt has to pass all of them before it goes live.

How do regression checks catch drift?

Drift is the slow slide in quality nobody notices because nobody reads every output any more. A regression check is the defence: run the full golden set whenever a prompt, skill or model changes, compare the score with the last version, and keep the old version live if the new one scores lower.

At Imagera AI, Shalini Verma runs this as a weekly golden-set review across eight production agents, which keeps first-pass approval above 85%. The broader result of the system, with a human gate on top, was first-pass approval rising from 55% to 88% and drafting time falling from 45 to 8 minutes.

How do you reduce the judge's biases?

Swap the order when comparing two versions and average the results, so position does not decide the winner. Ask for the score before the explanation, and cap the length the judge may reward. Where you can, use a judge from a different model family than the writer.

Most of all, keep a person in the loop for anything that ships. The judge decides what is worth a reviewer's attention; the reviewer decides what goes out.

Finally, remember what the score is for. A high judge score means the copy meets your written standard; it does not prove the copy will perform. Pair it with live results, such as click-through or conversion against a control, so the standard itself keeps getting better.