How to test whether an AI agent follows your brand guidelines

TL;DR

Test real tasks against specific brand rules. Inspect what the agent retrieved as well as what it made, use exact checks for exact requirements and a rubric for judgment, and keep uncertainty visible. Repeat the same cases after changes to the brand, prompt, model or tools.

Decide what the test should prove

Test three questions separately:

1. Did the agent retrieve the intended brand’s current guidance and assets?
2. Did it apply them correctly?
3. Did it flag missing facts or conflicts?

A wrong name may come from an old guide, a retrieval failure or ignored instructions. An overall “brand score” cannot tell you which to fix.

OpenAI recommends real tasks, repeated evaluation and checking automated scores against people’s judgments. Anthropic starts with specific, measurable criteria.

Build a small set of real jobs

Choose cases for the requirements they exercise:

Retain failures from real work and keep some cases out of daily prompt refinement.

Keep unchanged work too. Codex used guidance and files from Rainbrand to make a cotton guide and care insert for Son of a Tailor, a clothing brand. The care-insert front was reviewed twice without artwork changes; copy, logo, colour and typography remained passes.

The saved example tests repeated assessment of that piece. Keep unchanged outputs separate from repairs so a changed verdict is not mistaken for a changed design.

Write checks before looking at outputs

Define an observation another reviewer could repeat:

Separate release-blocking rules from preferences. Allow uncertain for insufficient evidence and not applicable for an absent element; first decide whether the task required that element.

Keep the comparison fair

• Decide whether you are comparing models or whole workflows. An old attachment versus a maintained brand tests the setup too.
• Retain the task, context, model where known, settings, tool results and output.
• Use fresh conversations for first-task behaviour; deliberately include history when testing long-running work.
• Repeat important cases under the same policy and retain failures.
• Change one factor at a time when diagnosing a miss.

Inspect what the agent retrieved

A Rainbrand Brief supplies relevant guidance and assets for a task. Save it with the output.

If a tagline is wrong, check whether the current wording reached the agent. Fix the source or retrieval when context is wrong; investigate instruction-following when it was right.

After applying a brand edit, request a fresh Brief to test current guidance. To compare models following identical instructions, give both the same saved guidance. Do not quietly substitute an easier test-only brand.

Use the right reviewer for each rule

Where practical, hide which model made each output. Ask reviewers for evidence; inspect disagreements before changing the rule or accepting a verdict. Compare a model grader with people on known successes and failures, and keep task facts and requirements alongside the findings.

Record failures you can fix

Keep a result per rule and task:

Task: [the actual deliverable]
Context used: [saved guidance or Brief reference]
Candidate: [workflow and model, when known]
Check: No unsupported product claims
Verdict: [pass / fail / uncertain / not applicable]
Evidence: [quote or location in the output]
Cause to investigate: [context / retrieval / generation / review]
Next change: [one proposed correction]

Report applicable failures, release blockers and uncertainty. A pattern such as correct names but repeated logo-placement failures suggests a concrete production fix; an average hides it.

Rerun when the work changes

Repeat affected cases after brand, model, connection or workflow changes. Keep repaired failures in the set and add cases from new mistakes.

Start with your team’s last three brand corrections: make each a check and run it on the next draft.