AI vs Human Campaign Audit Accuracy: A Reproducible Test

"Our AI audits your ad account as well as a senior specialist" is now a standard claim, and it is almost always unfalsifiable as stated. There is no public benchmark, no shared task definition, and no agreed scoring — so the claim survives because nobody can check it.

This article does not give you a number. It gives you the protocol, because a number from us would be exactly as unverifiable as everyone else's. What follows is a test design you can run in an afternoon on an account you control, with a scoring rubric, the specific traps that make naive versions of this comparison worthless, and honest priors about where each side tends to win.

We are publishing the method rather than a score deliberately. A vendor-reported accuracy figure on a vendor-chosen dataset is marketing. A protocol you can run on your own account is evidence.

What "audit accuracy" has to mean before you can measure it

The first problem is that "audit" is not one task. Bundle these together and you get a meaningless average:

• Configuration defects — conversion tracking broken, wrong attribution setting, missing negative keywords, duplicate audiences, budget capped below target, geographic targeting including regions you do not ship to. These have a ground truth. Something is either misconfigured or it is not.
• Performance diagnosis — why CPA rose last month. This has no single correct answer, only better and worse reasoning.
• Strategic recommendation — what to do next quarter. This is judgement and cannot be scored for accuracy at all.

Only the first category is measurable. Any test that claims to score the second two is scoring agreement with whoever wrote the answer key, which is a measure of conformity, not accuracy.

So the test is scoped to configuration defects. That is a narrower claim than "audits as well as a specialist", and that narrowing is the point — it is the part you can actually check.

The protocol: seeded defects

The method is borrowed from software fault-injection. You take a real account, introduce a known set of defects, and measure what each auditor finds.

Step 1 — Build the account

Use a copy of a real account, not a synthetic one. Synthetic accounts fail this test in a specific way: they lack the accumulated cruft — the paused-but-not-deleted ad groups, the legacy naming conventions, the audience someone built in 2023 and forgot — that makes real auditing hard. An AI that scores well on a clean synthetic account tells you nothing.

Google Ads and Meta both support copying campaign structures into a sandbox or a paused parallel account. Do not run this on a live spending account.

Step 2 — Seed a defect list

Write down, in advance and in a sealed file, the defects you introduce. A workable set of 20 spread across severity:

Crucially, also include three or four things that look like defects and are not — a deliberately low budget on a testing campaign, a broad-match campaign that is intentionally exploratory, an unusual attribution setting that is correct for this business. These are your false-positive traps, and they are the single most informative part of the test.

Step 3 — Run both auditors blind

The human auditor gets account access and the same time budget you would give an AI — which in practice means a fixed time box, say 90 minutes. The AI gets the same account access.

Neither sees the defect list. Neither knows how many defects exist. Not disclosing the count matters more than anything else in the design: an auditor told "find 20 issues" will produce exactly 20 items, padding with noise once real findings run out, and your precision measurement is destroyed.

Step 4 — Score it

For each auditor, classify every item they report:

• True positive — a seeded defect, correctly identified and correctly characterised.
• False positive — an item reported that is not a defect, including the traps.
• False negative — a seeded defect not reported.

Then compute:

Precision = TP / (TP + FP)      "when it flags something, how often is it right"
Recall    = TP / (TP + FN)      "of the real problems, how many did it find"

Report both, separately, weighted by severity. A single blended accuracy number hides the entire interesting result, because the two systems typically fail in opposite directions.

The traps that invalidate naive versions of this test

Most informal comparisons people run are broken in at least one of these ways.

Counting findings instead of scoring them. "The AI found 34 issues and the human found 12" is not a result. It is a result about verbosity. Without precision, more findings is not better — it is usually worse, because every false positive costs a specialist an hour to dismiss.

Letting the AI define the categories. If you score against the AI's own output taxonomy, you will find that it has excellent coverage of the things it looks for. The defect list must be written before either auditor runs, by someone who is not building the test around one system's strengths.

Ignoring severity. Finding twenty naming-convention inconsistencies while missing a broken conversion action is a catastrophic audit that scores well on raw recall. Weight by severity or the number is actively misleading.

No false-positive traps. Without deliberately planted non-defects, you cannot measure precision honestly, and precision is where the practical difference between these systems lives.

A single account. One account is one draw. Account structure, vertical and platform all shift the result. Three accounts is a weak signal; one is an anecdote.

Scoring reasoning as accuracy. Tempting and wrong. Keep the diagnosis and recommendation sections out of the scored portion, and evaluate them separately as a qualitative read.

Honest priors: where each side tends to be stronger

Having scoped the test, it is worth being straight about what you should expect to see, so you can tell a surprising result from a confirmatory one.

Where automated audits are structurally advantaged. Exhaustive enumeration. A machine checks every campaign, every ad group, every setting, every time, and does not get bored on the fourth hour or skip the low-spend campaigns. Configuration defects in the long tail of an account are precisely where human attention degrades, and it is not a close contest. Consistency is the other one: the same account audited twice gets the same answer.

Where human auditors are structurally advantaged. Context that is not in the account. That a low budget is deliberate because the client's warehouse is capacity-constrained this quarter; that the odd attribution setting was a considered decision; that the "missing" geographic targeting reflects a distribution agreement. This information does not exist in the data, so no system reading only the data can get it right. This is exactly what the false-positive traps are designed to surface, and it is where blended-average reporting hides the real trade-off.

The practical implication. These profiles are complementary, not competitive — high recall on enumerable defects from one side, high precision on contextual judgement from the other. Which is why the useful production pattern is not "replace the auditor" but "let the machine enumerate and the specialist adjudicate". An audit surface that flags exhaustively and then requires human approval before acting is capturing both advantages; one that either acts autonomously on its own findings or relies on a person to remember to check the long tail is capturing neither.

That approval-gated shape is how Soku's account analysis is built — it reads the connected ad accounts and GA4, surfaces what it finds, and acts only on what a human confirms. We would rather you tested that claim with the protocol above than took our word for it.

Running it yourself: a checklist

1. Clone a real account into a paused environment.
2. Write a sealed defect list of ~20 items across three severity tiers, including 3–4 non-defect traps.
3. Seed the defects.
4. Run both auditors blind, same access, same time box, count undisclosed.
5. Classify every reported item as TP, FP or FN against the sealed list.
6. Report precision and recall separately, weighted by severity.
7. Repeat on at least three accounts before believing the result.
8. Publish your method if you publish your number.

Step 8 is the one the industry keeps skipping, and it is why no one can currently answer this question. A number without a protocol is a claim; a protocol without a number is still useful to everyone else.

Frequently asked questions

Can AI audit a Google Ads account as accurately as a human?

On enumerable configuration defects, automated checking has a structural advantage in recall and consistency. On judgements that depend on business context not present in the account data, it does not, and the failure mode is confident false positives. Any answer that does not separate those two cases is not answering the question.

Why not just publish your own accuracy number?

Because a vendor-run test on a vendor-chosen account is not evidence, and we would be asking you to accept exactly the kind of claim this article argues against. The protocol is the contribution.

How many seeded defects should I use?

Around 20 is enough to compute stable precision and recall without making the test take a day. Below about 12, single findings swing the percentages too much to be readable.

Should I tell the auditors how many defects there are?

No. Disclosing the count causes both human and automated auditors to pad to that number, which destroys the precision measurement — the most useful half of the result.

Does this work for Meta and TikTok too?

Yes, with a platform-specific defect list. The method is platform-agnostic; only the taxonomy of what counts as a misconfiguration changes.

What time box is fair to the human auditor?

Whatever you would actually pay for. The realistic comparison is not "unlimited human time versus instant AI" — it is the audit you would really commission against the one you would really automate.