Soku AI
Ask a performance team how they test creative and you will usually hear a process that cannot produce a reliable answer: four ads in one ad set, different hooks and different formats and different offers, running until one "wins", where winning means it had the best CPA on the fourth day.
Every part of that is broken. The variables are confounded, the sample is too small to distinguish anything, and the stopping rule guarantees that noise gets promoted. This page is the framework we would use instead.
Last verified: 2026-08-18. The statistical reasoning here is general; the platform mechanics referenced reflect Meta and TikTok behaviour as of that date.
The core problem: you cannot afford the test you think you are running
A proper A/B test needs enough conversions per variant to distinguish a real difference from random variation. The arithmetic is unforgiving, and it is the reason most creative testing is theatre.
To detect a 20% relative difference in conversion rate with conventional confidence, you need roughly hundreds of conversions per variant — not per test, per variant. At a $40 CPA and four variants, a properly powered test costs tens of thousands of dollars and takes weeks.
Almost nobody has that budget for a creative test. So the practical question is not "how do I run a rigorous A/B test" but "what can I learn reliably with the volume I actually have?"
Three honest answers:
- Test bigger differences. A 20% effect needs a huge sample. A 2× effect needs far less. Concept-level differences are large; button-colour differences are not. Test things that could plausibly move the number a lot.
- Use upper-funnel metrics for early reads. Thumbstop rate, hold rate and CTR accumulate hundreds of times faster than conversions. They are proxies, not the goal — but they are readable in a day rather than a month.
- Accept directional answers. "This concept is probably better" is a legitimate and useful conclusion. Pretending it is "statistically significant" when it is not is where the damage happens.
The hierarchy: concept before execution
The single most valuable structural change is separating two kinds of test that most teams run simultaneously.
Concept tests ask: what is the idea? Problem-agitate versus social proof. Founder-to-camera versus product demo. Price-led versus outcome-led. Differences here are large — 2× to 5× swings are common — so they are detectable at realistic volume.
Execution tests ask: how well is this idea made? Hook wording, first-frame image, caption style, CTA phrasing, length. Differences are small — 5% to 20% — so they need volume most accounts do not have.
The rule: never run them at the same time. Find the winning concept first, at low volume, using upper-funnel proxies. Then invest in execution variants of the winner only, and only if you have the conversion volume to read them.
Running both together is what makes the classic four-ad test unreadable: when a founder-to-camera video with hook A beats a product demo with hook B, you have learned nothing about hooks or about formats, because you changed both.
A worked structure
Round 1 — Concept (3–5 concepts, one execution each). Judge on thumbstop and CTR after roughly 25,000–50,000 impressions per concept. Cheap, fast, and the differences are big enough to see. Kill the bottom half.
Round 2 — Execution (3–4 variants of the surviving concept). Now judge on conversion metrics, and run it long enough to matter. This is where you spend.
Round 3 — Iterate the winner. Produce new executions of the proven concept continuously. This is the ongoing work; rounds 1 and 2 are periodic.
The stopping rule
This is where most testing discipline dies. Someone checks the dashboard on day three, sees variant B ahead, and pauses variant A.
That is peeking, and it is not a minor sin. If you check repeatedly and stop the moment you see a difference, you will find a "significant" difference between two identical ads a large fraction of the time. Early data is dominated by variance; whichever variant happens to be ahead early is very often not the better one.
Set three things before launch, in writing:
- The primary metric. One. Not "CPA, but also CTR, and we care about ROAS." Choosing after the fact means choosing whichever metric tells the story you already believed.
- The minimum duration. At least one full week to cover day-of-week effects, and at least one full attribution window. Stopping inside the attribution window systematically favours variants with faster conversion paths, which is not the same as better variants.
- The minimum volume per variant. Whatever your budget honestly supports — write the number down, and do not read the result until you reach it.
Then genuinely do not act until all three are met. If you cannot resist looking, have someone else hold the results.
What to test, in priority order
Effect sizes are roughly ordered largest to smallest. Test top-down, because the big levers are the only ones most accounts can actually measure.
- Offer — not creative at all, and almost always the largest effect. If you can test the offer, do that first.
- Concept / angle — the narrative frame. Large effects.
- Format — UGC vs polished, static vs video, talking-head vs demo. Large.
- Hook (first 3 seconds) — the highest-leverage execution variable in video. Moderate to large.
- Ad copy structure — long vs short, benefit-led vs problem-led. Moderate.
- Thumbnail / first frame — moderate for static, smaller for video.
- CTA wording — small. Frequently tested, rarely decisive.
- Colour, font, minor layout — very small. Not worth a test slot at most volumes.
Most teams spend their testing capacity on items 6–8 because those are the easy things to vary. The effect sizes are inverted relative to the effort.
Platform mechanics that break naive tests
Meta's delivery optimisation actively works against clean A/B tests. Inside a single ad set, Meta allocates impressions toward whichever ad it predicts will perform, quickly and on thin evidence. So the "winner" partly caused its own win by receiving more and better-matched delivery. For a genuine comparison, use Meta's A/B test tool, which splits audiences rather than letting delivery pick — or accept that same-ad-set comparisons are directional only.
Audience overlap contaminates split tests. If two ad sets target overlapping audiences, they bid against each other and the same user sees both. Use the platform's built-in split-testing feature, which enforces mutually exclusive audiences.
The learning phase confounds early data. Performance during learning is not representative. Do not read results from an ad set that has not exited it.
Creative fatigue means results decay. A winner is a winner at a frequency. As frequency climbs, performance degrades. A test result has a shelf life; re-testing the same pair months later can legitimately reverse.
Reading a result honestly
Three questions before you declare anything:
Is the difference bigger than the noise? A quick sanity check: if a variant has fewer than ~30 conversions, you cannot distinguish anything short of a very large effect. Below ~100, only large effects. Treat everything else as directional.
Did the metric that matters move, or a proxy? A 40% CTR lift with flat CPA means the ad got more clicks from people who do not convert. That is not a win, and sometimes it is a loss.
Would you bet on it repeating? The most useful practical test. If the honest answer is "not really", record it as directional and let it inform the next round rather than promoting it as proven.
The minimum viable version
If the above is more process than your team will sustain, do these four things and you will still be ahead of most advertisers:
- Change one thing at a time, and make it a big thing.
- Decide the primary metric and stopping point before launch, in writing.
- Do not stop early, even when it looks obvious — especially then.
- Write down what you learned about the concept, not just which asset won. The asset expires in six weeks; the insight about what your audience responds to does not.
That last point is the compounding one. Teams that keep a written record of concept-level learnings get better at making creative. Teams that only track which ad won keep re-running the same test.
Frequently asked questions
How long should I run an ad creative test?
At minimum one full week to cover day-of-week effects, and at least one full attribution window. Duration alone is not sufficient — you also need enough conversions per variant, which at typical CPAs often takes longer than a week.
How many ads should I test at once?
For concept tests, 3–5 with one execution each. For execution tests, 3–4 variants of a single proven concept. More variants split volume further and make every result less readable.
What sample size do I need for a creative test?
It depends on the effect you want to detect. Detecting a 20% relative difference typically requires hundreds of conversions per variant. Below ~30 conversions per variant, only very large differences are distinguishable — treat everything else as directional.
Should I test creative in one ad set or separate ad sets?
Separate ad sets via the platform's built-in A/B test tool if you want a clean comparison. Within a single ad set, delivery optimisation reallocates impressions toward its early favourite, so the winner is partly an artefact of delivery rather than of the creative.
Can I use CTR instead of conversions to pick a winner?
For concept-level screening, yes — CTR and thumbstop accumulate far faster and the differences are large. But confirm on conversion metrics before scaling, because higher CTR with flat conversion rate means you bought cheaper clicks from worse-fitting people.
