Soku AI
Every attribution model you have ever used answers the wrong question. Last-click, data-driven, multi-touch — they all ask which touchpoint should get credit for this conversion. That question assumes the conversion happened because of advertising.
Incrementality testing asks the only question that matters for budget decisions: would this conversion have happened anyway?
The gap between those two questions is where most wasted ad spend lives. Branded search is the classic case — it converts beautifully in every attribution report and a meaningful share of it is people who were going to type your name in regardless. Retargeting is the other: it reports the highest ROAS in the account and it is aimed at people already deep in a purchase decision.
This guide covers what incrementality testing is, the three methods, and how to run one. But it leads with the part almost every other guide omits, because it determines whether the rest is even worth reading: the arithmetic of whether you have enough conversions to run a valid test at all. Most businesses being told to "run a lift test" cannot, and no amount of methodology fixes that.
What incrementality actually measures
An incrementality test is a randomized controlled experiment. You withhold advertising from a randomly chosen group, serve it to everyone else, and compare conversion rates. The difference is incremental lift — the conversions that exist only because the ads ran.
The output is a number you cannot get any other way:
Incremental lift % = (conversions in exposed group − conversions in control group) ÷ conversions in control group
If your Meta campaign reports 1,000 conversions and a lift test shows 30% incrementality, then 300 conversions were caused by the ads and 700 would have happened anyway. Your reported CPA is not your true CPA — the true one is roughly 3.3× higher.
That multiplier is why this matters. Attribution reports tell you how to allocate credit. Incrementality tells you whether to keep spending. Teams routinely discover that the channel reporting the best ROAS is the least incremental, because platforms are very good at finding people who were already going to buy.
The feasibility question: can you actually run one?
Here is the calculation the "how to run an incrementality test" articles skip, and it is the first thing you should compute.
An incrementality test is a statistical comparison between two groups. To detect a difference of a given size, you need a minimum number of conversions in each group. The standard requirement — 95% confidence, 80% statistical power — gives you a clean rule for count data:
Conversions needed per group ≈ 15.68 ÷ (relative lift)²
That constant is (1.96 + 0.84)² × 2, the standard z-scores for 95% confidence and 80% power. The result is unforgiving, because the lift term is squared:
| True incremental lift | Conversions needed per group | Total conversions needed with a 10% holdout |
|---|---|---|
| 50% | 63 | ~630 |
| 30% | 175 | ~1,750 |
| 20% | 392 | ~3,920 |
| 15% | 697 | ~6,970 |
| 10% | 1,568 | ~15,680 |
| 5% | 6,272 | ~62,720 |
Now read the widely repeated heuristic — "you need at least 200 conversions in the control group" — against that table. Inverting the formula, 200 conversions in control buys you the ability to detect a 28% lift, and nothing smaller. A hundred conversions detects 40%.
That is the sentence worth taking away from this article. If you run a test with 200 control conversions and it comes back "no significant lift," you have not learned that your ads are non-incremental. You have learned that their incrementality is probably below 28% — which describes almost every healthy, well-optimized channel. Most published lift tests at small scale are underpowered, and an underpowered null result is routinely misread as proof that a channel does nothing.
The practical decision rule:
- Under ~600 conversions in the test window: you cannot run a valid platform lift test. Anything you get back is noise.
- 600–4,000 conversions: you can detect large effects (20–50%). Useful for questions like "does this entire channel do anything," not for tuning.
- 4,000–16,000: you can detect 10–20%. This is where lift testing starts informing budget decisions.
- 16,000+: you can detect 5–10% and iterate.
This is also the real explanation for the platform budget minimums. Meta recommends roughly $30,000/month for Conversion Lift, and a minimum audience of about 200,000 with a holdout commonly in the 5–20% range. That is not a revenue gate — it is the spend level at which a typical advertiser generates enough conversions for a 10% holdout to contain a testable number. Google has lowered its threshold from around $100,000 to $5,000, which widens access but does not repeal the arithmetic: the binding constraint is conversion volume, not budget.
The three methods
1. User-level holdout (platform-native lift studies)
The platform randomly selects a share of your target audience and never shows them your ads, then compares conversion rates. Meta Conversion Lift and Google's Conversion Lift based on users both work this way, and Google has moved lift studies into the Experiments section of Google Ads.
Strengths: cleanest randomization available, since the platform controls exposure at the individual level. Lowest noise per conversion.
Weaknesses: it only measures within that platform's own universe — a Meta lift test cannot see that the same user also saw your Google ad. You are trusting the platform to grade its own homework, and the holdout is invisible to you.
Use when: you have the volume, and the question is "how incremental is this specific campaign."
2. Geo holdout
Split markets rather than users: run the campaign in a set of test geographies, go dark in a matched set of control geographies, and compare. Match test and control markets on baseline conversion rate, population, and seasonality using DMA-level historical data before you start — badly matched markets are the single biggest source of false results.
Strengths: platform-independent and cross-channel. It measures total business impact, including the offline and word-of-mouth effects no pixel sees. Nobody is grading their own homework.
Weaknesses: far noisier, because geographies differ in ways you cannot fully control. It needs 4–8 weeks minimum to average out local noise, and it costs real revenue in the dark markets.
Use when: you need a cross-channel or whole-business answer, or you are below platform minimums but have geographic scale.
3. Always-on ghost bidding
The platform enters the auction for the holdout group, records that it would have won the impression, then does not serve it. This isolates the effect of the ad from the effect of being the kind of person the algorithm targets — the sharpest available control for selection bias.
Strengths: the most methodologically rigorous design, and it can run continuously rather than as a discrete study.
Weaknesses: limited availability, generally through measurement vendors rather than self-serve, and the least transparent to audit.
How to run one: seven steps
1. Compute feasibility first. Take your conversions in the intended test window, decide the smallest lift that would change a decision, and check it against the table above. If the numbers do not work, stop here and use a geo test or a longer window instead of running an underpowered study you will misinterpret.
2. Write the decision before the test. "If incremental CPA is above $X we cut this channel's budget by half." Committing in advance is what stops a null result from being explained away, and it forces you to name the minimum detectable lift honestly.
3. Pick one variable. One channel, one campaign type, one question. Tests that change two things at once produce results you cannot act on.
4. Set the holdout size deliberately. A larger holdout gives more statistical power and costs more forgone revenue. 10% is the common default; if you are volume-constrained, a 20% holdout roughly doubles your control conversions and materially improves detectable lift — at the cost of suppressing more real demand.
5. Run it for at least one full conversion cycle. Two weeks minimum for user-level tests, four preferred, 4–8 weeks for geo. If your median time from first click to purchase is 12 days, a 14-day test measures almost nothing. Check your actual lag before choosing a duration.
6. Do not peek and stop early. Checking significance repeatedly and stopping when it first appears inflates your false-positive rate substantially. Set the end date up front and honour it.
7. Report the confidence interval, not just the point estimate. "22% lift" is not a finding. "22% lift, 95% CI 4% to 40%" is — and it tells your CFO the honest truth, which is that you know the effect is positive and you do not know its size precisely.
Reading the result honestly
A null result is not zero incrementality. It means the true effect is smaller than what your test could detect. Report it as "we can rule out a lift greater than X%," never as "the channel doesn't work."
A very high lift on a small test is usually noise. Underpowered tests that do reach significance overestimate effect size — an inherent property of the design, not bad luck. Treat an 80% lift from a 300-conversion test with suspicion.
Incrementality is not stable. It moves with saturation, seasonality, competition, and creative. A test result is a reading from a moment, not a constant. Re-test the channels you rely on at least annually.
Expect the result to be uncomfortable. The channels that report best usually test worst, because platform attribution is optimized to find people who convert, not people who convert because of the ad. If your lift test confirms every number in your dashboard, check the test setup.
Where this fits in your measurement stack
Incrementality does not replace attribution — the two answer different questions at different cadences. A workable stack:
- Attribution reporting — daily and weekly operations. Which campaigns and creatives to adjust now.
- Incrementality tests — quarterly or per major decision. Whether a channel deserves its budget at all.
- Marketing mix modelling — annually. Long-run, whole-business allocation across everything including offline.
The pattern that works: use incrementality to calibrate a correction factor per channel, then apply it to your daily attribution numbers. If Meta tests at 30% incremental, divide reported Meta conversions by roughly 3.3 in your planning model and keep operating off the dashboard day to day. You get causal grounding without needing a test running at all times.
This is precisely the divergence Soku is built to surface: reading the ad platforms and GA4 together, and flagging where reported performance and real business impact are drifting apart — the signal that a channel is due a lift test before you scale it further.
For the tooling side of the same problem, see our comparison of paid-media reporting tools and our guide to measuring AI ad creative ROI.
Frequently asked questions
What is incrementality testing in simple terms?
It is a controlled experiment that withholds your ads from a random group of people or markets, then compares their conversion rate to the group that saw the ads. The difference is the conversions your advertising actually caused. Unlike attribution, which assigns credit among touchpoints, incrementality tells you whether the conversion would have happened without any advertising at all.
How many conversions do I need to run an incrementality test?
Roughly 15.68 ÷ (lift)² per group, at 95% confidence and 80% power. Detecting a 20% lift needs about 392 conversions per group; a 10% lift needs about 1,568. With a typical 10% holdout, that means roughly 3,900 and 15,700 total conversions in the test window respectively. Below about 600 total conversions, no valid platform lift test is possible.
What is the difference between incrementality testing and attribution?
Attribution distributes credit for conversions that already happened across the touchpoints that preceded them. Incrementality measures whether those conversions would have happened without the advertising. Attribution can never detect a channel that takes credit for demand it did not create — branded search and retargeting are the standard examples.
How long should an incrementality test run?
At least one full conversion cycle. Two weeks is the practical minimum for user-level platform tests and four weeks is better; geo holdout tests need 4–8 weeks to average out local market noise. If your median time from first click to conversion is two weeks, a two-week test captures almost none of the effect.
What is a good incremental lift percentage?
There is no universal benchmark, and be sceptical of anyone quoting one — it depends entirely on channel, brand awareness, and saturation. Prospecting on a new audience is often highly incremental; branded search and retargeting are usually much less so. What matters is your incremental CPA against your target, not the lift percentage against someone else's average.
Can I run an incrementality test on a small budget?
Not a platform lift study, generally. Google has lowered its minimum to around $5,000 but the real constraint is conversion volume, not spend. Below roughly 600 conversions in the test window, use a geo holdout over a longer period, or accept that you can only test very large effects — and design the test to answer "does this channel do anything at all" rather than trying to tune it.
Does a null incrementality result mean my ads don't work?
No, and this is the most common misreading. It means the true lift is smaller than your test was powered to detect. A test with 200 control conversions can only detect a lift of about 28% or larger, so a null result there is entirely consistent with a genuine 15% lift. Always report a null as an upper bound, not as zero.
