Ad Incrementality Testing Case Study

The challenge

The account looked healthy by every number the team reported. ACoS had been flat at 38% for three years, the top campaigns had been running for years, and the core catalog held the first organic result on fourteen of its twenty most important search terms. Nobody questioned the spend, because every dollar of it had an attributed sale next to it.

That attribution was the problem. A sale credited to an ad is not necessarily a sale the ad caused: when the same listing occupies the sponsored slot and the first organic result on the same screen, a meaningful share of shoppers would have clicked either way, and the ad simply collects the credit. In a category of fast, low-consideration purchases the effect is at its strongest, since the shopper takes the first plausible result and stops. The brand had no way to distinguish paid sales it had bought from paid sales it had merely re-labeled, so its true cost of growth — total spend against total revenue — had been drifting upward for three years while ACoS stayed flat.

Our approach

We replaced attributed performance with a measured one.

Alternating-week switch-back tests — the same terms paused one week and running the next, repeated over eight weeks per cluster, so each term is compared against itself rather than against a forecast.
Testing ordered by organic rank, starting with terms where the listing already held position one and the overlap was most likely.
Total-account economics as the scoreboard — TACoS and contribution, not campaign ACoS, which is the metric that cannot see this problem at all.
The freed budget redeployed onto terms where the brand ranked ninth or worse, where a paid click buys visibility it does not otherwise have.

How we worked

Overlap inventory

Every search term where the account held a top-three organic position and also ran paid — 14 of the 20 core terms, carrying 38% of spend.

First holdout

Four terms switched off and on in alternating weeks across an eight-week cycle; total units on those terms fell far less than the paid units did.

Rolling tests

Successive eight-week switch-back cycles across the remaining overlap, one cluster at a time, rather than one account-wide switch-off.

Selective retreat

Paid withdrawn entirely on nine terms, kept at reduced bids on five where the test showed genuine incremental volume.

Redeployment

The recovered budget moved to weak-rank terms and new keyword expansion, where the incrementality is by definition total.

Quarterly re-test

Organic positions move, so the overlap inventory and its tests are re-run every quarter.

The results

Total advertising cost of sales fell from 27% to 16% while revenue grew 24% — the account got bigger on 27% less spend.

Revenue did not fall when the ads came off, because the organic result the brand already owned caught most of the traffic the ad had been claiming. The redeployed budget is where the growth came from: money moved out of terms the brand was winning anyway and into terms it was not visible on. The uncomfortable part was cultural — campaign ACoS ended the engagement around 50%, worse on paper than the 38% it started at, because the easiest and most flattering sales had been removed from it while the business underneath got cheaper to run.