Losers lose.Winners win.

10% never see the offer. That's how you know it paid.

Holdout testing for Shopify upsells. Why app dashboards flatter you, how a randomised holdout group works, and how to read a good week, a noisy week and the profit underneath.

Short answer: a holdout test randomly keeps a share of eligible orders, usually 10%, away from your upsell. Compare revenue per eligible order in the offer group with the holdout group. The difference, times the eligible orders in the offer group, is what the offers really added. Everything else is a dashboard being kind to you.

Why app dashboards over-credit

Open any upsell app and you'll see a big number: “revenue generated”. It's the sum of every offer a buyer accepted. It feels like proof. It isn't.

That number counts every accepted offer as new money. Three kinds of sale hide inside it:

  • Sales you'd have made anyway. The buyer who takes the matching lead today might have ordered it next week at full price. The dashboard books it as upsell revenue. Your bank account sees a sale moved, not a sale added, and at a discount too.
  • Sales that come back. Refunds and returns on add-ons rarely flow back into the app's headline figure.
  • Sales that cost more than they earned. A 20% discount and an extra parcel don't show up in “revenue generated”.

None of this means upsells don't work. It means the dashboard can't tell you how much they work. Only a comparison with orders that never saw the offer can do that. This is what people mean by an incrementality test: measure the increment, not the total. For upsells, it's the one piece of Shopify A/B testing that actually settles arguments.

Losers
Winners
Screenshot the app's “revenue generated”.
Compare revenue per eligible order against a holdout.
Count every accepted offer as a win.
Count only the difference between the groups.
Call the winner after a good Tuesday.
Stack weeks until the gap is bigger than the noise.

How a 10% holdout works

The rules are short. Getting them right is the whole game.

  1. Only eligible orders count. Eligible for the offer you're testing, that is. An order paid with Apple Pay, Google Pay, Klarna or iDEAL never sees Shopify's one-click post-purchase page (see why your upsell didn't show), so it doesn't belong in a one-click test. Most setups stop there; we give those orders a Thank you page offer and measure it the same way, with its own holdout. Either way, both groups are drawn from orders that could have seen the offer.
  2. Randomise per order, before the offer loads. Roughly 1 in 10 eligible orders goes to the holdout by chance. Not “weekends”, not “first-time buyers”, not “orders under €50”. Random. That's what makes the two groups comparable.
  3. The holdout sees nothing. No offer, no discount, no “maybe next time”. They go straight to the Thank you page.
  4. Measure revenue per eligible order in each group. Total revenue in the group, divided by the orders in the group. Include accepted add-ons, take off refunds. If you can, also count what the same buyers order in the next few weeks, because that's where “would have bought it anyway” shows up.
  5. The lift is the gap. Revenue per order with offers, minus revenue per order in the holdout. Multiply by all eligible orders assigned to the offer group to get the estimated total.

Why a holdout and not just “before vs after”? Because last month had different ads, different weather, a different product launch. Before-and-after comparisons blame or credit the upsell for all of it. A holdout runs at the same time, on the same traffic, so the only difference between the groups is the offer.

Tip for cutting noise: the original basket is already paid before the coin flip, so it's the same in both groups on average. If your reporting allows it, compare the revenue added after the original order (add-ons, later orders, minus refunds). Same answer on average, far less wobble from one big basket.

A worked week

Here's what a clean week looks like on an illustrative pet store running a matching-lead offer after a harness purchase.

Week 6Offers onHoldout
Eligible orders3,118347
Share of eligible orders90%10%
Take rate5.2%n/a
Revenue per eligible order€61.04€59.88
Lift per order+€1.16

Illustrative example, modelled figures.

The arithmetic:

  • Lift per eligible order: €61.04 − €59.88 = €1.16.
  • Estimated weekly lift: €1.16 × 3,118 eligible orders in the offer group = €3,616.88, call it €3,617. Count every order assigned to the offer group, not only those where the offer loaded, or the comparison is biased.
  • Sanity check: 5.2% take rate means about 162 add-ons (3,118 × 0.052 = 162.1). €3,617 ÷ 162 is about €22 per add-on, which matches a lead priced around there. If those two numbers disagree wildly, something in the tracking is off.

Now the honest part. Order values swing. On a store with a €60 average, individual orders can easily sit €40 above or below it. The average of 347 random orders will drift by a couple of euros from week to week by chance alone (roughly €40 ÷ √347 ≈ €2.15). A €1.16 gap is smaller than that wobble. So week 6 on its own says “promising, keep it running”, not “proven”. The proof comes from stacking weeks, and from the noise-cutting trick above.

One week is a reading. Several weeks in the same direction is a result.
Free 15-minute audit

Want a holdout on your thank-you page?

We check how many of your orders are eligible, what a fair holdout looks like at your volume, and the first offer we'd test against it.

No setup fee. No retainer. You only pay from the profit we add.

An inconclusive week

Same store, a quiet week in January. Fewer orders, same offer.

Week 9Offers onHoldout
Eligible orders82091
Take rate5.0%n/a
Revenue per eligible order€58.40€59.10
Lift per order−€0.70

Illustrative example, modelled figures.

The holdout “won”. Should you kill the offer? No. With 91 orders in the holdout, the chance wobble on its average is around €4 (€40 ÷ √91 ≈ €4.19). Two big baskets landing in the holdout is enough to flip the sign. The take rate held steady at 5%, so buyers still want the lead.

The call: inconclusive, keep collecting. Don't change the offer, don't change the price, don't declare anything. Add this week to the running total and read the cumulative numbers. Changing the offer every time a small week looks bad is how you end up testing nothing at all.

How many orders before you call it

There's no magic number, and anyone who gives you one without seeing your orders is guessing. What decides it:

  • How big the lift is per order. A €3 gap shows up much faster than a €1 gap.
  • How much your order values swing. A store selling €25 socks and €400 jackets is noisier than a coffee store where most orders are two bags.
  • How many orders land in the holdout. The wobble shrinks with the square root of the count: four times the holdout orders halves the noise.

Practical rule: you want hundreds of holdout orders at the very least, and for a small per-order lift on raw revenue, often well into the thousands. Wait until the gap is clearly bigger than the week-to-week wobble and points the same way for several weeks.

That's why volume matters. An illustrative €100k-a-month store at a €60 average order has about 1,667 orders a month (€100,000 ÷ €60). If 70% are eligible, that's about 1,167 eligible orders, and a 10% holdout of roughly 117 orders a month. That store needs several months for a confident read on a small lift. A store doing ten times that gets there in weeks. It's one reason we work with stores doing €100k+ a month: below that, the scoreboard takes too long to say anything.

Revenue lift is not profit

A holdout tells you the revenue the offers added. Your margin wants one more step. Take the costs that come with each add-on:

  • Product cost of the add-on.
  • Shipping: usually close to zero if it goes in the same parcel, painful if it needs a second box.
  • Discount: already out of the revenue if you measure what was actually paid; don't take it off twice.
  • Payment fees and pick-and-pack.
  • Refunds on add-ons.

Back to week 6. Say each €22 add-on carries €7 product cost, €1.50 extra pick-and-pack and packaging, €0.40 payment fees and a €1.10 allowance for refunds. That leaves €22 − €7 − €1.50 − €0.40 − €1.10 = €12 kept per add-on, about 55%.

Apply that to the lift: €3,617 × 12 ÷ 22 ≈ €1,973 contribution for the week. Still a clear win, but about half the headline. Now picture the same offer at 30% off with a second parcel. The revenue lift might look bigger and the contribution can go negative. Our AOV guide walks through exactly that trap, and the calculator runs it with your costs.

Illustrative example, modelled figures.

Testing against an offer you already run

Most stores aren't starting from zero. There's an offer live already, set up two years ago and never touched. Don't switch it off to test something new. Make it the baseline.

  • Keep the 10% holdout. It keeps telling you whether offers beat nothing.
  • Split the other 90% between the incumbent and the challenger, 45% each, at random.
  • Compare challenger vs incumbent on revenue per eligible order (and contribution). The challenger has to beat the incumbent, not just the holdout.
  • Change one thing. New product or new price or new sequence. Change three and you won't know which one did it.

Two arms of 45% each have less data than one arm of 90%, so head-to-head tests take longer to settle than offer-vs-holdout. Plan for it. What to test first is in our post-purchase upsell guide and the keep-or-kill scoreboard shows one call worked through.

The weekly scorecard

A good scorecard fits on one screen and ends in a decision. Ours lands in Slack every Monday and has the same parts every week:

LineWhat it tells you
Eligible orders per groupWhether the split is close to 90/10 and the week has enough volume to read.
Take rateWhether buyers want the offer. Steady take rate in a noisy week = keep collecting.
Revenue per eligible orderThe two numbers that matter, side by side.
Lift per order, this week and cumulativeThe reading, and the result.
The callKeep, change or kill, with one line on why.
Next testWhat goes up against the winner this week.

Keep means the cumulative lift is positive and the profit maths holds. Change means buyers want it but the price, discount or audience is wrong. Kill means several weeks of nothing, or a revenue lift that turns into a loss after costs. Losers keep every offer forever because the dashboard number keeps going up. Winners kill offers that don't pay and spend the slot on one that might.

FAQ

What is a holdout group?

A randomly chosen share of eligible orders, usually 10%, that never sees the offer. Because the choice is random, the holdout shows what those orders would have been worth without the offer. The gap in revenue per eligible order between the two groups is the lift.

Is a holdout test the same as an A/B test?

It's a type of A/B test where B is “no offer at all”. A normal A/B test compares two offers against each other; a holdout tells you whether running any offer beats running nothing. The best setup does both: a holdout plus your current offer against a challenger.

Doesn't a 10% holdout cost me money?

If the offers work, yes: the holdout orders miss out on the lift. That's the price of knowing. On a store where offers add about a euro per eligible order, the holdout costs roughly a euro on each held-out order. Without it you can't tell a good offer from one that only moves sales you'd have made anyway.

Why does my app dashboard show more revenue than the holdout?

Dashboards count every accepted offer as new revenue. Some of those buyers would have bought the same item later, some refund, and discounts eat part of the price. The holdout only counts the difference between the two groups, so it is usually lower and much closer to the truth.

Losers lose. Winners win.

Stop guessing. Keep score.

We build and run your post-purchase and thank-you page offers, hold out 10% of eligible orders, and send the scorecard every Monday. We're paid from a share of the extra revenue our offers add, measured against that holdout.

Free 15-minute call. For Shopify brands doing €100k+ a month.