Transcript
Welcome to Hard Questions, a SpendSignal podcast. I'm Sachin Uppal. Every episode, we take one question that no ad dashboard can answer, and work through it end to end. Today's question: Which channel's number, if it's wrong, costs me the most money — and what's the single test that settles it? My guest is Priya Nair, VP Marketing at a Series B consumer brand.
Priya, you have more measurement than anyone we've talked to. Attribution, a mix model, lift tests. So you must know exactly what's working.
(laughs) I have more numbers than anyone. That's different. Every tool gives me a confident answer. They just give me different confident answers.
Give me an example.
Meta. Ads Manager says ROAS 3.2. The MTA tool says 2.1. The Robyn model from last year said 1.7. Our Haus test in Q2 said Meta was clearly incremental, at the spend level and in the region we tested. So is Meta a 1.7 or a 3.2? At ₹80 lakh a month, that's the difference between "scale it" and "cut it."
What do you tell the board?
The truth, carefully dressed up. I show a range of methods and explain why they differ. And the board asks the only question that matters: "So what are you going to do, and how sure are you?" And I don't have a good answer to "how sure."
Why not?
Because none of these tools tell me where they are uncertain. Ads Manager gives one number. MTA gives one number. Robyn technically gives distributions, but my analyst left, and nobody wants to re-run it. The Haus test is the most rigorous thing I have, but it's one channel, one quarter. And I can only afford a handful of tests a year. So the real question I can't answer is: which test should I run next?
Say that question in plain words.
"Where is my plan most likely to be wrong, weighted by how much money is riding on it?" If Google Search could be anywhere between 1.9 and 2.3, who cares — it's above breakeven either way. If Meta could be anywhere between 1.2 and 3.1, and breakeven is 1.45, then the whole Meta decision could flip. That's where I should spend my test budget. But nothing ranks that for me.
OK. So this is the question SpendSignal was really built around. Let's go end to end. You connect your ad accounts and Shopify, set your economics once — what you keep from revenue after discounts, returns, margin and fulfilment — and that gives every channel a breakeven.
Same as the other episodes.
Same foundation, different reading. The mix model doesn't give one number per channel. It refits itself many times on resampled data and gives each channel's incremental ROAS as a range. Then it compares that range with your breakeven. If the whole range is above breakeven: confident scale or hold. Whole range below: confident trim or cut. Range crosses breakeven: it's labelled uncertain, and it refuses to pretend.
That alone is more honest than my board deck.
Then it goes one step further. It looks at which channel has the widest range relative to its estimate, and turns that into a recommendation: "Run a four-week geo holdout on Meta before moving more budget." So instead of picking next quarter's Haus test by politics, you pick it by where the model is least sure.
What about my other numbers? The MTA, the old Robyn run?
SpendSignal shows platform ROAS next to incremental ROAS for every channel, so the gap between "what Meta claims" and "what Meta causes" is explicit — that's the over-credit number. It also has two engines that can be compared side by side. Where they agree, you can move with confidence. Where they disagree, that's another signal that a channel needs a test before a decision.
And the forecast? The board always asks "what's the number next quarter?"
Forecasts come as P10, P50 and P90, split into baseline revenue — what you'd make without paid media — and the media contribution on top. So you can say "our base case is ₹X; our downside, one in ten, is ₹Y." That's a far better board conversation than one number you'll miss.
Let me be the CFO. "So your tool says it doesn't know. Why are we paying for it?"
Because "doesn't know" is attached to a specific channel and a specific amount of money. Knowing that Google Search is narrow means you stop debating it and stop paying for tests there. Knowing that Meta flips inside its range means you hold Meta flat — don't scale, don't cut — put new budget where the range is fully above breakeven, and run the one test that settles the biggest open question.
And after the test?
The test result becomes the anchor for that channel, and the model keeps the rest of the plan current between tests. Tests and the model stop competing; the model tells you which test to run, and the test tells you whether the model's range was right.
What's the honest limitation?
The ranking is by range width relative to the estimate — you still bring the judgement about how much money rides on each channel. It's not a formal expected-value-of-information calculation. It needs real variation in spend to narrow any range; a channel you've spent flat for a year will stay uncertain until you move it or test it. And it's built on your Shopify revenue, so if revenue isn't all in Shopify — say Amazon sales — those need to come in too.
So my next board slide is...
"Here's every channel with its range against breakeven. Google Search, YouTube: confidently above, so new budget goes there. Meta: range 1.2 to 3.1 against 1.45, so ₹80 lakh a month rides on a call we can't make yet. We're holding it flat and running a four-week holdout. Forecast: base case ₹X, downside ₹Y."
That's the first slide where "I'm not sure" makes me look more credible, not less.
That's the point. Every tool gives you a number. The hard part is knowing which number not to trust.
That's Hard Questions. Thanks, Priya. If you want to ask this of your own data, connect your accounts at spendsignal.co. See you next episode.