Transcript
Welcome to Hard Questions, a SpendSignal podcast. I'm Sachin Uppal. Every episode, we take one question that no ad dashboard can answer, and work through it end to end. Today's question: Which of the ads I scaled last week were real winners — and which were just luck? And which 'losers' am I about to kill too early? My guest is Ananya Rao, Paid Social Lead at a performance agency.
Ananya, walk me through Monday morning.
Coffee, then Ads Manager sorted by CPA, last seven days. Anything with good CPA and some spend gets scaled — we bump budgets twenty percent or duplicate it into the scaling campaign. Anything with bad CPA after a reasonable spend gets paused. Madgicx rules do some of this automatically overnight. Motion gives me a nice visual board for the client call: here are your top ads, here's the hook rate.
Sounds disciplined.
It looks disciplined. (pause) Can I tell you the dirty secret?
Please.
Half of what we scale stops working within two weeks. We call it fatigue. Sometimes it is fatigue. But a lot of the time, I think the ad was never good. It was lucky.
Explain "lucky."
Say an ad spent ₹6,000 and got nine purchases. CPA ₹667. Target is ₹900. Looks like a winner. Another ad spent ₹2.4 lakh and got 290 purchases at ₹830. The first one looks better. But nine purchases is nothing. If two of those people had bought a day later from a different ad, the CPA would be ₹860. The second ad's number is solid; the first one's is a coin flip.
And your tools rank them...
By CPA. Ads Manager doesn't put a confidence interval on anything. Madgicx rules are "if CPA below X after spend Y, scale" — it's still raw CPA. Motion is brilliant for showing what the creatives look like and comparing thumb-stop rates, but it's not going to tell me whether a CPA gap is statistically real. So every week we're scaling noise and killing ads that never had a fair trial.
What does that cost?
Two things. Money: we pour budget into ads that regress to average. And learning: if a client's best concept got paused after nine unlucky days, we never find out. Then the creative team gets told "UGC doesn't work for this brand," which is a conclusion built on noise.
OK. So SpendSignal handles this with what we call the decision engine. You connect Meta, and it pulls ad-level spend, purchases, revenue and frequency, daily. For every ad, it compares this week with last week and with the other ads in the same campaign.
That's what I do in my head.
Right, except it does it with actual statistics. Every creative gets three tests. One: versus target — is its CPA significantly better or worse than your target, given how many purchases it has? Purchases are counts, so it uses the right kind of test for counts, not a gut call. Two: versus its own past — is it significantly worse than itself last week? That's the real fatigue signal. Three: versus peers — is it significantly better or worse than the other ads in its campaign?
And it gives me p-values?
P-values, plus an effect size, so you know whether a difference is real and whether it's big enough to care about. Then the combination of the three tests maps to one action. Better than target, past and peers: "Lead creative. Scale up." Fine on target but significantly worse than its own last week: "Fatigued. Scale down, brief new creatives." Not enough purchases to be different from anything: "Not enough evidence — hold."
"Hold" is the button I've never had.
It's probably the most valuable one. So your Monday screen says something like: "Of the twelve ads you scaled last week, four are significantly better than target. Five are indistinguishable from average — their good CPA was within noise. Three are significantly worse than their own last week: they're fatiguing, even though their CPA still looks OK. And two ads you paused had too few purchases to judge."
And the fatigue — can it tell real fatigue from a bad week?
That's what the versus-past test is for, and there's a fatigue map next to it: frequency against CPA, bubble size by spend. Upper right is tired and expensive. If an ad's CPA rose but its frequency is 1.4 and the whole campaign got worse, that's probably auction weather, not fatigue. If frequency is 4 and it's significantly worse than itself while peers are fine, that's fatigue.
Let me push. Statistics on 30 ads a week — won't everything just say "not enough evidence"?
Some will, and that's the truth about those ads. But it also stops you from wasting the decisions you can make. A 290-purchase ad that's significantly better than peers is a clear call. And it tells you which ads need more budget to reach a verdict, which is a much better use of test budget than pausing them.
How do I show this to a client who just wants "top 5 ads"?
You still show top ads, but sorted by what's proven, not by what's loudest. "These four are real. These five need another week. These three are fatiguing — brief replacements now, before CPA moves." And the creative team gets a different brief, which is the next episode.
(laughs) Fine. What's the limitation?
It judges ads on purchases and revenue, so if your conversion tracking is broken, it's wrong in exactly the same way Ads Manager is wrong. It uses a week-over-week window, so a brand-new ad has no "past" to compare to. And it recommends; it doesn't flip budgets for you — on purpose.
So my Monday ritual goes from "sort by CPA" to "sort by evidence."
That's the whole episode.
That's Hard Questions. Thanks, Ananya. If you want to ask this of your own data, connect your accounts at spendsignal.co. See you next episode.