When campaign numbers don't add up, how do you prove the problem isn't the campaign?

Context

In mobile advertising, an advertiser pays for attributed conversions. When a real, ad-driven install is recorded as organic, three things happen at once: the campaign looks like it underperformed, budget decisions get made on wrong numbers, and someone starts questioning whether the ad platform is doing its job.

There's an asymmetry underneath that took me a while to see clearly. The advertiser is often indifferent to attribution: they got the install either way. The ad platform is not indifferent at all, because attribution is what its revenue is measured on. So a tracking failure and a performance failure are indistinguishable from the outside, and only one of them is the campaign's fault.

I spent about 20 months at Appier as a Technical Solution Engineer intern, onboarding 6–8 campaigns a day across Japan, Korea, Greater China, the US and EMEA.

What changed how I saw this work was training. Once I was onboarding new interns while simultaneously explaining live incidents to account managers and BD across five markets, it stopped being a checklist I ran and became a body of reasoning I had to be able to transfer. Ruling out false organic attribution was the single biggest piece of it.

The question

This case didn't come from a client complaint. It came from us. The intern team ran a pre-launch verification on every campaign, and this one failed the check.

The main question: we triggered a conversion ourselves, on our own test device, and Appier never received the attributed postback. The advertiser confirmed their MMP had recorded the click-to-install. So where did the attribution go?

Sub-questions:

1. Did the event break between the app, the MMP and Appier, or did it arrive and get classified differently?
2. If it was classified as organic, was that the client's configuration, the MMP's fraud detection, or our own test procedure?
3. Could this be answered without escalating to a full-time engineer every time?

What would change the decision: if causes clustered into a few types, the fix isn't faster firefighting. It's reordering when we check, and removing the causes we create ourselves.

Data

• Source: Appier's cross-MMP integration data, queried to locate individual test records
• Grain: a single conversion event, traced along its path
• Period: roughly 20 months of day-to-day verification
• Volume and cost: I personally handled around four of these a month, each taking half a day to a full day to diagnose and return to a working state 🟡
• Known limits:
  • These were flagged cases, not a random sample. I can say which causes recur; I cannot say how often they occur across all campaigns.
  • Client-side implementation was frequently unobservable: I could see the symptom, not the app's internals.
  • Client-level commercial data is out of scope for what I can discuss.

Approach

Step 1: Find where the data stopped, before theorising about why.

An event has a path: the app fires it, the MMP receives and attributes it, the attributed postback reaches Appier. Each step is separately checkable, and checking them in order eliminates whole categories of explanation at once.

We queried the cross-MMP integration data for the specific record our test device had generated.

If the event isn't there at all, the next question is whether any demand-side platform sees it. If no DSP has the event, the problem is upstream of all of us: either the client's app isn't firing it or the MMP's SDK isn't implemented correctly, and the client needs to work directly with their MMP. That single comparison separates "our integration is broken" from "the instrument was never installed."

In this case the event was there. The MMP had received it. It had simply labelled it organic.

That narrowed things sharply: not a delivery failure, a classification failure. Nothing broke in transit. Something decided our ad-driven install wasn't ad-driven.

Step 2: Order the remaining causes by how cheaply you can rule them out.

This is the part I'd defend hardest, and it isn't obvious. The instinct is to order suspects by likelihood. The better ordering is by cost of elimination, because a cheap test that clears a whole branch is worth more than an expensive test on a likelier one.

So the sequence went:

First, re-test on a different device and network. Minutes, no external dependency, entirely within our control, and it rules out our own test procedure tripping the MMP's fraud detection. One major MMP chains campaign, ad group and creative parameters together; if a tester doesn't substitute a fresh set on each run, repeated identical combinations look like fraudulent traffic and get reclassified as organic.

Second, check the attribution window. Either from configuration screenshots the advertiser supplied, or directly in the MMP dashboard where we had been granted access. A real conversion falling outside the configured window is indistinguishable, in the data, from one that never came from an ad.

Third, and in parallel if the first two clear: ask the client whether they've set custom rules on IP ranges or device language, and contact the MMP directly. That last one is genuinely rare, and it's last precisely because it's the slowest: it puts you in someone else's queue.

Step 3: Notice that it's a category, not an incident.

I hit this reasoning repeatedly. More to the point, I kept explaining it: to new interns, to account managers and BD across five markets, later to campaign managers, most of whom had no reason to know what an attribution window is. It was the repetition of the explanation, more than the repetition of the diagnosis, that told me this couldn't keep living in my head.

What I considered and rejected:

Ask a full-time engineer each time. Fastest per case, wrong at any volume. It makes one person a bottleneck, spends expensive time on a repeatable task, and guarantees the interns never learn to reason through it.

Build a proper internal tool with a full backend. The most capable option and the one I could least justify. It needed permissions I didn't have and a maintainer after I left, and I was an intern with a known end date. A tool nobody owns after you leave is a liability you hand your team, not an asset.

What I built sat between those: SQL queries, written together with a full-time engineer since the database access and query logic were theirs, wrapped in an internal API endpoint and surfaced through a spreadsheet interface the interns already knew how to use. Paired with the written diagnostic path, it let anyone answer did this conversion reach the MMP, and how was it classified without asking a person.

Findings

Attribution-window misconfiguration accounted for roughly two-thirds of cases; other postback settings for about a quarter. 🟡 (estimates from cases I personally worked, not a measured distribution, see Limitations)

The remainder were the fraud-rule cases. They were the smallest group, and they were the ones I learned the most from, because they were the only category where the data was completely correct and we were the ones generating the false signal. Our test procedure was producing a pattern that any sensible fraud rule should catch.

That's also why they sit first in the diagnostic path rather than last. Not because they're common (they aren't) but because they're the cheapest branch to eliminate and the only one entirely within our control.

Recommendation

Verify before launch, not after. If causes cluster into a small set, they belong on the pre-launch checklist. Across the campaigns I onboarded, post-launch tracking failures became rare; what remained were later configuration changes on the advertiser's side affecting performance rather than breaking attribution outright: a different problem, owned further down the chain. 🟡 (self-observed, no control)

Remove the failure modes you create yourself. I built a small mobile test page that automatically substitutes fresh campaign, ad group and creative parameters on each run, verifies the test device's IP resolves to the correct country, and switches between IDFA/GAID substitution and probability matching depending on what the case requires. It cut test-procedure errors substantially: it removed an entire category of false alarm before it could be raised.

Write the reasoning down where it can be maintained. I kept a reference document of the anomalies interns encounter, with response templates, and updated it in the weekly TS intern meeting whenever a new one appeared. Documentation that isn't on a maintenance rhythm goes stale and then gets distrusted, which is worse than not having it.

How I'd measure it: the share of tracking issues found after launch rather than before.

Put in cost terms, the case for doing any of this is simple. Four cases a month at half a day to a day each is somewhere between two and four working days: roughly ten to twenty percent of one person's month, spent re-deriving the same reasoning. That's the number worth attacking, not the individual case.

Key learning

Written reasoning outlasts the person who wrote it, and that's the point. The query made lookups fast, but what stopped the same question being asked over and over was recording why the answer was the answer. After I left for military service, former colleagues told me the documentation had become sufficient on its own for onboarding new interns: they could work from it without a face-to-face handover. 🟠 (reported to me secondhand) That outcome mattered more to me than the tool did.

Knowing what a stakeholder is actually measured on is what unblocks them. Configuration details from advertisers sometimes arrived slowly, and chasing them didn't help. What helped was making the consequence legible: a campaign that can't be attributed correctly is a campaign whose performance can't be defended, and getting campaigns live and performing is what account managers and BD are measured on. Framed that way, the same request moved much faster. That wasn't a communication trick; it was finding the one place where my problem and their incentive were already the same problem.

And I now read clean data differently. Most analysts receive the table after it's assembled. I spent 20 months watching tables get assembled wrongly, and learning that "the instrument is broken" and "the instrument is correctly measuring something you didn't intend to do" look identical on a dashboard and require completely different responses. Telling those apart quickly turned out to be most of the job.