Case study
The report the budget was decided on showed 12,000 sessions out of 540,000: two weeks after go-live
Two reports on the same property over the same period, two different numbers. It wasn't our server-side release: it was an operator that filtered instead of classifying.
A consumer e-commerce store on Shopify, physical product, continuous spend on Google and Meta.
The constraint: we had just taken server-side tracking live, two weeks earlier. When a report stops adding up right after a release, suspicion falls on the release, and the first thing to do is rule out your own work before looking elsewhere. There was little time for it, because budget allocation was decided on that report every Monday.
The result: correctly assigned sessions doubled after the fix, and roughly 30,000 sessions in the social grouping alone were returned to the right channel.
Below is how we ruled out our own release in ten minutes, what was actually there, and the second thing that surfaced once the first was fixed.
What this case covers
- how to rule out your own release when a report stops adding up right after it
- how to tell whether a custom report is measuring or filtering
- how to distinguish a rise in unattributed data that is a fault from one that is a recovery
- how to stop a theme change from disappearing without a trace
Contents
- The client and the starting point
- Diagnosing the channel grouping
- Traffic with no source and the consent bridge
- Method note on the numbers
- What didn’t work
- What we would do now
- Conclusion
The client and the starting point
A Shopify store with traffic in the hundreds of thousands of sessions per analysis period, steady spend on search and social, and an in-house marketing team that uses a custom channel grouping as its reference report.
A channel grouping is the rule that decides which channel each visit lands in, based on campaign name and source. The default one is provided by the platform. The custom one is written by the company, and exists to mirror its own campaign structure instead of the standard one.
The starting position, with the source of each figure:
| Metric | Value | Source |
|---|---|---|
| Sessions in the default grouping | ~540,000 | GA4 |
| Sessions in the custom grouping | ~12,000 | GA4, same period |
| Creation date of the custom grouping | 27 May 2026 | GA4, change history |
| Date of our server-side release | 9 June 2026 | internal release log |
Diagnosis: an operator that filters instead of classifying
IN SHORT
What we did: compared the custom grouping with the default one over the same period, then read the conditions one by one.
Why: two reports on the same property over the same period returning different numbers have a different rule, not different data.
What we found: every condition used exact match instead of partial match, and the platform is case-sensitive.
The number for this section: assigned sessions doubled after the operator was replaced.
The first check was on dates, and it took ten minutes.
The custom grouping had been created on 27 May. Our release was on 9 June. The problem had existed for two weeks before we arrived, and nobody had seen it because nobody had ever put the two reports side by side.
With the release ruled out, we went and read the conditions. Every rule used the exact match operator, which assigns a session to a channel only if the campaign name is identical, character for character, to the string written in the rule.
Real campaigns are not named the way rules are. The condition looked for a single word; the campaign had a compound name with a prefix and a suffix. No match, therefore no assignment, therefore everything into the residual grouping.
On top of that came case sensitivity. Even where the name was conceptually right, one different letter dropped the session into the same residual grouping.
The fix was replacing exact match with partial match across all conditions. No code, no release, just configuration inside the platform’s interface.
Correctly assigned sessions doubled. Paid search and shopping campaigns reappeared in the report they had vanished from. In the social grouping alone there were roughly 30,000 sessions classified wrongly.
One clarification that matters, so the case doesn’t say more than it does. The team was not blind to overall traffic, because the default report always worked and always showed all 540,000 sessions. They were blind inside the report built to measure, which is the one used to decide.
What worked
Checking creation dates before conditions. It is the fastest check and it ruled out the most expensive hypothesis, namely that we had broken something, before spending time on anything else.
Traffic with no source and the consent bridge
IN SHORT
What we did: analysed the share of sessions with no source and separated the two causes behind it.
Why: a number that gets worse after a release can be a fault or the effect of a recovery, and the two require opposite actions.
What we found: part of it was traffic recovered by server-side, the other part was the consent bridge gone from the theme.
The number for this section: 17% of sessions with no source and no medium.
With the grouping fixed, a second problem surfaced, harder than the first because it had two overlapping causes with opposite signs.
17% of sessions arrived with no source and no medium. No channel, no campaign.
Part of that 17% was the recovery working. Server-side tracking had begun to capture traffic that was previously lost entirely. Visits that did not appear at all and now did, but with no provenance label, because that information is lost upstream. It is a number that gets worse because the system started seeing more.
The other part was a fault, and we isolated it in a day. The bridge between the consent banner and the tag manager, that is the piece of code that tells tracking what the user chose, had disappeared from the theme.
The cause: some portions of code had been commented out by editing the theme from the platform’s editor instead of from version control. Without version control there is no record of who touched what and when. The bridge reappeared after a trivial update to the theme, and from there consent went back to being transmitted correctly.
What worked
Separating the two causes instead of treating the 17% as a single problem. The actions were opposite: on one part there was nothing to do because the system was working well, on the other there was an urgent restore.
Method note on the numbers
Declared source of truth: the GA4 default channel grouping. Not because it is more accurate in absolute terms, but because it is the only one of the two that does not depend on rules written by the company, and we needed a reference independent of the report under examination.
| Published number | Source | Window | Model |
|---|---|---|---|
| ~540,000 sessions | GA4, default grouping | June 2026 analysis period | sessions, platform standard definition |
| ~12,000 sessions | GA4, custom grouping | same window, same property | sessions, client’s custom rules |
| ~30,000 social sessions | GA4 | same window | sessions in the social grouping, before the fix |
| 17% with no source | GA4 | same window | share of sessions with source and medium |
The gap between the two groupings was 97.8% on the same property and over the same period. It is the number that opened the engagement, and it is also the kind of gap that never has a statistical explanation.
What we are not able to measure, and with what degree of certainty.
The doubling of assigned sessions after the fix is verified by direct observation in the report, but the pair of absolute values before and after comes from two readings over windows we did not reconcile. That is why we publish the ratio and not the two figures. It is a deliberate choice: two absolute numbers on different baselines put into relation are exactly the defect this kind of work exists to find, and we don’t commit it on our own case.
The split of the 17% between recovery and fault is estimable with low confidence. We know both causes were present, the first by construction of the server-side setup and the second by direct evidence in the theme. We have not measured how much each weighs, and it would take a comparison against the pre-release period on the same definitions.
What didn’t work
We released without comparing the two groupings. Testing for the server-side release verified events, deduplication and values, but did not include a comparison between the client’s custom report and the default one. Had we done it, we would have found the defect two weeks earlier, and for two weeks suspicion fell on our work. What we did instead: testing now includes a comparison between every custom report the client uses and its default counterpart, before handover.
We circulated internally a pair of absolute values on the recovery without verifying the baseline. In a working note the result of the fix was reported as a move between two precise figures, taken however from readings over different windows. The pair was withdrawn and replaced by the ratio. What we did instead: every pair of numbers put into relation now carries window and scope written next to it, and if they don’t match we publish a single value or the ratio.
What we would do now
Reconcile the pair of absolute values on the recovery. It needs a single reading over a declared window, and it would give the case back its most concrete number.
Measure the split of the 17% between recovery and fault. With a comparison against the pre-release period, using the same channel definitions.
Move the custom-report check into continuous monitoring. Today it is a testing step. An automatic comparison between custom and default would flag the divergence on the day it appears instead of two weeks later.
Extend the theme version-control rule to every installed app. The consent bridge disappeared through a manual edit. The same can happen to any other snippet inserted from an admin panel.
Conclusion
The report on which budget allocation was decided showed 12,000 sessions against the 540,000 in the default report, on the same property and over the same period. After the fix, assigned sessions doubled.
What this engagement leaves behind, picking the four opening points back up.
How to rule out your own release: you compare creation dates before conditions. It is the fastest check and it closes the most expensive hypothesis.
How to tell whether a custom report measures or filters: you put it next to the default over the same period. If the two diverge widely, the difference is in the rules and not in the data.
How to distinguish a fault from a recovery: you look at what changed in the system shortly before. A rise in unlabelled data right after an intervention that widens collection has at least one positive cause.
How to stop a change from disappearing without a trace: theme changes go through version control and not through the admin editor, so there is a record of who touched what.
When this work makes sense, and when it doesn’t
It makes sense if you use a custom channel grouping to decide where to move budget, and if you have more than one report answering the same question with different numbers.
It doesn’t make sense if your traffic volume is such that channel classification changes no spending decision. And it doesn’t make sense to expect the fix to bring more traffic, because the traffic was already there: only where it gets counted changes.
The first step is to open your custom grouping and the default one over the same period and compare total sessions. If the difference exceeds a few percentage points, it is the rules that aren’t working.