The situation
A metric the business watched closely appeared to collapse. The reasonable interpretation was that user activity had genuinely dropped, and the organisation was preparing to respond to that. Confirming or refuting it needed to happen quickly and be provable either way.
What I did
- Proved the underlying events were still present in the source at the expected volume, so the drop was not a change in user behaviour.
- Traced the cause upstream: a removed retry window combined with processing lag meant events were arriving too late to be attributed to an individual item.
- Pinpointed the day the divergence began, rather than accepting the date the problem was first noticed.
- Backfilled the broken partitions unsampled, so history was corrected rather than approximated.
- Opened structural follow-up work to move off the sampled source entirely, so the same failure could not recur.
What it means for you
A metric falling is not the same as the thing it measures falling. Establishing which one you are looking at, before the business reacts, is often worth more than the fix itself.
Context
These are from eight years owning the data warehouse and reporting function of a consumer marketplace platform, in-house rather than as an outside consultant. The employer and the internal system names are withheld; the numbers are the real ones.