AdTech QA checklist
Build reusable functional and integration coverage.
Test lab / Measurement testing
Impression Discrepancy Lab Scenario: Classifying a Reporting Gap
A reporting discrepancy is an investigation problem before it is a calculation problem.
Transparency: This is a synthetic, reproducible lab scenario created for education. It is not presented as a client result, Google result, DSP result, or production measurement.
Scenario setup
Use a fixed publisher, placement, date range, timezone, campaign, and creative. Assume system A reports an impression after a render event while system B records a downstream tracking event. These definitions are intentionally different in this example.
First comparison
Start with the same narrow slice rather than the full account. Record A, B, the difference, and the formula used. Then write down the event definition each system uses.
Classify the gap
If A counts renders and B counts tracking events, a difference can be expected when tracking is blocked or delayed. If both systems claim to count the same event, investigate identifiers, duplicate events, filtering, timezones, and delayed ingestion.
Resolution evidence
A strong conclusion identifies the scope, event definitions, identifiers checked, time window, likely cause, and confidence. Do not change tracking code until the baseline is captured.
Regression case
Keep one controlled transaction that can be followed through request, win, render, and measurement. Compare the same transaction across reporting systems after every tracking change.
Case study: Two platforms report different impression totals
The synthetic publisher ad server reports 10,000 impressions while an exchange dashboard reports 9,250 over the same named calendar day. A 7.5% difference is observable but not proof that either system is wrong.
Evidence ledger: what to capture
Export hourly event counts, timezone, event definition, request and impression IDs, deduplication rules, delayed-event handling, invalid-traffic filtering, and reporting refresh times.
Controlled experiment and expected observations
Align both reports to UTC and the same placement, device, and demand path. Reconcile requests, bids, wins, rendered impressions, and billable impressions as separate counters. Compare a stable historical hour before looking at a still-updating hour.
Reading the result without overclaiming
If the gap appears only in the final hour, reporting lag is plausible. If wins match but impressions diverge, focus on rendering, notification firing, and duplicate or missing impression events.
Correction, retest, and monitoring
Create a metric-definition contract and a daily reconciliation job that logs denominator, source, timezone, and processing cutoff. Escalate only after isolating the first event transition where counts diverge.
Important edge case
A VAST start event, client render callback, exchange impression notice, and ad-server counted impression are not necessarily the same event. Never label a difference fraud without evidence.
Hands-on verification worksheet
- Write a falsifiable hypothesis for this scenario: two platforms report different impression totals. State what observation would disprove it.
- Collect the minimum evidence before changing configuration. Export hourly event counts, timezone, event definition, request and impression IDs, deduplication rules, delayed-event handling, invalid-traffic filtering, and reporting refresh times.
- Create an isolated negative test and a positive control. Align both reports to UTC and the same placement, device, and demand path. Reconcile requests, bids, wins, rendered impressions, and billable impressions as separate counters. Compare a stable historical hour before looking at a still-updating hour.
- Classify the outcome using a precise event boundary. If the gap appears only in the final hour, reporting lag is plausible. If wins match but impressions diverge, focus on rendering, notification firing, and duplicate or missing impression events.
- After any correction, run the same fixture and retain before/after evidence. Create a metric-definition contract and a daily reconciliation job that logs denominator, source, timezone, and processing cutoff. Escalate only after isolating the first event transition where counts diverge.
- Include a note about alternative explanations. A VAST start event, client render callback, exchange impression notice, and ad-server counted impression are not necessarily the same event. Never label a difference fraud without evidence.
Practical investigation notes and worked methods
Build a comparable counting population
Two dashboards cannot be reconciled until their counting populations match. Define the publisher, ad unit, demand partner, device type, geography, UTC time interval and event type for the comparison. One platform may count rendered impressions while another counts a server-side auction win. Those are not interchangeable. Start with a single narrow hour and a single inventory slice; broad daily totals hide the exact handoff where divergence begins. Keep a written metric dictionary that specifies the triggering event, filters and deduplication behavior for each column.
Work through a synthetic funnel
Consider a controlled fixture with 100,000 eligible requests, 74,000 auction responses, 51,000 winning bids, 48,000 rendered creatives and 46,500 accepted impression events. These figures are hypothetical and intentionally represent different stages. A comparison between 51,000 wins and 46,500 impressions yields a numerical gap, but calling it a reporting discrepancy is misleading until the definitions are aligned. For the same event, compare counts after applying equivalent exclusions and time boundaries. Use an event-level identifier where permitted, or aggregated cohorts where privacy rules prevent joins.
Locate the first divergent handoff
Construct a funnel with requests received, requests validated, responses sent, bids selected, creatives rendered, tracking fired and tracking ingested. Record the numerator and denominator for each conversion. If publisher and exchange request counts agree but win counts differ, inspect auction filtering and win-notice logic. If render counts agree but impression counts differ, inspect beacon transport, deduplication and late-arriving events. If totals match by day but differ by hour, investigate timezone conversion, reporting cutoffs and event-time versus processing-time aggregation.
Control for late data and retries
Reporting systems often ingest events asynchronously. Repeat the same comparison after a fixed settling window, such as 24 hours, and document whether the gap narrows. Check whether a retry creates a second record, whether deduplication uses a stable event key, and whether a failed batch is replayed. Do not silently delete unmatched records to make totals agree. Preserve both raw and corrected counts and annotate any backfill. When a discrepancy depends on a particular timezone boundary, include requests immediately before and after midnight in the test fixture.
Turn the investigation into a testable outcome
The lab should end with an evidence-backed classification: definition mismatch, instrumentation loss, ingestion delay, invalid traffic filtering, timezone mismatch or unresolved. For each classification, define a verification test. A timezone fix should reconcile hour-level partitions without changing the full-day population. A beacon fix should increase accepted events without creating duplicates. An ingestion fix should close the gap after replay while preserving auditability. Publish the method and anonymized sample calculations, not private publisher revenue or user-level logs.