AdTech Engineering Research & Labs · Reproducible auction incident analysis

Designing an AdTech Incident Replay System

Production ad auctions are difficult to reproduce because configurations, partner behavior, privacy signals and time-dependent decisions change. A replay system should capture the smallest safe event envelope that explains a decision while avoiding storage of sensitive bidstream data.

Scope: This is an independently written engineering method with a synthetic teaching scenario. It is not a report of customer data, a production incident or measured industry performance.

1. Why this investigation matters

Production ad auctions are difficult to reproduce because configurations, partner behavior, privacy signals and time-dependent decisions change. A replay system should capture the smallest safe event envelope that explains a decision while avoiding storage of sensitive bidstream data. The key mistake is to begin with an attractive dashboard number before deciding what was actually measured. A useful investigation states its population, event definition, observation window and expected invariant. It then distinguishes a genuine system failure from a difference in counting or policy. This matters because an optimization that improves one stage may damage the next stage and leave publishers worse off.

The method below deliberately separates observation (what was recorded), interpretation (what might explain it), and action (what to change and how to verify it). Each finding should have a counterexample: what evidence would make the current explanation wrong? Writing down that question prevents teams from treating correlation as proof.

2. A system model that can be inspected

The investigation boundary is: incident ID → sanitized request envelope → versioned configuration → deterministic rule evaluation → simulated partner responses → expected versus observed diff. Each transition has its own event semantics, ownership and potential failure modes. A missing downstream event does not establish which upstream component failed; the same symptom can arise from suppression, transport loss, sampling or delayed reporting.

Build a small data dictionary before querying production telemetry. For each stage, record whether the count is based on browser events, server acceptance, exchange processing or billing notices. Use one time zone and state whether late-arriving records are revised. When a field is absent, record unknown rather than silently assuming false. A dashboard that mixes incompatible denominators can produce an impressive but meaningless percentage.

3. Worked synthetic case study

A synthetic replay reproduces a floor-rule regression: a placement floor changed from 0.40 to 1.40 in a versioned configuration, excluding three bids that would otherwise have qualified. The replay explains the filtering decision without claiming what those bidders would have done in production.

Start by drawing the event boundaries and selecting one narrowly defined slice: a single placement, a known configuration version and a bounded time window. Calculate the difference between adjacent stages, then check whether the populations actually refer to the same opportunities. Use synthetic IDs such as trace-001 in examples; do not publish live bid requests or user identifiers. A good incident write-up includes both the expected path and a path that was intentionally suppressed for a legitimate reason.

For a second pass, challenge the initial explanation. If the apparent loss disappears when late events are included, the problem is reporting timeliness rather than auction execution. If only one browser family is affected, investigate client behavior before adjusting exchange-wide settings. If the change aligns with a configuration edit, test a controlled rollback in staging. None of these checks should be skipped merely because one chart appears persuasive.

4. Investigation procedure

1. Version floor rules and bidder maps. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
2. Hash or discard user identifiers. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
3. Store synthetic partner responses. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
4. Separate deterministic decision replay from live network behavior. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
5. Compare output with a documented oracle. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
6. Set retention and access controls.. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.

Use an evidence log with columns for hypothesis, observable signal, query or reproduction step, result, confidence and next action. Preserve the raw definition of every metric used in a screenshot. For code or configuration changes, include a test that would have failed before the fix and passed afterward. If the issue cannot be reproduced, mark it unresolved and document the missing instrumentation rather than inventing a root cause.

5. A practical worksheet for operators

Copy the following worksheet into an issue tracker. It is designed to turn an interesting symptom into a falsifiable technical claim rather than a generic checklist.

6. How to validate the result

Validation should operate at three levels. First, verify that the instrumentation itself emits events at the documented boundaries. Second, reproduce the mechanism with controlled synthetic inputs, including an expected-success case and a deliberate failure case. Third, observe the relevant metric after the change while watching adjacent guardrails. A result is not reliable when only the headline number improves and downstream rendering, latency or billing is ignored.

Where a numerical estimate is useful, report the denominator explicitly. For example, a 5% decrease in timeouts among dispatched auctions is not the same as a 5% increase in revenue per eligible page view. Include confidence intervals or uncertainty bounds for sampled data, and avoid extrapolating from a small staging dataset to live traffic. If the sample is not representative, explain why the method is still useful and what remains unknown.

7. Failure modes and trade-offs

Replay cannot reconstruct unknown partner internal decisions or lost browser state. Treat it as a debugging aid and clearly label simulated outputs. Also consider selection bias: the events visible in logs may exclude the very failures under investigation. Retries can double-count requests, asynchronous reporting can shift events between days, and differences in partner contracts can make similar-looking fields non-equivalent. These are not reasons to abandon measurement; they are reasons to describe its limits.

Do not optimize an isolated metric without a decision rule. The safer sequence is to define the objective, establish a baseline, run a bounded experiment, review counter-metrics and record whether the result generalizes. A production change should have an owner and rollback plan. If the expected behavior is privacy-driven suppression or deliberate demand filtering, fixing the apparent loss would be a mistake.

8. Questions for a design review

• Which observable event proves the hypothesis, and which event would disprove it?
• Are all compared metrics defined over the same opportunities and time window?
• Can the scenario be reproduced with sanitized or synthetic data?
• What legitimate product or privacy decision could explain the symptom?
• Which downstream guardrail could deteriorate if the proposed fix succeeds?
• Who owns verification, monitoring and rollback after deployment?

9. What to publish as evidence

A credible engineering article should publish a clearly labeled synthetic example, its assumptions, a reproducible calculation or test, a decision table and known limitations. It should not imply that invented numbers were observed at a real publisher. For internal operational reports, link the versioned configuration and trace schema instead of exposing sensitive payloads. Readers should be able to apply the method to their own environment while understanding what cannot be concluded from the example alone.

10. References and further reading

The workflow and case study on this page are original educational material. Relevant underlying standards include the IAB Tech Lab OpenRTB 2.6 specification, ads.txt documentation and sellers.json guidance. Consult the actual standard and the specific partner contract before treating a field as mandatory or a discrepancy as a defect.