AdTech Engineering Research & Labs · OpenRTB field consistency monitoring

A Bidstream Data Quality Observatory

Schema-valid JSON can still be operationally unusable. A bid request may contain technically optional fields that a particular integration needs for routing, creative eligibility or supply-chain review. A quality observatory measures both protocol conformance and partner-specific completeness without conflating them.

Scope: This is an independently written engineering method with a synthetic teaching scenario. It is not a report of customer data, a production incident or measured industry performance.

1. Why this investigation matters

Schema-valid JSON can still be operationally unusable. A bid request may contain technically optional fields that a particular integration needs for routing, creative eligibility or supply-chain review. A quality observatory measures both protocol conformance and partner-specific completeness without conflating them. The key mistake is to begin with an attractive dashboard number before deciding what was actually measured. A useful investigation states its population, event definition, observation window and expected invariant. It then distinguishes a genuine system failure from a difference in counting or policy. This matters because an optimization that improves one stage may damage the next stage and leave publishers worse off.

The method below deliberately separates observation (what was recorded), interpretation (what might explain it), and action (what to change and how to verify it). Each finding should have a counterexample: what evidence would make the current explanation wrong? Writing down that question prevents teams from treating correlation as proof.

2. A system model that can be inspected

The investigation boundary is: ingest sampled requests → redact sensitive fields → validate schema → apply integration rules → aggregate defects → investigate drift. Each transition has its own event semantics, ownership and potential failure modes. A missing downstream event does not establish which upstream component failed; the same symptom can arise from suppression, transport loss, sampling or delayed reporting.

Build a small data dictionary before querying production telemetry. For each stage, record whether the count is based on browser events, server acceptance, exchange processing or billing notices. Use one time zone and state whether late-arriving records are revised. When a field is absent, record unknown rather than silently assuming false. A dashboard that mixes incompatible denominators can produce an impressive but meaningless percentage.

3. Worked synthetic case study

A synthetic sample contains 10,000 requests: 120 fail JSON parsing, 430 omit a partner-required placement field, and 90 contain an unexpected currency value. The groups must be deduplicated before computing a combined affected-request rate.

Start by drawing the event boundaries and selecting one narrowly defined slice: a single placement, a known configuration version and a bounded time window. Calculate the difference between adjacent stages, then check whether the populations actually refer to the same opportunities. Use synthetic IDs such as trace-001 in examples; do not publish live bid requests or user identifiers. A good incident write-up includes both the expected path and a path that was intentionally suppressed for a legitimate reason.

For a second pass, challenge the initial explanation. If the apparent loss disappears when late events are included, the problem is reporting timeliness rather than auction execution. If only one browser family is affected, investigate client behavior before adjusting exchange-wide settings. If the change aligns with a configuration edit, test a controlled rollback in staging. None of these checks should be skipped merely because one chart appears persuasive.

4. Investigation procedure

1. Maintain separate checks for required-by-standard and required-by-contract. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
2. Monitor null, missing and unexpected values. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
3. Record schema version. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
4. Use sampling weights. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
5. Detect sudden shifts by partner and placement. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.
6. Quarantine unsafe samples.. For this investigation, document the input evidence, the responsible system, the acceptance condition and the output artifact. Compare the result with an independent source where possible. If the check fails, isolate the smallest reproducible example and capture the configuration version before making changes.

Use an evidence log with columns for hypothesis, observable signal, query or reproduction step, result, confidence and next action. Preserve the raw definition of every metric used in a screenshot. For code or configuration changes, include a test that would have failed before the fix and passed afterward. If the issue cannot be reproduced, mark it unresolved and document the missing instrumentation rather than inventing a root cause.

5. A practical worksheet for operators

Copy the following worksheet into an issue tracker. It is designed to turn an interesting symptom into a falsifiable technical claim rather than a generic checklist.

6. How to validate the result

Validation should operate at three levels. First, verify that the instrumentation itself emits events at the documented boundaries. Second, reproduce the mechanism with controlled synthetic inputs, including an expected-success case and a deliberate failure case. Third, observe the relevant metric after the change while watching adjacent guardrails. A result is not reliable when only the headline number improves and downstream rendering, latency or billing is ignored.

Where a numerical estimate is useful, report the denominator explicitly. For example, a 5% decrease in timeouts among dispatched auctions is not the same as a 5% increase in revenue per eligible page view. Include confidence intervals or uncertainty bounds for sampled data, and avoid extrapolating from a small staging dataset to live traffic. If the sample is not representative, explain why the method is still useful and what remains unknown.

7. Failure modes and trade-offs

A missing optional field is not automatically a protocol violation. The observatory should describe the rule and business impact behind each finding. Also consider selection bias: the events visible in logs may exclude the very failures under investigation. Retries can double-count requests, asynchronous reporting can shift events between days, and differences in partner contracts can make similar-looking fields non-equivalent. These are not reasons to abandon measurement; they are reasons to describe its limits.

Do not optimize an isolated metric without a decision rule. The safer sequence is to define the objective, establish a baseline, run a bounded experiment, review counter-metrics and record whether the result generalizes. A production change should have an owner and rollback plan. If the expected behavior is privacy-driven suppression or deliberate demand filtering, fixing the apparent loss would be a mistake.

8. Questions for a design review

• Which observable event proves the hypothesis, and which event would disprove it?
• Are all compared metrics defined over the same opportunities and time window?
• Can the scenario be reproduced with sanitized or synthetic data?
• What legitimate product or privacy decision could explain the symptom?
• Which downstream guardrail could deteriorate if the proposed fix succeeds?
• Who owns verification, monitoring and rollback after deployment?

9. What to publish as evidence

A credible engineering article should publish a clearly labeled synthetic example, its assumptions, a reproducible calculation or test, a decision table and known limitations. It should not imply that invented numbers were observed at a real publisher. For internal operational reports, link the versioned configuration and trace schema instead of exposing sensitive payloads. Readers should be able to apply the method to their own environment while understanding what cannot be concluded from the example alone.

10. References and further reading

The workflow and case study on this page are original educational material. Relevant underlying standards include the IAB Tech Lab OpenRTB 2.6 specification, ads.txt documentation and sellers.json guidance. Consult the actual standard and the specific partner contract before treating a field as mandatory or a discrepancy as a defect.