Bid floors: test the rule, not just the final win rate.

Engineering field workbook: Floor Rule Experiment

Purpose. This exercise turns the guide into a reproducible investigation. The scenario is synthetic and is provided for learning, not as evidence of a production incident. Begin with a placement with a 1.20 USD CPM floor, multiple demand partners, and a mix of deal and open-auction traffic. The question to answer is not merely whether a component is present, but whether the relevant system behavior can be demonstrated with a traceable set of observations. Record the environment, configuration version, test date, and which participant owns each piece of evidence before drawing a conclusion.

Scenario: identify the first divergence

Suppose a floor increase raises average winning CPM but reduces fill and total revenue. Write down the expected behavior in one sentence and the observed behavior in another. Avoid diagnosing the entire system from a final dashboard number: many upstream failures collapse into the same visible symptom. Start at the earliest boundary that can be independently verified, then follow the transaction toward the user-visible outcome. Treat a missing log entry as an unknown until logging coverage is confirmed. A negative result is meaningful only if the test could have observed a positive one.

Evidence package to collect

For this investigation, capture eligible auctions, bid density, winning CPM, timeout rate, fill, and revenue per thousand eligible requests. Preserve raw samples and a normalized summary separately. Redact identifiers that could expose real users, partner credentials, proprietary bid details, or account-specific configuration. Label every timestamp with its time zone and specify whether it is client time, server receipt time, or reporting time. Keep an explicit column for missing information. This makes the exercise repeatable and prevents an apparent mismatch from being explained away by undocumented assumptions.

Controlled test procedure

First, freeze the test conditions: choose a known placement or request fixture, record configuration hashes, and ensure that unrelated releases are not running. Next, compare randomized cohorts and inspect bid distributions around each floor threshold before interpreting the result. Repeat the baseline at least twice to check whether the observation is stable. Change one variable at a time and record both the intended change and the actual request payload produced by the system. If a result changes, repeat the original baseline before attributing causality. Where production experiments could affect revenue or privacy, use synthetic fixtures or a non-billable test environment instead.

How to interpret conflicting signals

One useful distinction is between configured, requested, accepted, and counted. A setting can exist without appearing in a network request; a request can reach a partner without being accepted; and an accepted auction event can fail to become a billable impression. In this topic, higher average CPM does not imply higher total yield when fewer impressions monetize. Build a small decision table that identifies the system responsible for each transition and the observable proof needed to confirm it. When two dashboards disagree, compare their event definitions before comparing totals.

Negative controls and edge cases

Create one known-good fixture and at least three deliberate failure fixtures: a missing required input, a syntactically acceptable but semantically wrong value, and a delayed or partially unavailable dependency. For each fixture, specify the expected validation message, whether the system should fail open or fail closed, and whether any downstream report should change. Record what happens if the same request is retried, if a partner returns no response, and if the response arrives after the decision deadline. These checks reveal whether a diagnostic method is robust or only succeeds for a happy-path demonstration.

Decision record and next action

Conclude the investigation with four fields: confirmed observation, leading hypothesis, disconfirming evidence, and next smallest experiment. A useful outcome can be “insufficient evidence” when a partner-controlled log is unavailable. seasonality, traffic composition, deal priority, and floor enforcement location can bias comparisons. If the issue cannot be reproduced, document what was attempted and what instrumentation would make the next occurrence diagnosable. Do not claim a performance improvement until the outcome metric and comparison population are defined in advance.

Practical review questions

1. Which specific observation proves that the transaction reached the stage being investigated?
2. What alternative explanation would produce the same symptom without the suspected defect?
3. Which one-variable test could distinguish those explanations?
4. What data must be withheld or sanitized before sharing the example?
5. How would the team know the fix worked without relying on a single dashboard screenshot?

Takeaway: A high-quality technical investigation should be repeatable by another engineer. Keep the fixture, expected output, observed output, and limits of inference together so that a reader can test the claim rather than merely accept it.