That wasn't an outlier. It was one row of the audit:

The screenshot-rendering service I run makes a geographic promise — "show me this page as a visitor from Germany sees it" — by routing headless browsers through commercial proxy gateways, each labelled with a country by the provider. The exit country is the deliverable. And for several countries, the label was mostly false.

Registration is not geolocation

Nobody was lying, exactly. Providers report something close to the registration country of an IP block: who the block is allocated to, on paper. Websites — and GeoIP databases — see its geolocation: where traffic from that block actually surfaces. Those are different facts, and they diverge constantly. A block registered to a Hungarian company can be routed anywhere.

MaxMind's data model encodes exactly this distinction: a record carries two separate fields — country (geolocation, sometimes absent) and registered_country (registration). Read the wrong one and you rebuild the providers' mistake locally:

# country.iso_code only: registered_country is the registration signal
# and is exactly the answer this feature exists to stop trusting.
country = record.get("country")
if not isinstance(country, dict):
    return None
return normalize_country(country.get("iso_code")) or None

That comment lives in the code because it's a mistake you make exactly once, silently.

How do you even know where a proxy exits?

Swapping the provider's label for GeoLite2's answer just replaces one unaudited source with another — a single source gives you no error signal. So the design became a panel of five independent voters per proxy:

The through-the-proxy part is the point. A local database tells you where the world thinks an IP is. An oracle fetched through the proxy tells you what a website actually sees when that proxy connects — the only measurement that matters here.

The verdict function is small enough to show whole:

MINIMUM_VOTES = 3

def decide_verdict(votes, previous_verdict=None):
    """Strict majority (> 50%) of the votes cast, else None.

    Fewer than MINIMUM_VOTES cast is a failure to gather evidence rather than a
    disagreement, so the previous verdict is kept: a failed check must never be
    indistinguishable from "wrong country" (safety invariant 1). A genuine
    split returns None, which excludes the proxy from every country pool.
    """
    cast = [normalize_country(vote) for vote in votes if normalize_country(vote)]
    if len(cast) < MINIMUM_VOTES:
        return normalize_country(previous_verdict) or None

    tally = {}
    for vote in cast:
        tally[vote] = tally.get(vote, 0) + 1
    # sorted() only for deterministic order; a tied top count fails the majority test anyway
    winner, count = max(sorted(tally.items()), key=lambda item: item[1])
    if count * 2 > len(cast):
        return winner
    return None

Three rules hide in there:

1. Strict majority of the votes cast, not of five. 3–2 resolves; so does 2–1 when only three votes arrived.
2. Fewer than three votes cast is not a disagreement — it's a failure to gather evidence, so the previous verdict is kept. Without this, an egress hiccup at refresh time would reshuffle every country pool. A failed check must never be indistinguishable from "wrong country".
3. A genuine split returns None, and the proxy is quarantined out of every country pool. When five sources can't form a majority about one IP, assigning it anywhere is a guess.

The offline pair is a fast path: when both databases agree, the record resolves in microseconds and no request leaves the node. Only contested records probe the oracles, in batches of 50 proxies × 3 oracles. On the current 2,493-proxy inventory the two databases return different countries only 0.2% of the time; a cold first pass is still ~11% contested (270 proxies, 6 batches — the share includes every IP one database can't place at all), bounded at roughly 6 batches × the 15-second oracle timeout ≈ 90 seconds even if every probe runs long. Cheap enough to run on every catalogue refresh, every 25–35 minutes.

The bug I almost shipped

Here's the embarrassing part. A fresh record's verified_country field is seeded with the provider's label — deliberately, because that's what makes the kill switch honest: with COUNTRY_VERIFY_ENABLED=false a catalogue load keeps the provider's labels and behaves as if the feature didn't exist.

And I handed that same field to decide_verdict as the "previous verdict".

Follow the path: a proxy the panel has never examined, oracle probes fail, fewer than three votes → "keep the previous verdict" → the provider's label comes straight back out, gets a verification stamp written beside it, and gets counted as resolved in the status report. As the commit message puts it:

the 43-57% mislabelling this module exists to remove, republished as verified and reported as a success

The test suite — 218 tests at that point — was green the whole time. A whole-branch review pass caught it, not a failing test. The fix is a guard that refuses to treat a seed as a finding:

def stored_verdict(record):
    if not record.get("verification"):
        return None
    return normalize_country(record.get("verified_country")) or None

verification is stamped only by the panel itself, so its absence means: the value sitting in verified_country is a claim, not a verdict.

The oracle's answer arrives through enemy territory

Three of the five votes travel through the very proxy being judged, which makes the response body hostile input. aiohttp's response.text() and response.json() read to EOF — and EOF is the remote end's decision. ClientTimeout bounds how long a probe may run, not how much it may hand back, so an oracle body had no ceiling at all. With 150 probes in flight, that's a bad property.

async for chunk in response.content.iter_chunked(MAX_ORACLE_BODY_BYTES):
    total += len(chunk)
    if total > MAX_ORACLE_BODY_BYTES:
        return None
    chunks.append(chunk)
return b"".join(chunks)

The cap is 64 KiB — two orders of magnitude above the real answers, which are a few hundred bytes each. The detail I actually care about: it refuses rather than truncates. The first line of a Cloudflare trace parses perfectly well on its own, so a truncating cap would let a hostile proxy answer the panel with a prefix and be believed. An oversized body becomes an abstention — a withheld vote, identical to a timeout — never a wrong vote.

Two more traps from the same category:

• Cloudflare answers loc=XX for an address it can't place. XX has to abstain too: passed through as a vote, it can win a majority and become a country pool nobody asked for.
• 50 proxies × 3 oracles is 150 concurrent requests, and aiohttp's default connector limit is 100. Requests past the limit queue inside the same total timeout, time out from queueing alone, and silently withhold votes — so the connector is sized explicitly to PROBE_BATCH_SIZE * len(ORACLE_SPECS). No test would ever catch this one, because every load-path test stubs the probes.

Degrade to the label, never to an outage

The verifier improves labels; it must never take the service down. maxminddb is imported defensively, because the proxy layer imports the verifier at module scope and the service constructs its proxy catalogue object at import time — a missing wheel degrades to the old provider-label behaviour instead of refusing to boot. The two inert states log distinguishably (abridged here):

country verification disabled by COUNTRY_VERIFY_ENABLED; keeping provider labels
country verification skipped, no GeoIP database available; keeping provider labels

The first is the normal shipped-off state; the second is a provisioning alarm. Collapsing them into one line is how alarms stop being read.

How it was tested

The body cap was verified against a real aiohttp server, not just the unit-test fake: a body exactly at the cap accepted, one byte over refused, a dribbled chunked transfer assembled in full, an endless body refused in about a millisecond without draining it, and a gzip bomb refused after decompression.

The rollout was staged behind the flag: enable on one node with a second node as a control, and diff the two nodes' country pools — that diff is the measured effect of the feature. Every verification pass logs a per-source summary — resolved N of M records, K ambiguous — and the end-to-end check was gloriously low-tech: render a what-is-my-country page through the suspect pools and read the answer off the screenshot itself.

Takeaways

• An IP block's registration country and its geolocation are different facts that diverge constantly. Know which one each source reports — in MaxMind data, read country.iso_code, never registered_country.
• A failed measurement must be distinguishable from a bad result. "Couldn't check" quietly becoming "wrong" (or "fine") corrupts state on every transient outage.
• When the measurement channel is the thing you distrust, bound bytes as well as seconds. A timeout is not a size limit.
• Refuse hostile input outright instead of truncating it — truncation can turn an attacker-shaped body into a well-formed answer.
• Green tests measure your fakes. The two worst defects here — the label laundering and the unbounded oracle read — both sat exactly where the tests stubbed the network, and both were caught by review, not by a failing test.

That is not a fifteen-minute countdown. That is a unix timestamp. The "seconds remaining" on my cooldown key was, numerically, the current date:

>>> ttl = 1_784_764_800            # what TTL returned ≈ the unix time for that day
>>> ttl / (365.25 * 24 * 3600)     # years of "remaining TTL"
56.55578370978782
>>> 2026 + 56
2082

Every throttle and cooldown key in the keyspace was scheduled to expire around 2082. And one class of those keys had been quietly breaking a user-facing feature the whole time.

The setup

The screenshot-rendering service I run solo has the usual abuse controls on its signup endpoint: a per-IP request throttle, and a fifteen-minute cooldown on "here's your setup link" emails so an address can't be spammed with re-sends.

That state lives in Redis running in cluster mode. Django ships a perfectly good Redis cache backend, but it uses a non-cluster client — against a cluster it dies with MOVED redirect errors whenever the key it wants lives on another node of the cluster. So I had hand-rolled a small backend on top of BaseCache, using redis-py's RedisCluster client, which follows the redirects.

Here is the entire bug — the backend's TTL helper:

def _ttl(self, timeout):
    """Return ex= value for SET: None (no expiry) or a positive int, or 0 to delete."""
    timeout = self.get_backend_timeout(timeout)
    if timeout is None:
        return None
    return max(0, int(timeout))

It takes the caller's timeout — say, 900 seconds — runs it through Django's BaseCache.get_backend_timeout(), and hands the result to Redis as SET key value EX <n>.

One helper call. Two correct APIs. Incompatible units.

What get_backend_timeout actually returns

This is BaseCache.get_backend_timeout() in Django 5.0.6, minus its docstring:

def get_backend_timeout(self, timeout=DEFAULT_TIMEOUT):
    if timeout == DEFAULT_TIMEOUT:
        timeout = self.default_timeout
    elif timeout == 0:
        # ticket 21147 - avoid time.time() related precision issues
        timeout = -1
    return None if timeout is None else time.time() + timeout

time.time() + timeout. It does not return a duration. It returns an absolute unix deadline.

Why would a base class do that? Memcached. In the memcached wire protocol, an expiration value larger than 30 days is interpreted as an absolute unix timestamp, so clients traditionally send "expire at now + N" rather than "expire in N seconds". Django's base helper carries those memcached semantics.

Redis's SET ... EX, on the other hand, takes relative seconds. Feed it time.time() + 900 and Redis doesn't blink — 1.78 billion is a perfectly legal TTL. No error, no warning, no log line. Your fifteen-minute key is now a fifty-six-year key.

The giveaway is that the broken TTL isn't some random big number. It's now + 900, which for any human-scale timeout is numerically indistinguishable from now. A TTL that looks like a unix timestamp is a unix timestamp. It's the most greppable bug smell I've met in years.

The bitter footnote: django.core.cache.backends.redis — the built-in backend I couldn't use because of the cluster — knows all this. It overrides the helper:

# django/core/cache/backends/redis.py
def get_backend_timeout(self, timeout=DEFAULT_TIMEOUT):
    if timeout == DEFAULT_TIMEOUT:
        timeout = self.default_timeout
    # The key will be made persistent if None used as a timeout.
    # Non-positive values will cause the key to be deleted.
    return None if timeout is None else max(0, int(timeout))

Relative seconds. get_backend_timeout() is backend-specific by design, and I had copied the semantics of the wrong backend.

Why nobody noticed

Here's the cruel part: the rate limiter kept working perfectly.

The throttle keys hold a history list of request timestamps, and the read path prunes entries older than the window before counting them. Key expiry was never load-bearing for correctness there — an immortal key just meant Redis slowly accumulating garbage. The bug got to hide behind a feature that worked.

The email cooldown was the opposite. It's the textbook SET NX EX pattern:

return bool(self._redis.set(k, self._dumps(value), nx=True, ex=ex))

add() succeeds only if the key doesn't already exist, and the key is supposed to vanish after 900 seconds. With a 56-year TTL it never vanishes, so add() returns False forever, which means: any email address that had ever been sent a setup email could never be sent another one. Not after fifteen minutes. Not ever. First email went to spam and you clicked "resend"? Silence.

No exception, no 500, nothing in the logs — the code was executing exactly as written. The only symptom was an absence: emails that should have gone out and didn't.

To muddy things further, one caller — a set of usage counters — passed raw seconds straight to the client, bypassing _ttl() — so those keys expired on schedule, right next to keys that would outlive me.

That's the only reason I caught it: I'd shipped an unrelated throttle change that afternoon, was poking around the keyspace to check on it, and ran TTL out of idle curiosity.

The fix

Stop calling the base helper. Reimplement the built-in Redis backend's semantics locally: resolve the DEFAULT_TIMEOUT sentinel yourself, keep None as "persist", clamp non-positive to 0 (delete), pass everything else through as relative seconds.

def _ttl(self, timeout):
    """Return ex= value for SET: None (no expiry) or a positive int, or 0 to delete.

    Deliberately NOT BaseCache.get_backend_timeout(): that returns an ABSOLUTE
    unix deadline (time() + timeout, memcached wire semantics), while Redis
    EX/EXPIRE take RELATIVE seconds — feeding the deadline to ex= gave every
    key a ~56-year TTL (so 15-minute email-cooldown keys never expired and
    permanently blocked re-sends). Mirrors django.core.cache.backends.redis
    semantics instead: resolve the DEFAULT_TIMEOUT sentinel, keep None = no
    expiry, clamp non-positive to 0 = delete.
    """
    if timeout == DEFAULT_TIMEOUT:
        timeout = self.default_timeout
    if timeout is None:
        return None
    return max(0, int(timeout))

The logic delta is two lines. The docstring is longer than the code on purpose: the buggy version looked more idiomatic — it used the framework helper! — so the comment exists to stop a future me from "cleaning it up" back into the bug. The commit subject says it all: fix(cache): pass relative seconds to Redis EX, not an absolute deadline. (A coding agent pair-wrote the patch with me; the commit trailer credits it.)

Testing it — and cleaning up after it

The regression test asserts on the wire value: whatever lands in ex= must equal the seconds the caller asked for. A fake client records the kwargs:

def test_set_and_add_pass_relative_seconds_to_redis_ex(self):
    class FakeRedis:
        ...  # records the kwargs of every set() / expire() call

    fake = FakeRedis()
    backend = self._backend(fake)

    backend.set('web_signup_ip:203.0.113.7', [1.0], 3600)
    backend.add('web_signup_setup_email:a@b.c', True, 900)
    backend.touch('web_signup_ip:203.0.113.7', 3600)

    self.assertEqual(fake.set_calls[0].get('ex'), 3600)
    self.assertEqual(fake.set_calls[1].get('ex'), 900)
    self.assertEqual(fake.expire_calls, [3600])

Before the fix, that first assertion would have seen ex= ≈ 1,784,000,000 instead of 3600 — the test fails loudly on exactly the mistake production swallowed silently. A second test pins timeout=None → no ex= at all ("persist"), the other semantic I was now hand-rolling.

Then the part no code change can do. The fix only affects keys written after it deploys; the immortal keys were still sitting in production and, by definition, would never expire on their own. The commit message had to say it out loud: "existing immortal keys are cleaned up operationally." After deploying that evening, I swept 18 permanent keys out of the production keyspace by hand — throttle buckets, per-IP counters, and seven email-cooldown keys. Seven addresses that had been silently un-mailable could get their setup email again. Lifetime of the bug, from writing the backend to the sweep: one month to the day.

Takeaways

• Django cache timeout semantics are backend-specific, and get_backend_timeout() is a memcached-ism. If you hand-roll a cache backend, copy the timeout handling of the backend you target — Django's own Redis backend overrides the helper to return relative seconds — not the base class.
• A TTL that looks like a unix timestamp is a unix timestamp. Ten digits starting with 17 means someone added time() where a duration belonged. Cheap smoke test for any new cache layer: set a key through it, run TTL, sanity-check the number against the timeout you passed.
• Bugs that extend lifetimes fail silently. Nothing throws, nothing logs — every operation succeeds, just for fifty-six years instead of fifteen minutes. The only symptom is an absence. You find these by inspecting real data, not by watching error rates.
• Partial correctness is camouflage. The rate limiter pruned history on read, so it was immune to immortal keys and kept working — which hid the bug from the one code path (SET NX EX) that genuinely depended on expiry.
• Fixing the code doesn't fix the data the code already wrote. The 56-year keys survived the deploy and needed a manual sweep. A TTL bugfix ships in two parts: the patch, and the repair of everything the old code left behind.

[16:50:01.830Z] Startup log entry detected.
[16:50:01.830Z] FATAL: startup log entry not detected within 240s

Detected, and not detected. Zero milliseconds apart. The deploy aborted 34 seconds into a 240-second budget, on a container that was perfectly healthy — /health returned 200, and docker ps showed it still Up hours later.

The code responsible had shipped in November 2025 and ran clean for months. Then it fired once, on one node, and killed a healthy rollout.

What happened

The screenshot-rendering service I run has a fleet of downloader nodes behind it, deployed by a Jenkins job that rolls node by node. On each node it starts the new container and then gates on a startup line appearing in docker logs before moving on. Simple, boring, battle-tested.

Build #83 rolled the same image across the fleet, node by node. Five nodes passed. node3 printed the contradiction above and aborted, leaving the roll half-finished.

The investigation

A log that asserts X and not-X in the same millisecond is actually a gift: it tells you two different pieces of code evaluated the same question and disagreed. Here's the gate as it existed:

while [ ${elapsed} -lt ${startup_timeout} ]; do
  if docker logs "${CONTAINER_NAME}" 2>&1 | grep -qi 'Application startup complete'; then
    echo 'Startup log entry detected.'
    break
  fi
  sleep ${poll_interval}
  elapsed=$(( elapsed + poll_interval ))
done

if ! docker logs "${CONTAINER_NAME}" 2>&1 | grep -qi 'Application startup complete'; then
  echo "FATAL: startup log entry not detected within ${startup_timeout}s"
  docker logs "${CONTAINER_NAME}" || true
  exit 1
fi

The loop found the line, printed "detected", and broke out. Then the post-loop guard re-ran the exact same pipeline — and it "failed". Same container, same logs, milliseconds apart. How does an identical command return a different answer?

The script runs under set -euo pipefail. That's the whole story.

The mechanism

If you've never been bitten by this, here's the anatomy, from the bottom up.

A pipeline producer | consumer runs both processes concurrently, connected by a kernel pipe with a finite buffer — 64 KiB on Linux. When the consumer exits, the read end closes. If the producer then tries to write, the kernel sends it SIGPIPE, and it dies with status 128 + 13 = 141.

grep -q is designed to exit at the very first match — that's its optimization. So in docker logs | grep -q PATTERN, the moment the pattern flows past, grep is gone, and docker logs — still streaming the rest of the log — gets killed mid-write.

Normally you'd never notice, because a pipeline's exit status is the last command's status, and grep exited 0. But set -o pipefail changes the rule: the pipeline reports the rightmost non-zero status of any member. The producer died with 141, so the whole pipeline is 141. The if reads that as false.

"Pattern found" gets reported as "pattern not found."

And here's why it's a months-latent race rather than an everyday failure: in practice SIGPIPE bites only when the producer still has more than a pipe buffer's worth of output to write after the match. A freshly started container has a handful of log lines — they fit in 64 KiB, the producer finishes and exits cleanly, everyone's happy. The bug only bites when the log has grown chatty enough behind the marker. Whether it fires depends purely on how much output docker logs still has buffered when grep exits. That's why the same image passed on five nodes and died on node3.

There was a second defect stacked on top. Even on runs where the loop's probe won the race, the post-loop if ! re-ran the same racy pipeline and was allowed to overturn the successful result. The loop said "detected", broke out, and the redundant re-probe got a fresh chance to lose the race — which it took, in the same millisecond. That's the contradiction in the log.

The fix

Two changes (commit fd3e862, pair-written with a coding agent):

1. Record detection in a flag; make the failure gate test the flag instead of re-probing.
2. Make the one remaining probe SIGPIPE-proof: plain grep with stdout to /dev/null. Without -q, grep reads stdin to EOF, so the producer always finishes writing and never gets signalled.

But I have to admit: my fix had its own bug, which I caught 36 minutes later the same evening (commit 7cea8e3). The loop probes at t = 0, 10, …, 230 — that's 24 probes — and then exits with elapsed=240. The "redundant" re-probe I deleted was accidentally a real 25th detection opportunity at t ≈ 240s. Deleting it silently shrank the advertised 240-second window: a container whose marker landed in the last ~10 seconds would now fail a deploy that previously succeeded — precisely the slow-node case the gate exists for. The final shape restores that look as a promote-only probe (comments abridged):

startup_timeout=240
poll_interval=10
elapsed=0
startup_detected=0

# Do NOT "optimise" the grep below back to `grep -q`. This script runs under
# `set -o pipefail`: `grep -q` exits at the very first match, so the still-writing
# `docker logs` producer is killed by SIGPIPE (status 141), and pipefail then
# reports 141 as the whole pipeline's status -- which reads as "pattern not found"
# even though the pattern WAS found.
startup_log_ready() {
  docker logs "${CONTAINER_NAME}" 2>&1 | grep -i 'Application startup complete' >/dev/null
}

while [ ${elapsed} -lt ${startup_timeout} ]; do
  if startup_log_ready; then
    startup_detected=1
    echo 'Startup log entry detected.'
    break
  fi
  sleep ${poll_interval}
  elapsed=$(( elapsed + poll_interval ))
done

# Final look after the last sleep, so the full budget is honoured. Guarded by the
# flag: it can only promote 0 -> 1 and can NEVER overturn a detection.
if [ ${startup_detected} -ne 1 ] && startup_log_ready; then
  startup_detected=1
  echo 'Startup log entry detected.'
fi

if [ ${startup_detected} -ne 1 ]; then
  echo "FATAL: startup log entry not detected within ${startup_timeout}s"
  docker logs "${CONTAINER_NAME}" || true
  exit 1
fi

Once I knew the shape, I went looking for it elsewhere — and found the health gate in the same Jenkinsfile had the identical off-by-one: a 90-second budget probed at t = 0, 5, …, 85, so the last five seconds were advertised but never checked. The tempting one-character fix — relax the loop to -le — is actively wrong there, because the failure gate inferred failure from arithmetic: [ ${health_elapsed} -ge ${health_timeout} ]. With a probe landing at t = 90, success leaves health_elapsed == health_timeout and the gate would abort a container that had just answered. The deeper problem was inferring an outcome from elapsed-time math instead of recording it. It now sets an explicit health_ok flag, same promote-only final probe.

How I tested it

The best part of this bug is that it's reproducible in pure bash — no Docker, no Jenkins. A producer that emits the marker and then keeps writing well past the pipe buffer is all you need. This is the harness string from the pytest suite — everything inside the triple quotes is plain bash, and the test substitutes __PROBE__ with the probe form under test:

_STARTUP_GATE_PROBE_SCRIPT = """set -euo pipefail
emit_startup_logs() {
  echo 'Application startup complete'
  i=0
  while [ "$i" -lt 20000 ]; do
    echo 'INFO:     uvicorn trailing log line that keeps the producer writing'
    i=$((i + 1))
  done
}
if emit_startup_logs | __PROBE__; then
  echo DETECTED
else
  echo "MISSED status=$?"
fi
"""

Twenty thousand lines is ~1.3 MB queued behind a 64 KiB pipe buffer, so the producer is always still blocked in write() when a quiet grep exits — the production race, made deterministic. The test extracts the probe the Jenkinsfile actually ships and runs it against this producer: the shipped form prints DETECTED; substituting the legacy grep -qi form reproduces the exact production symptom, MISSED status=141.

A second layer extracts the entire gate block from the shipped Jenkinsfile with a regex and runs it under real bash, with counting docker and no-op sleep stubs on PATH so the 240s budget is instant. Marker on probe 1: exit 0, exactly one probe — behavioural proof the guard short-circuits and can never overturn a detection. Marker never: exit 1, FATAL, log dump. Marker only on probe 25: exit 0 — the regression guard for the window I clipped, and it fails against my own first fix. All of it fails against the original Jenkinsfile.

Takeaways

• Under set -o pipefail, producer | grep -q pattern is a footgun: "found" can come back as exit 141. If you need the pipeline's verdict, use plain grep pattern >/dev/null so the consumer drains stdin to EOF.
• Small outputs hide the bug. It reliably fires only when more than a pipe buffer (64 KiB) remains unwritten after the first match — so it passes review, passes staging, and detonates months later on whichever node happens to be chattiest.
• Record a success in a flag the moment you observe it. A re-probe is not idempotent when there's a race inside it; a second look must never be allowed to overturn a first positive.
• while [ elapsed -lt timeout ] polling loops stop one interval short of the advertised budget, and inferring the outcome from elapsed >= timeout arithmetic instead of a recorded result is how that lie becomes an abort. Count your probes.
• When the correct code looks like it's begging to be "optimised", leave the comment that explains why it must not be — and a test that goes red if someone does it anyway.