d07 — Log Analytics Pipeline

A fully worked design. Ingest at volume, index selectively, query interactively. The tension is that those three want opposite things, and the design is where you resolve it.

Table of Contents

• The Prompt
• 1. Requirements and Scope
• 2. Scale Numbers
• 3. API Surface
• 4. Data Model
• 5. High-Level Architecture
• 6. Deep Dive A: Ingest Backpressure Without Losing Logs
• 7. Deep Dive B: Index Cost vs Query Cost
• 8. Failure and Recovery
• 9. Bottlenecks and Evolution
• 10. Tradeoffs Explicitly Rejected
• The Hostile Critique
• The Revision
• References

The Prompt

"Design a logging platform. Every service ships logs to it, engineers search them during incidents, and we build dashboards on top. It costs us a fortune and it's slow exactly when we need it — during an outage."

"Slow exactly when we need it" is the design constraint, not a complaint about performance. An outage produces a log spike (error paths log more) at the same moment engineers start querying hardest. Ingest and query contend, and the naive design lets ingest win, so the platform is unavailable precisely during the incident it exists to help with.

That coupling is the thing to break, and naming it in the first two minutes is the strongest opening.

1. Requirements and Scope

Clarifying questions asked

"During an outage, is it more important to accept every log line or to keep search fast?" The fulcrum. Assumed search stays fast; ingest may shed low-severity logs — because a platform that cannot be queried during an incident has no value, while losing some DEBUG lines costs almost nothing.

"What's the query mix?" Assumed: 95% are last-hour, single-service, filtered searches (incident response); 5% are long-range aggregations (dashboards, trend analysis). Those want completely different storage, and treating them the same is the usual error.

"Retention?" Assumed 7 days hot (searchable in seconds), 30 days warm (searchable in minutes), 1 year cold (archive, restore on request).

"Structured or free text?" Assumed mostly structured (JSON) with a free-text message. That matters: structured fields can be indexed cheaply; free text cannot.

Functional

1. Ingest structured log events from thousands of hosts.
2. Search by time range + field filters + free-text, returning results in seconds.
3. Aggregate (count, percentile, group-by) over time ranges.
4. Tail live logs for a service.
5. Alert on query results.

Non-functional

Explicitly out of scope

• Metrics and traces — different shapes, different stores. (Metrics are numeric time series with low cardinality; conflating them with logs is how you get an unaffordable system.)
• Log generation and client libraries beyond the shipping contract.
• Access control beyond per-tenant isolation.

2. Scale Numbers

Volume. 5M events/s × 500 B = 2.5 GB/s = 216 TB/day raw. Compressed ~10× (logs are extremely repetitive) = 21 TB/day, 150 TB for 7 days hot.

That number is the design. At 216 TB/day raw, anything that touches every byte more than once is unaffordable, which rules out full inverted indexing of everything.

Index cost. A full inverted index over free text is typically 50–100% of the data size and costs more CPU to build than the data costs to store. Indexing everything: +150 TB and a large ingest CPU bill. Indexing only structured fields: ~5%. That is a 20× difference from one decision, and §7 is about where to draw the line.

Query. A last-hour search over one service: 1 hour = 900 GB compressed across all services; one service is ~1/500 of that = 1.8 GB. Scanning 1.8 GB at 1 GB/s/node across 10 nodes is 180 ms. So brute-force scan is viable for the common query — which is the insight that makes the cheap design work.

A 7-day aggregation over everything is 150 TB. At 10 GB/s aggregate that is 4 hours — not viable. Hence pre-aggregation (§7).

Burst. 20M/s for 15 minutes = 4× normal. Buffering it needs 20M × 500 B × 900 s = 9 TB of buffer. That is a lot of Kafka, and it is the argument for shedding rather than buffering everything.

Cardinality — the killer. If someone adds request_id as an indexed field, that is 5M distinct values/s. An inverted index on a unique-per-event field is larger than the data and provides no filtering benefit. High-cardinality fields must be excluded from indexing by policy, and this is the single most common way these systems become unaffordable.

3. API Surface

# Ingest — batched, compressed, per-agent
POST /ingest    {batch: [event...], agent_id, seq}   -> 202 {accepted, shed, shed_reason}

# Query
POST /search    {service, start, end,
                 filters: {level: "ERROR", region: "us-east"},
                 text: "connection refused",
                 limit, cursor}                       -> {events, cursor, scanned_bytes, partial}
POST /aggregate {service, start, end, filters,
                 group_by: ["status"], agg: "count",
                 interval: "1m"}                      -> {series, partial}
GET  /tail      ?service=x&filters=...                -> SSE stream

Four choices worth defending:

• 202 with {accepted, shed} — the agent learns exactly what happened. Silent shedding is the thing that destroys trust in a logging platform, because engineers cannot tell "no such log" from "we dropped it."
• scanned_bytes on every response — makes cost visible to the person who caused it. Engineers who can see that their query scanned 4 TB write better queries, and it is a one-field change.
• partial: true when a query hits its resource budget. An honest partial answer in 2 seconds beats a complete answer in 5 minutes during an incident — but only if the caller knows it is partial.
• seq per agent — lets the server detect gaps, so a client-side loss is visible rather than invisible.

4. Data Model

Storage: immutable time-partitioned segments in object storage

  s3://logs/{tenant}/{service}/{yyyy-mm-dd-hh}/{segment_id}.parquet

Segment layout (columnar):
  timestamp, level, service, host, region, trace_id, message, attrs<map>

  Per segment, a FOOTER holding:
    • min/max for every column          → partition + segment pruning
    • a Bloom filter per LOW-CARDINALITY indexed field
    • a sketch (HLL) of distinct values per field
    • row count, byte size

Metadata catalogue (a small database):
  segment_id → (tenant, service, hour, min_ts, max_ts, size, path, field_stats)

Why columnar (Parquet/ORC) rather than row-oriented: a search reads 3 of 20 columns, so columnar reads ~15% of the bytes. Compression is also far better — a level column of 5 distinct values compresses to nearly nothing, where a row format interleaves it with high-entropy message text.

Why Bloom filters per segment instead of a global inverted index: a Bloom filter is ~1.25 bytes/value at 1% error and answers "can this segment possibly contain X?" exactly in the negative. For a query filtering region=eu-west, most segments are eliminated with no I/O at all — the footer alone answers it. That gives ~90% of an index's benefit for ~2% of its cost. This is exactly the LSM Bloom-filter pattern.

Immutable segments, no updates. Logs are append-only by nature, which removes the entire compaction/vacuum problem and makes segments cacheable forever. Deletion happens at partition granularity (drop the hour), never per row.

5. High-Level Architecture

   agents (thousands)
        │  batched, compressed, with a local disk buffer
        ▼
 ┌──────────────────────┐
 │  Ingest gateway      │  authn · per-tenant rate limit
 │  PRIORITY SHEDDING   │  by severity when overloaded  ← DEEP DIVE A
 └──────────┬───────────┘
            ▼
 ┌──────────────────────┐
 │  Durable buffer      │  Kafka, partitioned by (tenant, service)
 │  short retention     │  the shock absorber; NOT the storage
 └──────────┬───────────┘
            ▼
 ┌──────────────────────┐
 │  Indexer workers     │  batch → columnar segment
 │                      │  build footer: min/max · Bloom · sketches
 └──────────┬───────────┘
            ▼
 ┌───────────────────────────────────────────────────────┐
 │  Object storage — immutable segments, hot/warm/cold    │
 └───────────────────────────────────────────────────────┘
            ▲                                    ▲
            │ prune via catalogue, then scan     │
 ┌──────────┴───────────┐            ┌───────────┴──────────┐
 │  Query workers       │            │  Pre-aggregation     │
 │  SEPARATE FLEET      │            │  rollups by minute   │
 └──────────────────────┘            └──────────────────────┘

The structural decision: query workers are a separate fleet from indexer workers, with separate autoscaling. That is what stops ingest starving query during an incident — the coupling named in the prompt. It costs some efficiency (two pools instead of one, per the bulkhead tradeoff) and it buys the property the whole system exists for.

The two hard parts — say these at minute 10:

1. Ingest backpressure: what to do when 20M/s arrives and you can absorb 5M/s.
2. Index cost vs query cost: what to index, given indexing everything is unaffordable and indexing nothing makes queries unbounded.

6. Deep Dive A: Ingest Backpressure Without Losing Logs

The situation

An outage starts. Error paths log more, retries log more, debug logging gets enabled. Volume goes 5M/s → 20M/s. You can absorb 5M/s.

Three bad options and one good one.

Bad option 1 — buffer everything

Kafka absorbs it. 15 minutes of 4× is 9 TB of buffer, and the lag becomes 45 minutes — so logs from the incident become searchable after the incident is over. You have preserved every byte and destroyed the platform's purpose.

Bad option 2 — reject uniformly

Shed 75% at random. Now every service's logs have holes, including the one service you are trying to debug. Random shedding destroys the signal proportionally everywhere, which is the worst possible distribution of the damage.

Bad option 3 — block the agents

Backpressure to the application. Applications block on logging → the logging platform takes down the services. This has happened to real systems and it is the most dangerous option, because it converts an observability problem into an availability problem.

Logging must never block the application. Say this explicitly; it is a principle, not a preference.

The design — priority shedding with agent-side buffering

Four levers, in order of engagement:

1. Agent-side buffering with bounded local disk. The agent buffers to local disk (say 1 GB), so a transient gateway problem loses nothing and the application never blocks. When the buffer fills, the agent sheds — by severity, locally, where the most context exists.

2. Severity-based shedding at the gateway.

FATAL / ERROR   never shed
WARN            shed above 80% capacity
INFO            shed above 60%
DEBUG / TRACE   shed above 40%

By the time you are at 4× capacity you are dropping DEBUG and INFO and keeping every ERROR — which is exactly what you want during an incident. The information density of the retained logs actually goes up under load.

3. Per-tenant fair shedding. Within a severity, shed proportionally to each tenant's share of the overload, so one runaway service cannot consume everyone's budget. A tenant at 10× its baseline is shed far more aggressively than one at 1×.

4. Sampling instead of dropping, for high-volume repeats. Identical log lines (same service + template + level) get sampled at 1-in-N with a count, rather than dropped. "connection refused" × 48,213 carries almost all the information of 48,213 individual lines at 1/48,213 the cost. This is tail sampling applied to logs, and for the outage case it is the highest-leverage lever of the four.

And make the shedding visible

• POST /ingest returns {accepted, shed, shed_reason} — the agent knows.
• A shed counter per service per severity, queryable like any log field, so a search UI can show "1.2M DEBUG lines shed in this window" alongside results.
• A gap marker injected into the stream, so a reader sees an explicit hole rather than inferring one.

Silent loss is the thing that destroys trust in a logging platform. An engineer who cannot distinguish "this did not happen" from "we dropped it" will stop believing the tool, and then the platform has failed regardless of its uptime.

7. Deep Dive B: Index Cost vs Query Cost

The central economic tradeoff, and the one that determines whether the platform is affordable.

The two extremes

At 216 TB/day, full indexing is not affordable. But no index at all makes a 7-day aggregation a 4-hour scan.

The design — a three-tier approach

Tier 1: partition pruning (free). Partition by (tenant, service, hour). A query for one service in the last hour touches 1/500 × 1/168 of the data — a 84,000× reduction before any I/O. This is the cheapest and largest win available and it comes from the directory layout.

Tier 2: segment-level Bloom filters and min/max (~2% overhead). In the footer of each segment, per low-cardinality indexed field. A query for region=eu-west eliminates segments with no data I/O — the footer alone. This is ~90% of an index's benefit for ~2% of the cost, and it is exactly what an LSM engine does.

The policy that keeps it affordable — and this is the part to emphasize:

Indexable:      level, service, host, region, status_code, env, ...
                → low cardinality (< ~10k distinct), high selectivity
NOT indexable:  request_id, trace_id, user_id, session_id, timestamps
                → high cardinality; the index would exceed the data
                  and filter nothing

High-cardinality indexing is the single most common way these systems become unaffordable. A Bloom filter on request_id at 5M distinct values/s is larger than the logs and eliminates almost no segments, because every segment contains some request IDs. The registry should refuse to index a field whose measured cardinality exceeds the threshold, rather than letting a well-meaning engineer add it.

Tier 3: pre-aggregation for the dashboard queries (small, fixed cost). Long-range aggregations are 5% of queries and 95% of the scan cost. So precompute them:

Continuously, from the ingest stream:
  count by (service, level, status, minute)
  p50/p95/p99 latency by (service, minute)     ← t-digest or DDSketch, mergeable
  distinct-count sketches by (service, minute) ← HLL, mergeable

A 7-day dashboard query then reads 10,080 pre-aggregated rows per series instead of scanning 150 TB. Rollups are a fixed small cost and they eliminate the query class that would otherwise dominate.

The mergeability requirement is the load-bearing detail: percentile and distinct-count sketches must be mergeable so minute rollups combine into hours and days without re-reading raw data. Naive percentiles cannot be merged; t-digest and DDSketch can. Saying that specifically is a strong signal.

What is left, and why it is acceptable

A free-text search for a rare string, over 7 days, with no field filters. Partition pruning does not help (all services), Bloom filters do not help (free text), rollups do not help (not an aggregation).

Accept that it is a scan, and make it honest:

• Show scanned_bytes and an estimated cost before running, with a confirmation for expensive queries.
• Stream partial results as segments complete, newest-first — during an incident, the most recent matching line is usually the answer, and you get it in seconds even if the full scan takes minutes.
• Enforce a per-query resource budget; on exceeding it, return partial: true with what was found and a cursor to continue.
• Nudge toward adding a service or time filter, which restores the 84,000× pruning.

The honest framing: we optimize for the 95% of queries that are recent and filtered, we make the 5% that are aggregations cheap with rollups, and we make the remaining rare case possible and visibly expensive rather than fast. Pretending you can make everything fast at 216 TB/day is what produces the "costs a fortune" complaint in the prompt.

8. Failure and Recovery

The clock-skew row matters more than it looks. Partitioning by event time means one host with a clock a week in the future creates a partition a week ahead, and one a week behind rewrites a closed partition. Partition by ingest time; keep event time as a queryable field. Then a skewed host produces confusing query results for that host only, rather than corrupting the storage layout for everyone.

Deliberately accepted: during a 4× spike we lose DEBUG and INFO logs, counted but not recoverable. I accept that because retaining them would push searchability lag past the point where the platform is useful during the incident — and the incident is when it matters. FATAL and ERROR are never shed.

9. Bottlenecks and Evolution

1. Indexer CPU, immediately. Building columnar segments with footers is CPU-bound — compression and sketch construction dominate. Fix: cheaper compression for hot data (LZ4 or zstd level 1) and recompress to a higher level during the hot→warm transition, when it is off the critical path. Compression level is a knob with a 3× CPU range and a 1.5× size range; hot data should be at the cheap end.

2. Object storage request rate. A query touching 10,000 segments is 10,000 GETs. At S3's per-prefix rate limits this is a real constraint. Fixes: larger segments (target 128–512 MB — there is a genuine tension with ingest latency, since bigger segments take longer to fill), prefix sharding to spread the request load, and a local SSD cache of recent segments on query workers.

3. Catalogue growth. At 128 MB segments and 21 TB/day compressed, that is ~170k segments/day, 1.2M for 7 days. Manageable, but the catalogue query itself becomes the latency floor. Fix: hierarchical metadata — hour-level summaries that prune before descending to segments.

4. Query concurrency during an incident. Fifty engineers each running a broad search. Fix: a per-user query budget and result caching keyed on the query shape — during an incident many engineers run near-identical queries, so cache hit rates are unusually high exactly when it matters.

At 10× (2 PB/day): the design holds structurally, but the economics force tiered retention by severity — keep ERROR for 30 days and DEBUG for 6 hours — and the free-text scan case becomes genuinely unaffordable, so free-text search gets restricted to the hot tier only. That is a product decision, not a technical one, and it should be surfaced as such.

10. Tradeoffs Explicitly Rejected

Rejected: Elasticsearch-style full inverted indexing. Gives millisecond queries on anything. Rejected on cost: index build dominates ingest CPU, storage grows 50–100%, and a high-cardinality field can produce an index larger than the data. Flip condition: at 10–100× less volume, or where sub-second arbitrary search is the product rather than a support tool, full indexing is the right answer and this design is over-engineered.

Rejected: no index at all, pure scan. Cheapest ingest. Rejected because a 7-day aggregation becomes a 4-hour scan, which fails the dashboard use case entirely. Bloom filters and rollups buy that back for ~2% overhead.

Rejected: buffering the whole spike. Preserves every byte. Rejected because 45 minutes of lag makes the logs searchable after the incident ends — preserving the data while destroying its value. Flip condition: for an audit-log system where completeness is a compliance requirement and latency is not, buffer and accept the lag. That is a genuinely different product.

Rejected: uniform random shedding. Simple and fair-looking. Rejected because it damages every service's logs proportionally, including the one being debugged. Severity-based shedding concentrates the loss where it costs least.

Rejected: blocking the application on log writes. Guarantees no loss. Rejected absolutely — it makes the logging platform able to take down every service that uses it. Async, bounded local buffer, agent-side shedding.

Rejected: partitioning by event time. More intuitive for queries. Rejected because a clock-skewed host writes into future or closed partitions, corrupting the layout for everyone. Ingest-time partitioning contains the damage to that host's own query results.

Rejected: mutable segments with updates. Rejected because logs are append-only by nature, and immutability removes compaction, enables permanent caching, and makes retention a partition drop instead of a delete. Nothing is gained by allowing updates.

The Hostile Critique

C1. "Severity shedding keeps every ERROR. During an outage, ERROR volume is what goes up 20×. So the class you promised never to shed is exactly the class that overwhelms you. What actually happens at 20M/s of pure ERROR?"

C2. "You partition by ingest time and store event time as a field. An engineer searches for what happened between 14:00 and 14:05. Your partitions are ingest-time. Walk me through which partitions you read and what you might miss."

C3. "Bloom filters per segment on low-cardinality fields. service has 500 values and every segment is single-service already — so that Bloom is useless. level has 5 values, so every segment contains every level and the Bloom always says yes. Name a field where your Bloom filter actually eliminates a segment."

C4. "Pre-aggregated rollups by minute, by service, by status. How many series is that, and what happens when someone adds a customer_id dimension to a dashboard?"

C5. "Fifty engineers, incident, identical queries, you cache by query shape. The first one takes four minutes and the other 49 wait on it or duplicate it. Which, and what does the cache do while it's being populated?"

C6. "Segments are immutable and retention is a partition drop. GDPR deletion request for one user's data, spread across every service's logs for a year. Go."

The Revision

R1 — Severity is not enough; add template sampling for ERROR (answers C1)

The critique is correct and it invalidates the simple version of the policy. During an outage ERROR is the growth, so "never shed ERROR" is a promise that cannot be kept at 20M/s.

Change: ERROR is never shed as a class, but identical ERRORs are collapsed.

• Compute a log template fingerprint at the agent — the message with variable parts (numbers, UUIDs, IPs) masked. "connection refused to 10.2.3.4:5432" → "connection refused to <ip>:<port>".
• Within a window, keep the first N occurrences of each template in full, plus a count and a small reservoir sample of the variable parts.
• 48,213 identical connection errors become one retained record with count=48213, three sampled instances, and the distinct set of ports seen.

Information retained: nearly all. Volume: 1/16,000. During an outage, error logs are overwhelmingly repetitive — that is what an outage is — so this is where the compression is.

And the ordering is now: template-collapse first (lossless in information, massive in volume), then severity shedding, then per-tenant fairness. I had the order wrong: collapsing should come before shedding, because it is nearly free in signal.

Cost: fingerprinting costs agent CPU (a regex pass per line), and a template that masks too aggressively merges genuinely distinct errors. Mitigated by keeping the sampled instances, so the detail is recoverable.

R2 — Query by event time, read by ingest time, with a skew bound (answers C2)

The critique identifies a real gap I hand-waved.

Change: the query planner translates an event-time range into an ingest-time range using a recorded skew bound.

• Every segment's footer already records min/max of both event_time and ingest_time.
• A query for event-time [14:00, 14:05] reads every segment whose event-time range overlaps — which the catalogue answers directly from footer stats, without reading data.
• The catalogue also tracks, per service, the observed distribution of ingest_time − event_time. The planner scans ingest-time partitions covering [14:00 − p99.9_skew, 14:05 + max_lag].
• Anything outside that is reported: "3 hosts have clock skew > 1 h; their events may be missing from this range" — named, not silently absent.

Cost: an event-time query reads somewhat more partitions than a pure ingest-time one. Bounded by the p99.9 skew, which is small for healthy fleets and observable when it is not — which turns a silent correctness problem into a visible operational one.

R3 — Index the fields that actually discriminate (answers C3)

The critique is exactly right and it exposes lazy thinking: I listed fields by cardinality without checking selectivity within a segment.

The correct criterion is not "low cardinality" but "low cardinality and clustered" — a field whose values are unevenly distributed across segments, so knowing the value eliminates segments.

Change: the useful indexed fields are:

And service is right to drop as an index, because the partition path already encodes it — the critique is correct that a Bloom would be pure waste. level likewise: every segment has every level, so it filters nothing and should be a columnar predicate pushdown instead (read the level column, ~1 byte/row compressed to almost nothing, and skip row groups).

And the exception worth making: trace_id is high cardinality but "find this one trace" is a critical query. Handle it with a separate, small trace-id → segment index built only for a short hot window (say 24 h). It is expensive per byte and tiny in total, and it turns an otherwise-impossible query into a point lookup. Blanket rules about cardinality are wrong; the question is always whether the query justifies the index.

R4 — Bound rollup cardinality explicitly (answers C4)

The critique names the classic metrics-system failure and I walked into it.

The arithmetic: 500 services × 5 levels × 40 status codes × 1,440 minutes/day = 144M series/day. Adding customer_id at 100k values multiplies by 100,000 → 14 trillion. That is a cardinality explosion and it destroys the rollup layer entirely.

Change:

1. Rollup dimensions are a fixed, registered whitelist. service, level, status_class (2xx/4xx/5xx, not the exact code — 40 values → 3), region. Total: 500 × 5 × 3 × 20 = 150k series, × 1,440 minutes = 216M rows/day, which is fine.
2. Adding a dimension requires a registration with a declared cardinality bound, and the system rejects it if the measured cardinality exceeds it.
3. High-cardinality dashboards fall back to sampled scans, not rollups — with the cost shown. A customer_id breakdown is a scan, and it should be visibly expensive rather than silently destroying the rollup layer.
4. Alarm on series growth, because this fails gradually and then suddenly.

Cost: a dashboard that needs a fine-grained breakdown is slower. That is the correct trade — the alternative is that one dashboard makes the aggregation layer unaffordable for everyone.

R5 — Single-flight the query, stream partial results (answers C5)

The critique identifies both a thundering herd and a poor incident experience.

Change, two mechanisms:

1. Single-flight on query shape. The first query installs a promise; the other 49 attach to it rather than duplicating or waiting blindly. One scan, 50 consumers. This is the cache-stampede fix applied to queries.
2. Stream partial results to all attached consumers as segments complete, newest-first. During an incident the newest matching line is usually the answer, so all 50 engineers see results in seconds, not after four minutes — even though the full scan is still running.

t=0.0s   engineer 1 starts; scan begins on the newest segments
t=0.3s   first matches stream to consumer 1
t=1.2s   engineers 2..50 attach; they immediately receive everything found so far
t=4m     scan completes; the result is cached for the next attach

And a cancel path: if every consumer disconnects, the scan is cancelled. Otherwise a broad query launched and abandoned burns four minutes of the cluster during an incident.

Cost: streaming partial results means the answer is not stable while it is arriving, so the UI must show progress and a "complete" state. Worth it — the alternative is 50 duplicate four-minute scans, which is a self-inflicted second outage.

R6 — Crypto-shredding, and be honest about the limits (answers C6)

The critique names the genuine conflict between immutability and erasure.

Change: the same mechanism as d06, adapted — but with an important honest caveat that logs make worse.

1. Structured fields identified as personal (user_id, email, ip) are encrypted at write with a per-subject key. Deletion destroys the key; the data remains and is unreadable.
2. Free-text message is the hard part. Personal data can appear anywhere in a log line, and it is not feasible to encrypt every message per-subject. Two honest options, and I would recommend both:
   • Prevent it at the source — agent-side redaction of patterns (emails, card numbers, tokens) before shipping, plus a lint rule in code review. This is the real fix, because personal data in free-text logs is a bug regardless of GDPR.
   • Accept a shorter retention for free text than for structured fields — 30 days rather than a year — so the erasure window is bounded by retention.
3. A deletion registry so a restored cold archive re-applies deletions on restore. Otherwise restoring a backup resurrects deleted data, which is a common and serious gap.

The honest statement: structured personal data is erasable on demand; free-text is handled by redaction at source plus bounded retention. Claiming full erasure of arbitrary free-text logs would be a lie, and the design should say so to the people relying on it rather than discover it during an audit.

References

• ../WARMUP.md#49-delivery-semantics · #410-load-control
• d05-load-shedding.md — the shedding mechanics used in §6
• ../../coding/WARMUP.md#chapter-9-deduplication-and-probabilistic-structures — Bloom filter sizing and the error direction
• ../../coding/harness/problems/text_index/ — segments, tombstones and merging as a timed problem
• Facebook. Scuba: Diving into Data at Facebook. VLDB 2013 — the brute-force-scan-is-fine argument
• Grafana. Loki design — index the labels, not the log body. The clearest published statement of §7
• Uber. CLP: Compressed Log Processor. — log template extraction, the mechanism behind R1
• Dunning & Ertl. Computing Extremely Accurate Quantiles Using t-Digests. — mergeable percentiles
• Flajolet et al. HyperLogLog. AOFA 2007 — mergeable distinct counts
• Amazon Builders' Library. Instrumenting distributed systems for operational visibility.