« Phase 10 · Warmup · Track Overview

Core Contributor — Working on the Engines Themselves

What it takes to contribute to Temporal, resilience4j, Envoy, or the gateway your bank builds in-house. Read this if you want to understand the systems rather than configure them.

Table of Contents

• 1. Why read the engines
• 2. Temporal: deterministic replay
• 3. Temporal: the architecture
• 4. Workflow versioning
• 5. resilience4j and Polly
• 6. Envoy: resilience in the data plane
• 7. Adaptive concurrency
• 8. Transparency logs
• 9. Building an in-house gateway
• 10. Testing this class of system
• 11. Contributing

1. Why read the engines

Because the failure modes are only legible from the inside. "My Temporal workflow behaved differently on replay" is unactionable until you know that replay re-executes your workflow code against a recorded history. Once you know that, the bug is obvious and usually one line.

Because the constraints look arbitrary until they don't. "Don't use datetime.now() in workflow code" reads as a style rule. It is a correctness requirement, and understanding why prevents an entire class of incident.

2. Temporal: deterministic replay

The central idea, and it is genuinely elegant:

Workflow code is replayed from an event history. Side effects go through activities, whose results are recorded in that history. On replay, activity calls return their recorded results instead of executing.

So a worker that crashes mid-workflow can be replaced by another worker that re-runs the workflow function from the top — reaching the same state without repeating any effect — and then continues from where the history ends.

   HISTORY                          REPLAY
   ─────────────────────────        ────────────────────────────────────
   WorkflowExecutionStarted         run workflow(input)
   ActivityScheduled(place_hold)    → place_hold() ... returns recorded "H-1"
   ActivityCompleted("H-1")           (NOT executed)
   ActivityScheduled(open_case)     → open_case()  ... returns recorded "C-9"
   ActivityCompleted("C-9")           (NOT executed)
   ActivityScheduled(post_refund)   → post_refund() ... no recorded result
   ── end of history ──               → EXECUTES for real

Which forces the constraint everything else follows from: workflow code must be deterministic. The same input and the same history must produce the same sequence of commands. So:

The SDKs enforce much of this by sandboxing the workflow environment and patching the offenders — which is why the Python SDK's sandbox exists and why importing a module with side effects at workflow scope fails in a confusing way.

The replayer is the tool to know. Temporal ships a Replayer that runs a workflow's recorded history against your current code and fails if it diverges. Wire it into CI with histories captured from production and you catch determinism breaks before deploying them — which is the difference between finding a versioning bug in CI and finding it in a stuck workflow.

3. Temporal: the architecture

   ┌──────────┐  gRPC   ┌───────────────────────────────────────┐
   │  Client  │────────►│  Frontend                             │
   └──────────┘         │     │                                 │
                        │  History  ── event histories, timers  │
   ┌──────────┐         │  Matching ── task queues              │
   │  Worker  │◄───────►│  Worker   ── internal system workflows│
   │ (yours)  │  poll   │     │                                 │
   └──────────┘         │  Persistence (Cassandra/MySQL/Postgres)│
                        └───────────────────────────────────────┘

The property that matters: your workers hold the code and the data; the service holds the history. The service never sees your business logic and never executes it. That is what makes self-hosting and Temporal Cloud the same programming model, and it is also why the service can be multi-tenant without seeing tenant data.

Worth reading in temporalio/temporal:

4. Workflow versioning

The hardest real problem, and the one nobody anticipates.

A workflow runs for four hours. You deploy a change. The in-flight workflow now replays against new code with an old history — and if the new code would have made different calls, replay detects a non-determinism error and the workflow is stuck.

Three strategies:

Patching. workflow.patched("add-fraud-check") returns True for new executions and False when replaying a history that predates the patch. Precise, and it accumulates: a workflow with six patches is unreadable, so there is a deprecate_patch lifecycle to clean up once the old executions have drained.

Versioned task queues. Run old and new workers side by side; old executions stay on the old queue. Clean, and it costs you running two deployments until the long tail drains — which for a four-hour workflow is fine and for a thirty-day one is not.

Workflow-name versioning. PaymentInvestigationV2 as a new type. Simplest to reason about, worst for code duplication, and correct when the change is large enough that patching would be a nest of conditionals.

The judgment: patch for small changes, new task queue for a deploy-wide change, new workflow type for a redesign. And know the drain time of your longest workflow, because it is the lower bound on how long you maintain both.

5. resilience4j and Polly

Small, readable, and worth reading precisely because the ideas are simple and the details are not.

resilience4j (Java, functional): decorators compose.

Supplier<String> decorated = Decorators.ofSupplier(this::call)
    .withBulkhead(bulkhead)              // ← outermost: reject before spending anything
    .withCircuitBreaker(circuitBreaker)
    .withRetry(retry)                    // ← innermost: retries happen inside the breaker
    .decorate();

Order matters and this is the interview question. Bulkhead outermost, so a rejected call costs nothing. Retry innermost, so each attempt is recorded by the breaker — put retry outside the breaker and one logical call registers as one event no matter how many attempts failed, which makes the breaker blind to exactly the failures it exists to catch.

The internals worth reading: CircuitBreakerStateMachine (an atomic reference and a state object, so transitions are lock-free) and the ring-bit-set sliding window (failure counts in a bitset — O(1) updates and no per-event allocation).

Polly (.NET) covers the same ground with ResiliencePipeline, and its RateLimiter and Hedging strategies are worth knowing. Hedging — fire a second request if the first is slow, take whichever answers — is a latency tool that is catastrophic on non-idempotent operations, which makes it a good test of whether someone has internalized this phase.

6. Envoy: resilience in the data plane

Envoy moves timeouts, retries, breakers and bulkheads out of application code and into the sidecar, where they apply to every language uniformly.

The vocabulary maps only approximately, which trips people up:

So Envoy's circuit_breakers is a bulkhead and Envoy's circuit breaker is outlier_detection. Say "Envoy circuit breaker" in a design review and half the room hears a different thing.

Two properties worth knowing:

Outlier detection is per-host. It ejects the sick instance rather than the whole service, which is usually what you want and is strictly better than an application-side breaker that cannot tell one backend from another.

retry_budget is the important one. Per-request retry limits still allow 3× amplification across the fleet; a budget bounds it globally. Set it (10% is the default) — it is the single highest-value line in an Envoy retry config.

Source worth reading: source/common/upstream/outlier_detection_impl.cc and source/common/router/retry_state_impl.cc.

7. Adaptive concurrency

Fixed bulkhead limits are wrong twice: too low wastes capacity, too high fails to protect. Adaptive limiters infer the limit from observed latency.

Netflix concurrency-limits implements TCP-congestion-control algorithms for RPC:

• Vegas — infer the queue depth from RTT_min versus current RTT; increase the limit while the queue is small, decrease as it grows.
• Gradient2 — compare a short-window RTT to a long-window one; the ratio is the signal.
• AIMD — additive increase, multiplicative decrease. Crude, robust, and a fine default.

The elegance is that it needs no configuration: no threshold, no throughput floor, no tuning as traffic patterns change. The cost is that it is harder to reason about during an incident — "why did we reject that?" has a statistical answer rather than a configured one, and that is a genuine operational trade rather than a free win.

Envoy has this natively as adaptive_concurrency filter.

8. Transparency logs

The industrial version of §16 of the deep dive.

Certificate Transparency (RFC 6962) defines an append-only Merkle log with two proofs:

• Inclusion — "record X is in the tree with root R", in O(log n) hashes;
• Consistency — "the tree with root R₁ at size N is a prefix of the tree with root R₂ at size M", which is what proves nothing was retroactively inserted or deleted.

Consistency proofs are what a hash chain cannot give you cheaply, and they are the property an auditor actually wants: not "this record is intact" but "the log has only ever grown".

Sigstore Rekor (sigstore/rekor) is the most readable production implementation, built on Google's Trillian. Worth reading: pkg/api/entries.go (the append path) and Trillian's merkle/ package (the proof construction).

Trillian (google/trillian) is the general engine, and its key design decision is worth internalizing: the log is sequenced asynchronously. Entries are queued and batched into tree revisions, which means an entry is not immediately provable — there is an inclusion delay, typically seconds. For a bank's action log that is usually fine, and it must be stated, because "the record is written" and "the record is provably in the log" are different moments.

9. Building an in-house gateway

Often correct — the mediation logic is your control model. What it takes to be respectable:

The side-effect class is mandatory at registration. Not a default with a comment. register() raises.

One entry point. Every action goes through execute(). The moment there are two paths, one of them is less careful, and that is the one an incident will find.

Idempotency at the storage layer. A unique index or SET NX, never check-then-act.

Structured refusals. Distinct error codes for contract, conflict, in-flight, approval and circuit — because callers must branch on them, and a single 400 Bad Request makes a retry loop retry a 422 forever.

The audit record as a versioned schema. You will read seven-year-old records with today's code. Version the schema from record one, and never remove a field.

Property tests on the invariants:

# For any sequence of calls with the same key and hash, effect count == 1
# For any sequence, the audit chain verifies
# A saga's compensations are a suffix-reverse of its completed steps
# The gateway never executes when any refusal path was taken

Fault injection built in, driven by a header so it works in shared environments. If injecting a timeout requires a code change, nobody will test the timeout path.

Deterministic tests. Injected clock, injected downstream, derived ids. Every test in the lab runs in 0.1 s and none of them are flaky, and that is not an accident — it is the direct consequence of not calling time.time() or uuid4() anywhere in the production path.

10. Testing this class of system

Crash testing deserves the emphasis. Kill the process between reserving a key and storing the response; retry. Every team that does this discovers something, and it is usually a design gap rather than a bug — which is exactly the kind of thing you want to find deliberately on a Tuesday rather than accidentally during a quarter-end.

11. Contributing

Temporal (temporalio/temporal, and the SDKs) — Go server, SDKs in Go/Java/Python/TypeScript/.NET. The SDKs are the friendlier entry point: determinism checks, better error messages, testing utilities. Server contributions need an RFC for anything touching history semantics.

resilience4j (resilience4j/resilience4j) — Java, small, very readable. Good first contributions: metrics, Spring Boot integration, docs. Reading the CircuitBreakerStateMachine end to end is an afternoon and worth it.

Polly (App-vNext/Polly) — .NET, active, welcoming.

Envoy (envoyproxy/envoy) — C++, large, high bar. Filters are the modular entry point. Read the outlier-detection implementation regardless of whether you contribute; it is the clearest production breaker code in the open.

Sigstore Rekor (sigstore/rekor) — Go, moderate size, and the best way to understand transparency logs by reading rather than by paper.

For all of them the useful preparation is the same: implement the mechanism yourself first — the lab is a small version of exactly that — then read theirs and find every place they differ. The differences are where the real engineering is.