« Phase 13 · Warmup · Track Overview

Core Contributor — Working on the Engines Themselves

What it takes to contribute to Terraform, Kubernetes, Envoy, Gatekeeper or the platform modules your bank builds. Read this if you want to understand the systems rather than configure them.

Table of Contents

• 1. Why read the engines
• 2. Terraform: core and providers
• 3. Writing a provider
• 4. Kubernetes: the controller pattern
• 5. The scheduler
• 6. Device plugins
• 7. Envoy: the filter chain
• 8. Gatekeeper and Kyverno
• 9. Building platform modules
• 10. Testing infrastructure code
• 11. Contributing

1. Why read the engines

Because the abstractions leak in ways only the source explains. "Why did Terraform replace this?" is answered by a ForceNew: true in a provider's schema, and nothing else will tell you.

Because the failure modes are architectural. A Kubernetes controller's behaviour under a partition, or a webhook's behaviour when its own pods are down, follows from the design — and both are things you will meet at 3 a.m.

2. Terraform: core and providers

The split is the architecture, and it explains most of Terraform's behaviour:

   terraform CLI  ─── gRPC ───►  provider plugin (a separate process)
        │                              │
   graph, state, plan            the cloud's API

Core knows nothing about clouds. It walks a graph, calls PlanResourceChange and ApplyResourceChange over gRPC, and manages state. Providers know one API each.

Which explains three things people find surprising:

A provider is a separate process. Hence the plugin protocol, hence terraform init downloading binaries, and hence a provider crash appearing as plugin did not respond.

Core computes the plan; the provider refines it. Core diffs config against state, then asks the provider — which is where ForceNew and computed attributes are applied. So a plan is a negotiation, not a pure diff.

"Known after apply" is a value core does not have yet. It propagates through the graph, and a resource whose count depends on one cannot be planned — which is why for_each over a computed value fails and is the most common "why can't Terraform plan this?" question.

Worth reading in hashicorp/terraform:

The graph transformers are the interesting part: the graph is built by applying a sequence of transformers (attach state, attach schema, add dependencies, prune, order). Reading that list is the fastest way to understand what a plan is actually doing.

3. Writing a provider

Not exotic — banks write internal providers for internal services, and the schema is where the semantics live:

"address_prefixes": {
    Type:     schema.TypeList,
    Required: true,
    ForceNew: true,          // ← THIS is why the plan says "forces replacement"
    Elem:     &schema.Schema{Type: schema.TypeString},
},
"tags": {
    Type:     schema.TypeMap,
    Optional: true,
    // no ForceNew → updated in place
},
"fqdn": {
    Type:     schema.TypeString,
    Computed: true,          // ← "known after apply"
},

Four flags carry most of the meaning:

The Sensitive caveat matters: it hides a value from the CLI's output and does nothing about state, which is why the state backend is a secrets store whether you treat it as one or not.

The four CRUD functions must be idempotent and drift-aware. In particular Read has to handle "gone" by clearing the id rather than erroring — that is how Terraform learns a resource was deleted out of band, and a provider that errors instead produces an apply nobody can get past.

The framework to use now is terraform-plugin-framework rather than the legacy SDKv2; it has a real type system and much better handling of null versus unknown, which is where SDKv2 providers accumulate their bugs.

4. Kubernetes: the controller pattern

Every controller is the same loop, and once you have written one the whole ecosystem reads differently:

for {
    obj := workqueue.Get()          // an item, deduplicated and rate-limited
    desired := obj.Spec
    actual := observeReality(obj)
    if diff := compare(desired, actual); diff != nil {
        act(diff)
    }
    updateStatus(obj)
}

The machinery around it:

Three properties that follow, and each is a real operational fact:

Level-triggered, not edge-triggered. The reconciler reads current state and acts; it does not process events. So a missed event is harmless — the next resync picks it up. This is the single most important design property of Kubernetes, and it is why the system is robust to controller restarts.

Reconcile must be idempotent. It will be called repeatedly for the same object, including with no change. A reconciler with a side effect that is not idempotent produces duplicates on every resync.

Status is the controller's output. spec is what the user wants; status is what the controller observed. A controller that writes to spec is fighting the user, and the confusion that produces is hard to unpick.

Worth reading: kubernetes-sigs/controller-runtime before the main repo. It is the distilled version, and pkg/reconcile and pkg/manager are an afternoon.

5. The scheduler

   pod (unscheduled)
        │
   PreFilter ──► Filter ──► PostFilter ──► PreScore ──► Score ──► Reserve
        │                                                            │
        └───────────────────► Permit ──► PreBind ──► Bind ◄──────────┘

The scheduling framework makes each stage a plugin extension point, which is how Volcano and friends are built. The stages that matter here:

Filter — feasibility. NodeResourcesFit checks requests; TaintToleration checks taints; NodeAffinity checks selectors. This is where a GPU pod is excluded from CPU nodes.

Score — ranking. NodeResourcesBalancedAllocation spreads, NodeResourcesFit with MostAllocated packs. For GPUs you want packing, not spreading, and the default is spreading — which is a one-line config change with a large effect on fragmentation.

Permit — the extension point gang scheduling uses. A plugin can hold a pod in a waiting state until its whole group is schedulable, then admit them together. That is Volcano's mechanism, and it is why gang scheduling is a plugin rather than a patch.

Two things worth internalizing:

One pod at a time is the default, by design. The scheduler's throughput comes from not coordinating. Gang scheduling reintroduces coordination, which is why it needs a different scheduler rather than a flag.

Preemption is PostFilter. When nothing fits, the scheduler looks for lower-priority pods to evict. A preemption plugin that does not understand gangs will evict one pod of a running gang, gain one GPU, and kill an entire job — which is worse than not preempting.

6. Device plugins

How a GPU becomes a schedulable resource:

   nvidia-device-plugin (a DaemonSet)
        │  gRPC over /var/lib/kubelet/device-plugins/
   kubelet ──► node.status.capacity["nvidia.com/gpu"] = 8
        │
   scheduler sees the resource and can place against it
        │
   at pod start: Allocate() returns device paths + env vars
        │
   nvidia-container-runtime injects them into the container

The plugin API is small — ListAndWatch (advertise, and update on change) and Allocate (assign to a container) — and reading it explains several behaviours:

GPUs are integers. The extended-resource model has no fractional quantity, which is why 0.5 is not expressible and why sharing needs MIG or time-slicing.

MIG instances are advertised as separate resources — nvidia.com/mig-1g.10gb — so a pod requests a profile, not a GPU. Which is why changing the profile changes the resource name and therefore every manifest that requested it.

Time-slicing is a plugin config, advertising one physical GPU N times. No isolation, and the scheduler cannot tell — which is why the noisy-neighbour failures it produces look like application bugs.

Worth reading: NVIDIA/k8s-device-plugin, and the GPU Operator for how the whole stack is assembled.

7. Envoy: the filter chain

   listener ──► filter chain ──► router ──► cluster ──► endpoints
                     │
                 http filters, IN ORDER:
                   jwt_authn ──► ext_authz ──► rbac ──► lua ──► router

Filters run in order, and the order is the security design. jwt_authn before ext_authz means the PDP receives validated claims rather than a raw token; reversing them means the PDP is authorizing on unauthenticated input.

Two mechanisms worth knowing:

xDS is the configuration protocol — LDS (listeners), RDS (routes), CDS (clusters), EDS (endpoints), SDS (secrets). A control plane (Istiod) streams updates; Envoy applies them without a restart. The S matters: it is a gRPC stream, so the control plane pushes rather than the proxy polling, which is why mesh config changes take effect in seconds.

The threading model. Envoy is single-threaded per worker with no cross-thread locking on the hot path; config updates are applied via thread-local storage with an eventual-consistency window of milliseconds. Which is why a filter that blocks is catastrophic — it blocks a whole worker — and why ext_authz is asynchronous.

Worth reading in envoyproxy/envoy: source/extensions/filters/http/ext_authz/ (the PEP hook), source/common/router/ (retries, and the retry_budget implementation), source/common/upstream/outlier_detection_impl.cc.

8. Gatekeeper and Kyverno

Gatekeeper is OPA as an admission controller. A ConstraintTemplate defines a parameterized Rego policy; a Constraint instantiates it. The separation is the good idea: platform engineers write templates, teams instantiate them with their own parameters.

Kyverno is Kubernetes-native — policies are YAML that looks like the resources it validates, which makes it far more approachable, and generate is genuinely useful (every new namespace gets a default-deny NetworkPolicy automatically).

The decision rule: Kyverno unless you need Rego's expressiveness or already run OPA. If you are already running OPA for the control plane (Phase 09), one policy language across both is worth something.

Both share the mechanism from the deep dive, and both have the same two sharp edges: failurePolicy (a Fail webhook whose own pods are down deadlocks the cluster) and audit mode (admission is write-time only, so yesterday's workloads are invisible until you scan for them).

9. Building platform modules

The internal work that actually determines whether the controls hold.

A module is a paved road, not a wrapper. A module that exposes forty variables mirroring the provider adds nothing. A module that takes five and produces a compliant namespace — quota, network policy, service account with federated identity, default limits — is the thing teams will use because it is faster than doing it themselves.

Compose small modules; do not build one large one. A namespace module, a gpu-node-pool module, a private-endpoint module. One platform module that does everything cannot be adopted incrementally, which means it cannot be adopted.

Version and pin them.

module "namespace" {
  source  = "app.terraform.io/bank/namespace/azurerm"
  version = "~> 2.1"        # not a git branch
}

A module sourced from a branch changes under its consumers, which is Phase 12's mutable-tag problem in a different costume.

The private-endpoint module is the highest-value one you will write, because it is the one that encodes the six-step checklist from the warmup — endpoint, zone, record, VNet link, public access off, NSG. Every team that hand-writes it misses the link or the public-access setting, and the module is where that knowledge lives permanently.

Output what the next module needs, and nothing else. Outputs are an API; every one is a compatibility obligation.

10. Testing infrastructure code

Two that are usually missing.

OPA on the plan JSON. terraform show -json tfplan gives a machine-readable plan; conftest evaluates policy against it. This catches the thing CI linters cannot — a change that is dangerous in context, like a public_network_access_enabled flipping to true, or any resource marked forces replacement on a stateful type. Fifteen minutes to set up and it catches the incidents that matter.

Admission testing in kind. Spin up a cluster, install the policies, apply a manifest that should be denied, assert it was. Minutes, and it means a policy change is tested rather than hoped.

And the discipline that makes the whole thing tractable: the reachability analysis and the drift check are tests that run in production, on a schedule, with retained output. That retained output is what an examiner reads (Phase 15) — a year of daily runs with one finding and its ticket is a far stronger artifact than any point-in-time attestation.

11. Contributing

Terraform (hashicorp/terraform) — Go, large, BUSL now. OpenTofu is the MPL fork under the Linux Foundation and is the more welcoming target. Providers are the accessible entry point: terraform-provider-azurerm is very active and a new resource or a bug fix is a well-scoped contribution.

Kubernetes (kubernetes/kubernetes) — Go, enormous, SIG-structured. Start with a SIG that matches your interest (SIG-Scheduling, SIG-Node) and read its KEPs. controller-runtime and kubebuilder are far more approachable and are where most people actually contribute.

Volcano (volcano-sh/volcano) / Kueue (kubernetes-sigs/kueue) — Go, focused, and directly relevant to this phase. Gang scheduling and quota management are active areas, and the AI-workload use cases are exactly what maintainers want reports about.

Envoy (envoyproxy/envoy) — C++, high bar. Filters are the modular entry point. Read the ext_authz and outlier-detection implementations regardless.

Kyverno (kyverno/kyverno) — Go, active, welcoming. New policies for the policy library are a genuinely useful first contribution, and the AI-workload policies in this phase — GPU taints, signed images, digest pinning — are not well covered.

NVIDIA GPU Operator (NVIDIA/gpu-operator) — Go, and the MIG management code is worth reading whether or not you contribute.

For all of them the useful preparation is the same: implement the mechanism yourself first — the lab is a small version of exactly that — then read theirs and find every place they differ. The differences are where the real engineering is.