« Interview Prep · Track Overview

Architecture-Review Drills

Six designs, each with red flags planted. Read the design, find them, then check yourself. This is the other half of the interview: they will show you something and ask what you think.


Table of Contents


How to use this

Read each design cold. Give yourself five minutes and write down every problem you can see, then read the findings. Score yourself on how many you found and on how many you flagged that were not actually problems — a reviewer who flags everything is as unhelpful as one who flags nothing.

The findings are ordered by severity, which is also the order you should raise them in. A review that opens with a naming convention has already lost the room.

The standing checklist

Fifteen questions, assembled from every phase. Run them against any design, including your own.

  1. Is the SLO composed, or asserted?
  2. Is any dependency called "highly available" with no number and no source?
  3. Does every fallback fit the remaining latency budget?
  4. Does the retry policy mention idempotency and the side-effect class?
  5. Where does session state live?
  6. Where does tenant_id come from?
  7. Does every cache key start with the tenant?
  8. Is retrieval isolation structural, or a post-hoc filter?
  9. Is multi-agent delegation depth-bounded and cycle-checked?
  10. Is the delegation chain derived from a credential, or passed as a field?
  11. Is any caller-supplied URL used without an allow-list?
  12. Is there a control-plane call on the synchronous path, and what is its failure posture?
  13. Does each control emit an evidence artifact?
  14. Does the degradation ladder contain anything that is actually a control?
  15. Can you say what each layer denies?

Drill 1 — The onboarding agent

Design. A Teams agent answers HR and IT questions for new joiners. It retrieves from a shared Azure AI Search index containing HR policy, IT runbooks and the staff directory. The agent's service principal has Search.ReadAll. Answers are cached in Redis keyed by hash(question) with a 24-hour TTL for cost reasons. The agent can call it.reset_password and hr.book_leave. Availability target: 99.9%, "same as Azure AI Search". Session state is kept in the bot's process memory, and the bot runs as a single instance because "the load is tiny".

Find the problems. Five minutes.

Findings
  1. The cache key has no tenant and no viewer. hash(question) means two people asking "what is my leave balance?" share an answer. This is a data leak, it is the most severe finding, and it is the one to lead with.
  2. it.reset_password is an irreversible-ish action with no approval, no idempotency and no verification of the caller. A prompt-injected instruction in a retrieved runbook can trigger it. At minimum: dual control or a strong out-of-band verification, plus an idempotency key.
  3. Search.ReadAll is not least privilege. The agent can read every index in the tenant, including ones it was never meant to see. Scope per index and per data class.
  4. The staff directory in the same index as HR policy means salary-adjacent or personal fields are one retrieval away. Classification partitions, or a separate index.
  5. 99.9% "same as Azure AI Search" is not composed. The bot, the model, the search service and the tool endpoints are all serial. The real number is nearer 99.5%.
  6. Session state in process memory plus a single instance means every deploy drops every in-flight conversation, and there is no horizontal scaling path.
  7. A 24-hour cache TTL on HR policy answers is a staleness bug the day a policy changes. Cache the retrieval, not the answer, or invalidate on document change.
  8. No evidence artifacts named. When somebody's password is reset unexpectedly, there is nothing to investigate with.

The one-sentence verdict: "The cache key is a data leak and the password reset is an unapproved side effect reachable from retrieved content — both are blockers; the rest is fixable in a sprint."


Drill 2 — The research assistant

Design. An agent for the markets desk summarizes research notes, news and internal analyst commentary. Retrieval is hybrid over a single index with a metadata filter on desk and classification. The model is chosen by a router that prefers the cheapest provider meeting a quality threshold; on 429 it retries the same provider three times, then falls over to the next cheapest. Prompts include the user's recent conversation history for personalization. Answers cite sources. Cost is attributed per business unit monthly from the provider's bill.

Find the problems. Five minutes.

Findings
  1. Single index with a metadata filter — the whole Design 04 argument. Recall collapses for narrowly entitled users, result counts leak, one refactor from a breach, and it cannot be proved to an auditor. On a markets desk this is specifically an MNPI exposure: desk and classification do not model a deal barrier.
  2. "Cheapest provider meeting a quality threshold" has no residency gate. For a UAE bank with confidential research this is the fallback-residency seam, and it will be found by an auditor, not a test.
  3. Three retries against a provider that is rate limiting you makes the rate limiting worse and burns the latency budget before the fallback is even attempted. One retry with jitter, then fall over.
  4. Conversation history in the prompt is a cross-contamination risk — yesterday's confidential discussion appears in today's context for a question that did not warrant it. History must be entitlement-checked at use time, not just at write time.
  5. Monthly attribution from the provider's bill cannot attribute to a user, an agent or a request. Emit a per-call accounting record with tenant, agent_id, user_id, cached_tokens and route_reason, and reconcile it against the bill.
  6. Citations are claimed but not verified. Nothing checks that a cited document was actually retrieved. A grounding check — every citation appears in the retrieval set — is cheap and closes a whole class of confident fabrication.
  7. No taint handling. Research notes and news are external content; if the agent gains any side-effecting tool later, the containment rule has nowhere to attach.

The one-sentence verdict: "On a markets desk, a single filtered index is an information-barrier problem rather than a retrieval-quality problem, and the router has no residency gate — those two first, then the retry policy."


Drill 3 — The payments copilot

Design. Investigates held payments and can release them. Dual control above 50,000 USD: the request carries an approvals array of employee ids, and the gateway checks len(approvals) >= 2. Idempotency via a key the agent generates as uuid4() per attempt. The core-banking call is wrapped in a retry with exponential backoff, three attempts. The delegation chain is passed to the compliance agent as a JSON field {"chain": ["user", "orchestrator"]}. All actions are logged to a Splunk index with the agent id, tool and timestamp.

Find the problems. Five minutes.

Findings
  1. uuid4() per attempt defeats the entire purpose of an idempotency key. Every retry gets a new key, so three retries can release the payment three times. This is the most severe finding and it is a money bug.
  2. The retry wraps a non-idempotent call. Even with a stable key, retrying on timeout is the dangerous case: you do not know whether it ran. Timeout must not be retryable for a non-idempotent effect without a stable key and a store.
  3. len(approvals) >= 2 does not exclude the requester, the acting agent, or the chain. The user plus their own orchestrator satisfies it. Approvers must be distinct and outside the forbidden set.
  4. Approvals arrive as an array of ids in the request — unauthenticated and forgeable. An approval must be a signed artifact from an authenticated approver, not a string in a payload.
  5. The chain as a JSON field is forgeable by any hop. It must be derived from the act claim of a verified token.
  6. The audit record names the agent, not the human. No trace id either, so the approval cannot be linked to the action. This is the evidence-pack failure in its most common form.
  7. No circuit breaker. Three retries per request against a degraded core banking is a retry storm precisely when the dependency is weakest.
  8. 50,000 USD threshold with no velocity or aggregate limit. An attacker sends 49,999 repeatedly.

The one-sentence verdict: "The idempotency key is regenerated per attempt and the retry can double-pay — that is a money bug and it blocks; then the approvals are forgeable and exclude nobody, which makes dual control decorative."


Drill 4 — The multi-agent credit workflow

Design. A credit application is processed by five agents: intake, document extraction, financial spreading, risk scoring, and decisioning. Each agent may call others as needed — "the graph is dynamic so the workflow can adapt". Each agent authenticates with its own service principal. The decisioning agent can issue an approval up to 250,000 AED autonomously. Agents communicate over an internal HTTP API with a shared API key. State is passed between agents as a JSON blob. Progress is streamed to the applicant's portal via a webhook URL supplied in the original request.

Find the problems. Five minutes.

Findings
  1. "Each agent may call others as needed" with no depth bound and no cycle check. Five agents with dynamic edges will produce A→B→A on the first ambiguous input, and there is no limit to stop it. Depth bound and cycle detection at the authorization layer.
  2. A shared API key across all agents means there is no identity, no least privilege and no attribution — a compromise of any agent is a compromise of all. mTLS with per-workload SVIDs.
  3. Per-agent service principals lose the applicant and the human decision-maker. For a credit decision, attribution to a person is a regulatory requirement, not a nicety.
  4. Autonomous approval to 250,000 AED with no stated eval suite, no autonomy band and no sampling review. The number is not the problem; the absence of the evidence contract that would justify any number is.
  5. A caller-supplied webhook URL with no allow-list is SSRF, and it is also an exfiltration channel for whatever the progress payload contains.
  6. A JSON blob as inter-agent state has no schema, no version and no classification. It will drift, and one agent will start depending on a field another stopped sending.
  7. No adverse-action explanation path. A declined credit application in most jurisdictions requires a reason; a five-agent chain with no lineage cannot produce one.
  8. Five agents in series compounds reliability: at p = 0.97 per agent that is 0.86 end to end, before any tool calls.

The one-sentence verdict: "A dynamic graph with no depth bound, a shared key instead of identity, and an autonomous credit approval with no evidence contract — this is three separate blockers, and the webhook is an SSRF on top."


Drill 5 — The cost-reduction proposal

Proposal. Platform spend is 40% over budget. The team proposes: (a) route all traffic to the small model except when the user opts into "high quality"; (b) enable semantic caching with a 0.85 similarity threshold across all tenants; (c) reduce the retrieval top-k from 20 to 5; (d) sample evidence artifacts at 10% to cut storage; (e) drop the second model provider, since the fallback "has never been used"; (f) shorten the injection scan to the first 512 tokens of each document.

Find the problems. Five minutes.

Findings
  1. (f) is a control on the degradation ladder in disguise. Truncating the injection scan means an attacker puts the payload at token 513. This is the one to refuse outright, and the principle to state: quality may degrade, safety may not.
  2. (d) breaks the evidence pack. A sampled pack is not a pack — the action you get asked about is the one that was sampled out. Sample spans; never sample artifacts. And storage is not where the money is: ~5 KB × 8,000/day is 40 MB/day.
  3. (b) across all tenants is a data leak, full stop. Semantic caching needs tenant-first keys and must never serve entitlement-dependent answers. A 0.85 threshold is also far too loose — "what is" and "what was" our exposure are close in embedding space and different in answer.
  4. (e) removes the redundancy that the availability number depends on. "Never used" is not evidence it is unnecessary; it is evidence it was never tested. If it truly is not needed, the SLO must be restated downward — that is a conversation with the business, not a cost decision.
  5. (a) is legitimate but should be a measured trade: run the eval suite on the small model and state the quality delta. And "opt into high quality" is a bad UX for the exact users who most need it and least know to ask.
  6. (c) is legitimate and cheap to verify — measure recall@5 versus recall@20 on the graded set. If the reranker is good, 5 may be fine.
  7. Nobody has looked at cost per successful action. If 25% of runs fail and retry, quality work may be the cheapest available saving.

The one-sentence verdict: "(a) and (c) yes, with measurement; (b) with tenant-scoped keys and a much higher floor; (d), (e) and (f) no — and (f) is not a cost decision at all, it is removing a control."

And the meta-point worth making out loud: three of six proposals reduce cost by removing evidence or controls. That is the pattern to watch for in any cost exercise, because those are the line items with no immediate user-visible consequence.


Drill 6 — The incident, reviewed

Post-mortem. At 14:20 the platform began returning errors for 60% of requests. Root cause: a policy bundle push contained a malformed rule; the control plane rejected the bundle and returned 500 to every authorization call. The platform, calling the control plane synchronously per request, failed closed. Mitigated at 15:05 by rolling back the bundle. Resolved. Action items: "add bundle validation to CI" (owner: the team) and "consider caching policy decisions".

Find the problems. Five minutes.

Findings
  1. A synchronous control-plane call on every request is the architectural finding, and it is bigger than the incident. The bundle bug was the trigger; the design was the cause. Local bundle, pushed, with a TTL.
  2. Fail-closed on the control plane turned a dependency failure into a total outage. Fail static: last known-good bundle, alarm, hard stop past a defined age.
  3. 45 minutes to mitigate, and the mitigation was a manual rollback. Where was the automatic revert on a canary? A bundle push should be canaried and auto-reverted on error rate.
  4. "Owner: the team" completes nothing. An action item needs a named human and a date.
  5. "Consider caching policy decisions" is not an action item. It is a topic. It should be an ADR with a decision, or a ticket with a design.
  6. Mitigation is recorded as resolution. The bundle was rolled back; the cause — a synchronous dependency with a fail-closed posture — is untouched. Conflating the two is how incidents recur.
  7. No mention of the error budget. 45 minutes at 60% error is ~27 minutes of budget, which at a 99.65% target is 18% of the month gone. That should trigger the policy state, and the post-mortem should say which state.
  8. No detection time stated. 14:20 is when it began — when did anyone know? Time-to-detect is usually the most improvable number in a post-mortem and it is missing.

The one-sentence verdict: "The bundle bug is the trigger, not the cause — the cause is a synchronous control-plane dependency with a fail-closed posture, and until that changes the same class of incident recurs with a different trigger."


How to deliver a review

Finding the problems is half of it. The other half is saying them in a way that gets them fixed.

Lead with the blocker, not the list. One sentence: what would stop this shipping. Then the rest, in severity order. A review that starts at item 8 has already lost the room's attention for item 1.

Say what fails, not what is wrong. "The cache key has no tenant" is an opinion until you say "two users asking the same question get each other's answer." A concrete failure gets fixed; a principle gets debated.

Separate blockers from improvements explicitly. "These two block; these five I'd take in the next sprint; these three are taste." Reviewers who do not grade get their blockers ignored along with their preferences.

Name what is good. Not politeness — calibration. A review with no positives reads as reflexive, and the author stops distinguishing your severe findings from your mild ones.

Ask before asserting when the design might know something you do not. "What happens on a retry here?" beats "this is not idempotent" when you may have missed a store. You are right often enough that being wrong loudly is expensive.

And end with the question you could not answer from the document. It is usually the most useful sentence in the review, and it tells the author what the design failed to communicate.