Admin Dashboard

Phase 8 · Document 08 · LLM Gateways Prev: 07 — Streaming Proxy · Up: Phase 8 Index

Table of Contents

1. Why This Matters
2. Core Concept
3. Mental Model
4. Hitchhiker's Guide
5. Warmup Readings
6. Deep Readings and External References
7. Key Terms
8. Important Facts
9. Observations from Real Systems
10. Common Misconceptions
11. Engineering Decision Framework
12. Hands-On Lab
13. Verification Questions
14. Takeaways
15. Artifact Checklist

1. Why This Matters

The admin dashboard is the human control surface for the gateway — the place where a platform owner sees what's happening (usage, cost, errors, latency, provider health) and changes what happens (manage models/routes, issue API keys, set budgets, flip a provider off). Everything you've built — registry, routing, metering, policy — needs a UI/API for operators, finance, and team leads to use it without editing config files or SQL. Without it, the gateway is a black box only its authors can run; with it, finance does chargeback, team leads self-serve keys and budgets, and on-call sees the incident. It's the difference between an internal script and a product.

2. Core Concept

Plain-English primer: read + write surfaces over the gateway's data

The dashboard is two things over the gateway's stores: a read surface (visualize the metering + observability data) and a write surface (manage the registry, routes, keys, budgets, policy). Concretely:

Read (visibility):

• Usage & cost — per user/project/key/model/provider/route, over time; the metering aggregates. The feature finance wants.
• Performance — p50/p95/p99 TTFT/TPOT, throughput, error rate, fallback rate (Phase 7.08).
• Provider health — up/down, circuit-breaker state, rate-limit headroom (Phase 7.07).
• Budget burn — spend vs budget per tenant, with alerts approaching limits (06).

Write (control):

• Models — add/edit/deprecate registry rows, prices, capability flags (04).
• Routes — define/edit aliases (auto/coding) and routing rules/constraints (05).
• API keys (virtual keys) — issue per user/team with allowed models, budgets, rate limits (02, 06).
• Providers — credentials, enable/disable, health overrides.
• Policy — guardrails, data-residency rules, RBAC (09).

The admin API (the dashboard is a client of it)

Build the API first, render the UI on top — so automation and the UI share one surface and the UI is just a client:

GET  /admin/usage?group_by=project&from=...&to=...     # cost/usage aggregates [06]
GET  /admin/metrics                                     # p95/p99, error/fallback rate [7.08]
GET  /admin/providers                                   # health, breaker state [7.07]
POST /admin/models        /admin/models/:id (PATCH/deprecate)   # registry [04]
POST /admin/routes                                      # aliases + rules [05]
POST /admin/keys          {user, models, max_budget, rpm}       # virtual keys [06]
POST /admin/budgets       {project, monthly_budget, alert_at}   # budgets [06]

Who uses it (personas → views)

A dashboard is only useful if it serves its personas:

• Platform/on-call engineer — health, errors, latency, provider state, capacity; the incident view (Phase 7.10).
• Finance / FinOps — cost by team/project, trends, chargeback export, budget burn (06, Phase 15.05).
• Team lead / developer — self-serve API keys, their team's usage vs budget, available models.
• Admin / security — RBAC, audit log, policy config, key lifecycle (09).

It is itself a privileged surface (secure it)

The dashboard creates keys, sees usage, and changes routing/policy — so it's a high-value target: it needs authentication + RBAC (not everyone can issue keys or edit policy), an audit log of admin actions (who changed what), and protection of the master/admin key (09, Phase 14). Don't reinvent metrics visualization — many teams point Grafana at the gateway's metrics/DB for the read dashboards and build a small custom admin app for the write operations.

3. Mental Model

        ┌──────────────── ADMIN DASHBOARD (over the gateway's stores) ────────────────┐
   READ │ usage/cost by tenant [06] · p95/p99 + error/fallback [7.08] · provider health │
        │ [7.07] · budget burn [06]                                                     │
   WRITE│ models/prices/caps [04] · routes/aliases [05] · virtual keys+budgets+limits   │
        │ [06] · providers (enable/health) · policy/RBAC [09]                           │
        └───────────────────────────────┬───────────────────────────────────────────────┘
        build the ADMIN API first → UI is a client of it (automation + UI share one surface)
        personas: on-call · finance/FinOps · team lead · admin/security
        it's PRIVILEGED → auth + RBAC + audit log + protect the master key [09, Phase 14]
        reuse Grafana for READ dashboards; small custom app for WRITE ops

Mnemonic: the dashboard is read (usage/cost/health) + write (models/routes/keys/budgets/policy) over the gateway's stores — API-first, persona-driven, and itself a privileged, audited surface.

4. Hitchhiker's Guide

What to look for first: the cost-by-tenant view and key + budget management — they deliver the gateway's day-one operational value (visibility + self-serve control).

What to ignore at first: a beautiful bespoke analytics UI. Reuse Grafana for read dashboards over your metrics/DB; build only the write admin (keys/routes/budgets) that tools don't give you.

What misleads beginners:

• UI-first instead of API-first. Build the admin API first so automation and the UI share one surface (02 exposes such APIs).
• Forgetting the dashboard is privileged. It mints keys and edits policy — without RBAC + audit, it's a breach waiting to happen (09, Phase 14).
• No personas. A dashboard that serves no specific user (finance vs on-call vs team lead) serves none well.
• Reinventing metrics viz. Grafana already does p95/p99 + cost panels (Phase 7.08).
• Stale data. If usage/cost lags, finance and budget alerts are wrong — drive it from the live metering store.

How experts reason: they build an admin API over the gateway's stores, render persona-specific views (on-call/finance/lead/admin), reuse Grafana for reads and a small app for writes, and treat the dashboard as a secured, audited, RBAC'd surface. They make key/budget self-service to remove themselves as a bottleneck.

What matters in production: accurate, fresh cost/usage (finance trusts it), working key/budget/route management, RBAC + audit on admin actions, and an incident view tied to the runbook.

How to debug/verify: confirm dashboard cost matches metering totals (and invoices, 06); test that a non-admin can't mint keys/edit policy (RBAC); verify admin actions appear in the audit log.

Questions to ask: is there an admin API behind the UI? RBAC + audit on admin actions? cost-by-tenant + chargeback export? self-serve keys/budgets? does it reuse existing metrics tooling?

What silently gets expensive/unreliable: an un-RBAC'd dashboard (anyone mints keys), stale cost data (bad chargeback), and a hand-built metrics UI that diverges from the real observability (Phase 7.08).

5. Warmup Readings

6. Deep Readings and External References

7. Key Terms

8. Important Facts

• The dashboard is read (usage/cost/health/latency) + write (models/routes/keys/budgets/policy) over the gateway's stores.
• Build the admin API first; the UI is a client so automation and UI share one surface.
• Cost-by-tenant + chargeback export is the headline value for finance (06, Phase 15).
• It's a privileged surface (mints keys, edits policy) → needs auth + RBAC + audit log + master-key protection (09, Phase 14).
• Reuse Grafana for read dashboards over metrics/DB; build a small custom app for write ops (Phase 7.08).
• Serve specific personas (on-call, finance, team lead, admin) — generic dashboards help no one.
• Drive it from live metering so cost/budget data is fresh and matches invoices.
• LiteLLM ships an admin UI + API — reuse rather than rebuild the basics (02).

9. Observations from Real Systems

• LiteLLM's admin UI/API manages virtual keys, budgets, models, and shows spend — the reference self-hosted gateway admin (02).
• OpenRouter's activity/credits views are the hosted-aggregator equivalent (01).
• Most teams pair Grafana (reads) with a small custom admin app (writes) rather than building analytics from scratch (Phase 7.08).
• Enterprises require RBAC + audit on the admin surface for SOC2/compliance — admin actions are logged (09, Phase 14).
• Self-serve keys/budgets are what let a platform team support many product teams without becoming a ticket queue.

10. Common Misconceptions

11. Engineering Decision Framework

BUILD the admin surface:
 1. API FIRST: /admin/* for usage/metrics/providers (read) + models/routes/keys/budgets/policy (write).
 2. READS: reuse Grafana over metering DB + metrics (cost-by-tenant, p95/p99, fallback, health). [06,7.08]
 3. WRITES: small custom app for keys/budgets/routes/models/policy management.                   [04,05,06,09]
 4. PERSONAS: on-call (health/errors/capacity) · finance (cost/chargeback) · lead (self-serve keys) · admin (RBAC/audit).
 5. SECURE IT: auth + RBAC on every admin op; AUDIT LOG all writes; protect the master key.       [09, Phase 14]
 6. FRESHNESS: drive cost/budget from live metering; reconcile with invoices.                      [06]

12. Hands-On Lab

Goal

Stand up the gateway's admin surface: Grafana for cost/latency reads + a small admin API/app for keys, budgets, and routes — secured with RBAC + audit.

Prerequisites

• The gateway with metering + registry (or LiteLLM proxy with DB); Grafana; your web stack.

Steps

1. Read dashboards (Grafana): connect Grafana to the metering DB/metrics; build panels for cost by project/user (last 30d), p95/p99 TTFT, error + fallback rate, and budget burn (Phase 7.08).
2. Admin API: implement (or use LiteLLM's) /admin/keys, /admin/budgets, /admin/routes, /admin/models; create a virtual key with a budget + allowed models and confirm it works through the gateway (06, 05).
3. Write app: a minimal UI (or CLI) over the admin API to issue/revoke keys and set budgets — the self-serve surface.
4. RBAC: add roles (admin vs viewer vs team-lead); verify a non-admin cannot mint keys or edit policy (09).
5. Audit log: record every admin write (who, what, when); display it; confirm a key creation appears.
6. Reconcile: confirm the dashboard's cost totals match the metering aggregates (and a provider invoice figure).

Expected output

A working admin surface: Grafana cost/latency dashboards, an admin API + minimal write app for keys/budgets/routes, RBAC blocking unauthorized writes, and an audit log of admin actions.

Debugging tips

• Cost panel ≠ metering totals → query/timezone mismatch or stale aggregation.
• Non-admin can mint keys → RBAC not enforced on the admin endpoints.

Extension task

Add a budget-burn alert (e.g., 80% of monthly budget) surfaced in the dashboard and via notification (06).

Production extension

Put the admin app behind SSO + RBAC, ship the audit log to your SIEM, and link the on-call view from the runbook (09, Phase 14).

What to measure

Dashboard-vs-metering cost match; key/budget management correctness; RBAC enforcement; audit completeness.

Deliverables

• Grafana cost/latency/health dashboards over the gateway data.
• An admin API + write app for keys/budgets/routes.
• RBAC + an audit log of admin actions.

13. Verification Questions

Basic

1. What are the read and write responsibilities of the admin dashboard?
2. Why build the admin API before the UI?
3. Why is the dashboard a privileged surface?

Applied 4. Design persona-specific views for on-call, finance, and team lead. 5. What do you reuse (Grafana) vs build (write ops), and why?

Debugging 6. Finance's cost view disagrees with the provider invoice. Where do you look? 7. A team lead minted an admin key they shouldn't have. What failed?

System design 8. Design an admin surface with API-first design, persona views, RBAC, and audit for a multi-tenant gateway.

Startup / product 9. How does a good admin dashboard (self-serve keys, chargeback) reduce your support load and prove enterprise-readiness?

14. Takeaways

1. The dashboard is read (usage/cost/health/latency) + write (models/routes/keys/budgets/policy) over the gateway's stores.
2. Build the admin API first; the UI (and automation) are clients of it.
3. Cost-by-tenant + chargeback is the finance headline; self-serve keys/budgets remove the platform team as a bottleneck.
4. Reuse Grafana for reads, build a small app for writes.
5. It's privileged — RBAC + audit + master-key protection are mandatory (09).

15. Artifact Checklist

• Grafana dashboards: cost-by-tenant, p95/p99, error/fallback, budget burn.
• An admin API + a minimal write app (keys/budgets/routes/models).
• RBAC blocking unauthorized admin writes.
• An audit log of admin actions.
• A cost reconciliation (dashboard vs metering vs invoice).

Up: Phase 8 Index · Next: 09 — Enterprise Policy Engine