LLM Engineering Cheatsheet

Quick reference for the most common LLM engineering facts, formulas, and decisions.

Tokenization

Tiktoken quick check:

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
len(enc.encode("your text here"))

Context Windows (2024-2025)

Pricing Reference (approximate, verify current pricing)

Cost formula:

cost = (input_tokens / 1_000_000 × input_price) + (output_tokens / 1_000_000 × output_price)

Memory Requirements

Formula: params × bytes_per_param + KV_cache + activations

KV cache per token: 2 × num_layers × num_kv_heads × head_dim × bytes_per_element

Inference Parameters Quick Reference

By use case:

Architecture Key Numbers

Llama-3.1-70B specs: 80 layers, 8192 d_model, 64 Q heads, 8 KV heads (GQA), 32K vocab

Quantization Quick Reference

Model Selection Quick Decision Tree

Need structured JSON output?
  → gpt-4o-mini with json_mode OR fine-tuned smaller model

Need to run locally / no cloud?
  → Ollama + Qwen2.5-7B (or 14B if you have 16GB RAM)

Need to process 100K+ token documents?
  → Gemini 1.5 Pro (1M context)

Need best coding quality?
  → claude-3-5-sonnet OR gpt-4o OR deepseek-coder

Need cheapest possible for batch tasks?
  → gpt-4o-mini OR gemini-1.5-flash OR groq-llama3

Need best reasoning?
  → o1 OR claude-3-5-sonnet OR gemini-1.5-pro

Need open weights for fine-tuning?
  → Llama-3.1-8B (small) OR Qwen2.5-14B (medium) OR Llama-3.1-70B (large)

vLLM Key Flags

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --port 8000 \
  --tensor-parallel-size 2 \       # Number of GPUs
  --max-model-len 32768 \          # Context window
  --gpu-memory-utilization 0.90 \  # 90% of VRAM for KV cache
  --enable-prefix-caching \        # Cache repeated prefixes
  --max-num-seqs 256               # Max concurrent sequences

RAG Quick Reference

Chunk size guidance:

• Short factual docs: 256-512 tokens
• Long-form text: 512-1024 tokens
• Code: function-level (variable size)
• Overlap: 10-20% of chunk size

top_k retrieval → rerank → top_n to model:

• Retrieve top 20-50 → rerank → send top 3-8 to model

Embedding models:

• text-embedding-3-small: 1536 dim, $0.02/1M tokens
• text-embedding-3-large: 3072 dim, $0.13/1M tokens

Serving Performance Targets

Cost Estimation Formula

def estimate_monthly_cost(
    daily_requests: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    input_price_per_m: float,
    output_price_per_m: float
) -> float:
    monthly_requests = daily_requests * 30
    input_cost = (monthly_requests * avg_input_tokens / 1_000_000) * input_price_per_m
    output_cost = (monthly_requests * avg_output_tokens / 1_000_000) * output_price_per_m
    return input_cost + output_cost

# Example: 10k req/day, 500 input, 200 output, gpt-4o-mini
cost = estimate_monthly_cost(10_000, 500, 200, 0.15, 0.60)
print(f"${cost:.2f}/month")  # ~$22.50/month

Security Checklist (Quick)

□ Input length limit enforced
□ Prompt injection hardening in system prompt
□ Tool whitelist (agents only call approved tools)
□ Approval gates for irreversible agent actions
□ Output schema validation
□ PII scrubbed from logs
□ Rate limiting per user
□ Content moderation on public endpoints
□ API keys in secrets manager (not hard-coded)
□ No secrets in system prompts

Fine-Tuning Decision

Can solve with prompting? → Use prompting
Can solve with few-shot? → Use few-shot
Need up-to-date knowledge? → Use RAG
Need consistent style/format at scale? → Fine-tune
Have < 100 examples? → Don't fine-tune yet
Need to teach facts? → Use RAG, not fine-tuning

Key Open Source Models (2024-2025)