Knowledge 01 — Generative AI & LLM Foundations

Goal of this module. Build, from zero, the mental model of what an LLM actually is and does, just deep enough to make every downstream decision (RAG, agents, prompting, eval, cost) principled rather than cargo-culted. JD4 is an application role, not a model-training role — so this module favors the working engineer's "why does it behave like this" over the mathematician's derivation. For the deep transformer/attention math and inference internals, this track links out to the JD2 inference internals module rather than duplicating it.

Table of Contents

1. What a language model is

Strip away the hype: an LLM is a function that takes a sequence of tokens and outputs a probability distribution over the next token. That's it. Everything else — chat, reasoning, agents — is built by repeatedly sampling from that distribution and feeding the result back in.

P(next_token | all_previous_tokens) — the model has learned this conditional distribution from a vast corpus. "Generative AI" = sampling sequences from a learned distribution. A chat assistant is this loop wrapped in a conversation format and an instruction-following fine-tune.

The deep consequence for a banking engineer: the model has no database and no notion of truth. It predicts plausible continuations. Plausible ≠ correct. Every safety and grounding decision downstream flows from internalizing that one fact.

2. Tokens: the unit you bill, budget, and break things in

What a token is. Text is chopped into tokens — sub-word chunks — by a tokenizer (OpenAI models use Byte-Pair Encoding, BPE, via the tiktoken/o200k_base vocabulary). Common words are one token; rare words split into several. Roughly:

  • English: 1 token ≈ 4 characters ≈ 0.75 words. ~750 words ≈ 1,000 tokens.
  • Arabic, code, JSON, and many non-Latin scripts: fewer characters per token → more tokens per word. Arabic can be ~2× the token count of equivalent English. This is a real JD4 cost/latency factor for an Arabic banking assistant.

Why you care:

  • Cost is per-token (input + output). Output tokens cost more.
  • Latency grows with output tokens (each is generated sequentially — see §5).
  • Context limits are in tokens; overflow truncates or errors.
  • Budgeting context for RAG means counting tokens: system prompt + retrieved chunks + history + question must fit, with headroom for the answer.
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
len(enc.encode("Your account balance is 1,250.00 AED"))  # count before you send

3. Embeddings: meaning as geometry

What an embedding is. A function that maps a piece of text to a fixed-length vector of floats such that semantically similar texts land near each other in that high-dimensional space. "card was declined" and "transaction failed" point in nearly the same direction even with no shared words.

Why it exists. Keyword search matches strings; embeddings let you match meaning. This is the engine of vector/semantic retrieval (K02) and of clustering, dedup, and classification.

Under the hood. An embedding model is a transformer trained (often with contrastive objectives) so that the vector of a query and the vector of a relevant passage have high cosine similarity (the cosine of the angle between them; 1 = identical direction, 0 = unrelated). You compare with cosine similarity or (equivalently for normalized vectors) dot product / Euclidean distance.

The numbers that matter (Azure):

  • text-embedding-3-large → 3072 dimensions (truncatable via the dimensions parameter to trade accuracy for storage/speed).
  • text-embedding-3-small → 1536 dims, cheaper.
  • Dimensionality sizes your vector index and your Cosmos vector cost — a real design lever.
  • For Arabic/English, use a multilingual-capable embedding (the -3-* models are multilingual) so cross-language retrieval works.

4. The transformer, in one honest paragraph

The model is a stack of transformer blocks. Each block has self-attention (every token computes a weighted sum over all other tokens — "what should I pay attention to?") and a feed-forward network (per-token nonlinear transform). Attention is how the model relates "it" to the noun three sentences back, or a question to its context. The weights learned during training encode grammar, facts, and reasoning patterns as numbers in these layers. You do not need the matrix math to do JD4 well — but you must know that (a) attention is O(n²) in sequence length, which is why long context is expensive, and (b) the model's "knowledge" is frozen in the weights at training time, which is why it has a knowledge cutoff and needs RAG/tools for fresh or private facts. For the full mechanism, see JD2 K01.

5. Generation: autoregression, sampling, temperature

Autoregression. The model generates one token at a time: it predicts a distribution, picks a token, appends it, and predicts again — until a stop condition. Two phases:

  • Prefill — process the whole prompt at once (parallel, compute-bound).
  • Decode — emit tokens one by one (sequential, memory-bound). This is why a long answer is slow and costly: each output token is a full forward pass.

Sampling controls (you set these per call):

  • temperature (0–2): scales the distribution before sampling. 0 ≈ deterministic (always the top token) → use for extraction, classification, tool-arg generation, anything in banking where you want repeatability. Higher → more diverse/creative → rarely what a bank wants.
  • top_p (nucleus): sample only from the smallest set of tokens whose probability sums to p. An alternative knob to temperature.
  • max_tokens / stop: cap output length and define stop sequences — both cost and safety controls.
  • seed: request reproducibility (best-effort).

Banking default: low temperature, capped output, structured format, grounded context. You almost never want a "creative" bank bot.

6. The context window and why it matters

The context window is the maximum number of tokens (prompt + generated output) the model can attend to at once — e.g. 128K for GPT-4o, larger for newer models. Everything the model "knows" in this turn is either in its frozen weights or in this window.

Implications:

  • It's working memory, not long-term memory. Conversation history, retrieved docs, tool results, and the system prompt all share the budget. Overflow = you must truncate, summarize, or retrieve more selectively.
  • "Lost in the middle": models attend more reliably to the start and end of a long context than the middle. So put the most important instructions/grounding near the top and the question near the bottom; don't dump 100 chunks and hope.
  • More context ≠ better answers — and it's tokens you pay for and latency you add. Retrieval quality beats retrieval quantity (K02).

7. Why models hallucinate (and the four fixes)

Definition. A hallucination is a confident, fluent statement that is not supported by any source and may be false. In a bank, a hallucinated balance, rate, or policy is a serious incident.

Why it happens (mechanism, not hand-waving): the model samples plausible continuations from its learned distribution. When the prompt asks for something the weights don't reliably encode (a specific customer's balance, a niche policy clause, last week's rate), the most "plausible" continuation is a fabricated-but-well-formed answer. The model has no built-in "I don't actually know" signal unless trained/prompted to produce one.

The four fixes (in order of leverage):

  1. Ground it (RAG/tools). Don't ask the model to recall the fact; retrieve it and put it in context, or fetch it via a tool call. The model's job becomes phrasing provided facts, which it does well. (K02, K03).
  2. Instruct it to abstain. System prompt: "Answer only from the provided context; if the answer isn't there, say you don't have that information and offer to connect a human." (K06).
  3. Verify it (eval/guardrails). A groundedness evaluator checks every claim against the retrieved context; Content Safety Groundedness detection can flag at runtime (K08).
  4. Constrain the output. Structured output / JSON schema and tool calls reduce free-form fabrication for the parts that must be exact (§8).

The interview soundbite: "In a bank you never let the model be the source of truth for a fact that has a system of record — you retrieve or call a tool, instruct abstention, and verify groundedness."

8. Function / tool calling and structured output

The problem it solves. The model is great at language, terrible at being a source of live/exact data and at doing reliable actions. Tool calling lets the model request that your code run a function with arguments it generates, then continue with the result.

How it works:

  1. You describe tools as JSON schemas (name, description, parameters).
  2. You pass them with the chat request.
  3. The model, instead of answering, may return a tool call: {name: "get_balance", arguments: {account_id: "..."}}.
  4. Your code executes it (calls core-banking), and you feed the result back.
  5. The model phrases the final answer using the real value.
tools = [{
  "type": "function",
  "function": {
    "name": "get_account_balance",
    "description": "Return the current balance for an account the user is authorized to view.",
    "parameters": {"type": "object",
      "properties": {"account_id": {"type": "string"}},
      "required": ["account_id"]}
  }}]
# model returns tool_calls; YOUR code runs the lookup; you never let the LLM invent the number.

Structured output / JSON mode / JSON Schema. You can force the model to return JSON matching a schema (Azure OpenAI supports response_format with a JSON Schema and strict mode). Essential for extraction, routing decisions, and anything a downstream system parses — no regex-scraping prose.

This pattern is the atom of agents (K03): an agent is a loop that lets the model call tools repeatedly until the task is done.

9. Model families you must name

For JD4 you call Azure OpenAI deployments; know the families:

  • GPT-4o / GPT-4.1 — flagship multimodal (text + vision), fast, the default for grounded chat and reasoning over retrieved context. 4o-mini / 4.1-mini / 4.1-nano — cheap, fast, great for routing, classification, extraction.
  • o-series (reasoning models, e.g. o3/o4-mini-class) — spend extra "thinking" tokens for hard multi-step reasoning/math/planning; higher latency/cost — use selectively (e.g. complex agent planning), not for every chat turn.
  • text-embedding-3-large/small — embeddings for RAG.
  • Whisper — speech-to-text (IVR/voice banking). DALL·E — image gen (rarely needed in banking).
  • Open models in the Foundry catalog (Llama, Mistral, Cohere, DeepSeek, Phi) — for cost/residency/customization; deployable serverlessly. A bank might use a small open model for on-prem/edge classification.

Senior nuance: tier your models. Cheap model routes/classifies/extracts; flagship reasons over grounded context; reasoning model only for genuinely hard planning. This is the single biggest cost lever (K00 §3).

10. Training stages: pretraining → SFT → RLHF/DPO

You won't train foundation models in JD4, but you must speak the stages (and why fine-tuning is usually the wrong tool for a bank's facts):

  • Pretraining — predict-the-next-token on web-scale text. Yields raw capability and world knowledge up to a cutoff date.
  • Supervised fine-tuning (SFT) — train on curated instruction→response pairs so it follows instructions and adopts a helpful assistant style.
  • Preference optimization (RLHF / DPO) — align outputs with human preferences (helpful, harmless, honest) using reward models or direct preference optimization.
  • (Optional) domain fine-tuning — adapt style/format/vocabulary to a domain. Not for injecting changing facts — those go in RAG/tools. A bank fine-tunes (rarely) to enforce tone/format or a niche task, never to memorize balances or rates.

RAG vs fine-tuning, the line to remember: RAG/tools = knowledge that changes or is private; fine-tuning = behavior/style that's stable. Banks lean almost entirely on RAG/tools because their facts change and must be auditable to a source.

11. Common misconceptions

  • "The model knows facts." It models plausible text. Treat any unsourced fact as a guess.
  • "Bigger context = better answers." Often worse (lost-in-the-middle) and always costlier. Retrieve precisely.
  • "Temperature 0 makes it correct." It makes it deterministic, not true. Determinism ≠ grounding.
  • "Fine-tune it on our data so it knows our policies." Policies change and need citations → RAG, not fine-tuning.
  • "Tool calling means the model runs my code." No — the model requests a call; your code runs it and returns the result. The trust boundary is yours.
  • "Arabic is just English with different letters, cost-wise." Arabic typically costs ~2× the tokens — budget for it.

12. Interview Q&A

Q: Why do LLMs hallucinate, and how do you prevent it in a banking assistant? They sample plausible continuations with no truth model, so when asked for facts the weights don't reliably hold, they fabricate fluently. Prevent with: grounding (RAG + tool calls to systems of record), abstention instructions, groundedness evaluation/guardrails, and structured output for exact fields. Never let the model be the source of truth for a fact with a system of record.

Q: RAG or fine-tuning for our product FAQ that changes monthly? RAG. The content changes and must be traceable to a source for audit; fine-tuning bakes in stale facts and gives no citation. Fine-tuning would only be for stable style/format, not changing facts.

Q: How do you control cost and latency? Cost/latency are dominated by tokens, especially output. Tier models (cheap for routing/extraction, flagship for reasoning), cap output length, trim retrieved context to what helps, cache repeated prompts, and stream output for perceived latency. At scale, PTUs for predictable capacity.

Q: A user asks the bot in Arabic. What changes technically? Tokenization is denser (≈2× tokens → more cost/latency, watch context budget); ensure a multilingual embedding model and multilingual/Arabic-capable generation; handle RTL rendering in the UI; OCR/Document Intelligence must support Arabic script and diacritics (K05); evals must include Arabic test cases.

Q: What's the difference between embeddings and the chat model? The chat model generates text autoregressively (next-token prediction). The embedding model maps text to a fixed vector capturing meaning, for similarity/retrieval. Different jobs, different deployments; RAG uses both — embeddings to find context, the chat model to answer over it.

13. References

  • Attention / Transformers — Vaswani et al., "Attention Is All You Need" (2017).
  • BPE tokenization — Sennrich et al. (2016); OpenAI tiktoken docs.
  • Embeddings — Reimers & Gurevych, Sentence-BERT (2019); OpenAI text-embedding-3 model card.
  • Lost in the middle — Liu et al., "Lost in the Middle: How Language Models Use Long Contexts" (2023).
  • RLHF — Ouyang et al., "Training language models to follow instructions with human feedback" (InstructGPT, 2022); DPO — Rafailov et al. (2023).
  • Function calling / structured output — OpenAI & Azure OpenAI docs (function calling, structured outputs / JSON Schema).
  • Hallucination — Ji et al., "Survey of Hallucination in Natural Language Generation" (2023).
  • Deeper inference internals: JD2 K01 — LLM Inference Internals.