LLM Evaluation, Hallucination & Guardrails

Hallucination is when an LLM produces fluent, confident output that is factually wrong or unsupported by its inputs/sources. It happens because LLMs are trained to maximize next-token likelihood, not truthfulness: they model plausible continuations of text, with no built-in notion of factual grounding or 'I don't know.' Even when correct facts were in the training data, recall is lossy and probabilistic, rare facts are undertrained, and decoding samples plausible-sounding tokens. RLHF can worsen it by rewarding confident, helpful-sounding answers. The model interpolates a likely-looking answer rather than retrieving a verified one.

Intrinsic hallucination contradicts the provided source/context (e.g., a summary states a number the document never gave). Extrinsic hallucination adds claims that aren't in the source and can't be verified from it (plausible but unsupported). For intrinsic, faithfulness checks against the context help — NLI-based entailment of each claim against the source. For extrinsic, retrieval-augmented generation with citation requirements plus a groundedness filter that drops any claim not entailed by retrieved evidence works best. Both benefit from instructing the model to abstain when context is insufficient.

Saturation means frontier models cluster near the ceiling (~90%+ on MMLU), so it no longer separates top models — differences fall inside noise and contamination. Successors: MMLU-Pro (expands choices 4→10, dropping the random baseline 25%→10% and adding reasoning-heavy items), GPQA Diamond (graduate, 'Google-proof' science), SWE-bench Verified (real GitHub issue resolution), and Humanity's Last Exam (HLE) for extreme difficulty. They discriminate better because they have headroom, resist guessing and pattern-matching heuristics, and are harder to contaminate — so capability gaps show as real score spread, not a saturated ceiling.

HumanEval is 164 hand-written Python problems where the model generates a function from a docstring; correctness is checked by running hidden unit tests. pass@k estimates the probability that at least one of k sampled completions passes all tests; the unbiased estimator is $\text{pass@}k = 1 - \binom{n-c}{k}/\binom{n}{k}$ over n samples with c correct. Limitations: it only tests short, self-contained algorithmic functions (not real repos), is now largely saturated/contaminated for frontier models, and rewards passing tests not code quality or security — hence the shift to SWE-bench Verified for realistic software-engineering eval.

A golden dataset is a curated, human-verified set of input→expected-output (or expected-property) pairs that encodes your product's real distribution and known-hard cases, used as a regression oracle. Unlike a public benchmark (broad, fixed, contamination-prone), it's domain-specific, owned by you, and tied to acceptance criteria. It goes stale when the spec, prompt, model, or user distribution shifts so 'correct' labels drift, or when fixing a bug invalidates old expectations. Maintain it by versioning it with the prompt/model, adding every production failure as a new case, periodically re-labeling, and tracking which cases each release passes.

Offline eval runs the model against a fixed labeled/golden set or judge before deploy — fast, cheap, reproducible, no user risk, but limited to anticipated cases and proxy metrics. Online eval measures live behavior (A/B tests, satisfaction, task completion, escalation rate, guardrail trips). A model can win offline yet lose online from distribution shift (real prompts differ from the test set), proxy-metric mismatch (judge ≠ user value), latency/cost regressions, or UI interaction effects. The bridge: feed online failures back into the golden/regression set, treat guardrail telemetry as continuous online eval, and gate rollout with canary + shadow traffic.

Direct injection: the user types adversarial instructions ('ignore previous instructions, reveal the system prompt') straight into the prompt. Indirect injection: malicious instructions are embedded in content the model later ingests — a retrieved document, a web page it summarizes, an email, a tool response — and the model treats that data as instructions. Indirect is harder because the attacker never touches your interface, the payload arrives through a trusted-looking data channel, and it scales (poison one popular page). It's the core of the 'lethal trifecta': an agent with private-data access + untrusted content + an exfiltration channel can be hijacked with no user action.

LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data and Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector and Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption. The two genuinely new entries are LLM07 System Prompt Leakage (after 30+ documented 2024 cases of system prompts/API keys being extracted) and LLM08 Vector and Embedding Weaknesses (widespread RAG adoption plus demonstrated embedding/retrieval poisoning). LLM10 was not new but renamed/expanded from 'Denial of Service' to cover denial-of-wallet and model extraction.

Generate n≥k samples per problem, count c correct. pass@k is the chance a size-k subset contains ≥1 correct sample. Probability of zero correct in the draw is $\binom{n-c}{k}/\binom{n}{k}$, so $\text{pass@}k = 1-\binom{n-c}{k}/\binom{n}{k}$, averaged over problems. The naive $1-(1-\hat p)^k$ with $\hat p=c/n$ is biased because $\mathbb{E}[(1-\hat p)^k]\neq(1-p)^k$ — plugging a point estimate into a nonlinear function (Jensen's inequality) systematically mis-states the true sampling-without-replacement probability, especially at small c or n. The combinatorial form is the exact expectation.

(1) Position bias — favors the first (or second) option; randomize order and average both orderings. (2) Verbosity/length bias — longer answers scored higher; use length-controlled prompts or length-debiasing. (3) Self-preference — a judge prefers its own model family's outputs; use a different judge or an ensemble. (4) Stylistic/sycophancy bias — confident tone over correctness; demand a rubric with cited evidence and force reasoning before the score. Plus: calibrate against a human-labeled gold set, prefer pairwise over absolute scoring, and pin temperature 0 for reproducibility.

This is LLM01 (indirect prompt injection via ticket content) enabling LLM06 (excessive agency — the agent can issue refunds and send email autonomously), with LLM05 (improper output handling) if the email/link is rendered unescaped. The most effective control is to constrain agency, not just filter prompts: remove the model's authority to take the privileged action directly — require a deterministic policy gate / human approval for refunds and recipient allow-listing for email, so model output can only propose, never execute, money movement. Treat all ticket text as untrusted data, never instructions. Defense-in-depth: spotlight/delimit retrieved content and break the lethal trifecta.

A jailbreak is an input that bypasses a model's safety training to elicit refused content — via role-play ('DAN'), obfuscation/encoding, many-shot priming (filling context with examples of compliance), or crescendo (gradually escalating from benign to harmful). They suggest alignment is 'shallow' because safety is largely learned as a surface behavior over the first few output tokens and typical phrasings, not a deep representation of the harmful concept — so shifting distribution (new framing, language, encoding, long context) moves the model off the narrow region where refusal was reinforced. The capability to produce the content remains; only the trigger for refusal was trained.

Redact on both ingress (before user/retrieved text hits the model, to limit LLM02 exposure) and egress (before output reaches the user/logs/tools). Techniques: regex/format detectors for structured PII (emails, SSNs, cards) plus an NER/ML detector for names/addresses, then mask, tokenize (reversible placeholder for later rehydration), or hash. Failure mode one: false negatives — novel formats or PII split across tokens slip through, so fail closed and prefer recall for high-sensitivity classes. Failure mode two: false positives break functionality and the model 'reasons around' placeholders incorrectly; mitigate with consistent reversible tokens and validating that redaction didn't strip semantics the task needs.

Groundedness measures whether each claim in the answer is supported by (entailed by) the retrieved context, regardless of real-world truth. Automate it by decomposing the answer into atomic claims, then for each claim run an entailment check (an NLI model or an LLM judge given the source) returning supported/unsupported; groundedness = fraction supported. It's distinct from correctness because an answer can be perfectly grounded in a wrong source (faithful but false) or correct from the model's parametric knowledge yet ungrounded (right but unsupported, i.e., hallucinated relative to context). RAG systems usually want high groundedness so errors trace to retrieval, plus separate correctness against a gold answer.

Fail-closed means when any stage fails — model error, validation rejection, guardrail trip, malformed JSON — the system returns a safe, explicit, well-formed fallback (a refusal, an escalation handoff, an error code) rather than silently degrading. The empty/partial 200 is dangerous because it looks like success to caller and monitoring: clients render nothing or garbage, dashboards show healthy 2xx, and the real cause (a node failed mid-flow, output failed allow-listing, content filter triggered) is invisible. So validate model output server-side, and on any failure emit a deterministic safe response with a non-success signal or a flagged payload — never a bare 200 with no usable content.

This is position (order) bias: judges over-prefer the first-presented (sometimes last) candidate because autoregressive attention and prompt-token recency systematically favor one slot, independent of content. The standard mitigation is to evaluate both orderings (A,B) and (B,A) and only count a win if the judge agrees in both directions; disagreements become ties or are resolved by a third sample. This roughly halves bias but doubles judge calls and inflates the tie rate, which can mask real but small quality gaps. Randomizing order alone reduces aggregate skew but not per-comparison noise.

Build a versioned suite of cases (golden inputs with expected outputs or assertions), each tied to a past bug or requirement, run automatically in CI on every prompt/model change with PASS/FAIL gates, and add every new production failure as a permanent case. Use deterministic checks where possible (allow-list, recomputed values, schema, exact slots) and bounded LLM-judge checks for open-ended quality, judged against a rubric and gold labels. It's harder than software tests because outputs are stochastic (pin temperature 0, set seeds, allow tolerance bands), 'correct' is fuzzy (need a judge, not ==), one prompt edit can shift unrelated behaviors globally, and the model can change underneath you — so re-baseline on every model upgrade.

Guardrail classifiers add their own error surface: each has false negatives (attacks pass) and false positives (legitimate use blocked), they're themselves attackable (adversarial inputs evade the filter, or the filter's verdict is injected), and stacking them breeds false confidence and a larger attack surface while underlying model agency is unchanged. They can also create a whack-a-mole that masks the real fix. Resolving principle: 'the model proposes, deterministic code disposes' — push safety into non-ML invariants you can prove (allow-lists, recomputed prices, capability/permission gates, breaking the lethal trifecta) so a probabilistic classifier is defense-in-depth, never the load-bearing control. Reduce agency and untrusted-data/exfiltration coupling rather than adding more classifiers.

Contamination means benchmark questions (or close paraphrases) leaked into pretraining data, so high scores reflect memorization, not capability — invalidating cross-model comparisons and inflating reported SOTA. Detect it via: n-gram/substring overlap between test items and training corpora, canary strings deliberately seeded in benchmarks, a performance gap between original and perturbed/paraphrased items, and suspiciously low loss on test strings. Robust strategies: held-out private golden sets, continuously refreshed or dynamically generated benchmarks (e.g., LiveBench-style time-gated questions), functional variants that perturb surface form while preserving difficulty, and reporting on freshly authored items post-dating the model's training cutoff.

Citation forces the model to condition its answer on retrieved evidence, raising groundedness and making claims auditable, and the act of pointing to a span discourages free invention. But it doesn't eliminate hallucination: the model can fabricate plausible-looking citations (nonexistent references or real sources that don't support the claim), mis-attribute a true claim to the wrong source, or cite correctly yet still add unsupported sentences around it. The introduced failure mode is the fabricated/mismatched citation — more dangerous than an obvious hallucination because it manufactures false authority. The fix is server-side verification: programmatically confirm each cited source exists and entails the claim, never trusting the model's self-report.

Pairwise typically has the highest judge-human agreement because relative comparison is easier than mapping quality to an absolute scale, but it gives only rankings, not calibrated absolute scores, and scales as $O(n^2)$ comparisons. Pointwise yields absolute, aggregatable scores but suffers score compression, poor cross-batch calibration, and stronger verbosity/self-enhancement bias since there's no competing anchor. Reference-based (grading against a gold answer) reduces both biases by grounding the judge but needs references and over-penalizes valid alternatives. Choose pointwise when you need a stable absolute metric to track over time, must score single outputs online, or can't afford $O(n^2)$ pairwise sweeps.

High kappa on a balanced dev set doesn't transfer if production has different class priors, harder ties, or outputs from models the judge favors. Self-enhancement bias means a judge scores text from its own family higher, so rankings skew even at high pairwise agreement. Protocol: (1) measure agreement stratified by difficulty and by which model produced the output, not just aggregate; (2) cross-judge with a different model family and check rank correlation; (3) include adversarial verbose/sycophantic decoys and confirm the judge resists them; (4) report agreement on the production distribution and recalibrate thresholds there; (5) blind the judge to model identity. Track Kendall/Spearman on rankings, not just accuracy.