How Do You Make an LLM Cite Its Sources — and Prove It?

A footnote you can't click is worse than no footnote. It borrows the visual language of scholarship — the superscript, the numbered reference — to launder a claim the system never actually checked. Users trust it more than plain text, and that's exactly backwards when the citation itself might be hallucinated.

This post is about the engineering it took to make citations mean something: what the model asserts, where in the source it came from, and whether the system can prove the link between the two. I think of it as explainable AI in the only form that survives contact with skeptical users — not saliency maps or confidence prose, but a footnote that opens the actual page and highlights the actual sentence.

Why post-hoc citation fails

The naive design — and the first one everybody reaches for — is to generate the answer, then attribute it afterwards: embed the answer sentences, find the nearest source chunks, attach them as citations. It fails in two distinct ways, and both failures look fine in a demo.

Failure one: retrieval theater. The nearest chunk is about the same topic as the sentence, so the similarity score is high, but it doesn't actually support the claim. A sentence saying a device tolerates 40 °C gets attributed to a paragraph discussing storage temperature. Semantic proximity is not entailment. Nobody notices until a domain expert clicks through.

Failure two: citation hallucination. If you instead ask the model to emit its own citations in-band — "cite as [doc, page]" — it will happily produce well-formatted references to pages that don't contain the quote, or don't exist. The model is completing a plausible pattern, and citations are just another pattern.

The fix for both is the same architectural decision: the citation must be produced by the same mechanism that produces the text, and then verified by a mechanism that isn't a model at all. That's what provider-native citation grounding gives you. Several providers now accept source documents as structured input and return output spans linked to source spans — character offsets into a document you supplied, not free-text references the model invented. The link is part of the generation contract, not a guess bolted on afterwards. Post-hoc attribution asks "what does this sentence resemble?" Native grounding asks "what was this sentence generated from?" — and only the second question has a checkable answer.

Native grounding alone still isn't proof. It narrows the failure surface enormously, but the output is only trustworthy after validation — which is why grounding is one stage of a pipeline, not the pipeline.

One facade, four providers, capability gating

The platform routes every model call through a gateway that unifies Anthropic, OpenAI/Azure, Google/Vertex, and AWS Bedrock behind a single model facade. Callers ask for a capability profile, not a vendor SDK. This matters for citations because the providers are genuinely uneven: native citation grounding, streaming shapes, and structured-output modes differ per provider and per model.

The gateway holds a capability table and gates requests against it:

// Illustrative — the real table is per-model, not per-provider
const capabilities: Record<string, ModelCaps> = {
  "provider-a/model-x": { streaming: true, citations: true,  json: true  },
  "provider-b/model-y": { streaming: true, citations: false, json: true  },
  "provider-c/model-z": { streaming: true, citations: true,  json: false },
};
 
function route(req: AnswerRequest): ModelChoice {
  const need = req.requires; // e.g. ["citations", "streaming"]
  const eligible = models.filter(m => need.every(c => capabilities[m][c]));
  if (eligible.length === 0) {
    throw new CapabilityError(need); // fail loud — never silently degrade
  }
  return pickByPolicy(eligible, req);
}

The critical line is the CapabilityError. The tempting shortcut is to fall back: if the routed model can't do native citations, quietly switch to post-hoc attribution and ship the answer anyway. We made that illegal at the gateway level. A request that requires citations either lands on a model that can natively ground them, or it fails loudly and the caller decides what to do. Silent degradation is how you end up with two citation qualities behind one UI treatment — and the UI treatment is a promise.

Capability gating also future-proofs the facade. When a provider ships native citations for a new model, one row changes in the table and the routing pool widens. No client code moves, because no client ever saw a provider SDK — more on that below.

The evidence-first pipeline

Everything citation-shaped runs through four stages:

Gather assembles the candidate evidence. Our sources are mostly PDFs, and PDFs are where page anchoring is won or lost: a vision-LLM document-intelligence pipeline extracts content with page-anchored citations from the start, so every passage entering the pipeline already knows its document, page, and character offsets. If you lose the page number at extraction time, no amount of downstream cleverness recovers it honestly.

Prompt sends the passages plus the question through the gateway to a citation-capable model with native grounding enabled. The model's answer comes back with spans tied to specific source offsets.

Validate is the stage that earns the word "prove" in this post's title, and it has two layers — a mechanical one and a semantic one. I'll spend most of the rest of the post there.

Score decides whether the surviving evidence is sufficient to render an answer at all.

Page-anchored quotes and the click-to-source contract

The unit of citation in our system is not a document. It's not even a page. It's a quote at a page anchor:

{
  "type": "citation",
  "footnote": 3,
  "doc_id": "doc_8f3a91",
  "page": 12,
  "quote": "The unit operates in ambient temperatures from 0 to 40 °C.",
  "char_start": 1180,
  "char_end": 1242
}

That shape is a UX contract, and the front end holds us to it. Clicking footnote 3 opens the source viewer at page 12 of that document and highlights that exact sentence. There is nowhere to hide. A document-level citation ("see the manual") can be wrong invisibly; a highlighted quote on a rendered page is wrong visibly, immediately, to the exact person most likely to care.

This contract is what makes the first validation layer mechanical and cheap: does the quote actually appear on that page of that document? Normalize whitespace and Unicode, then check the string at the claimed offsets against the extracted page text. No model involved — it's substring verification. Any citation that fails is dropped before rendering, and the claim it supported is dropped with it. A claim that loses all of its citations does not get demoted to "uncited" — it gets removed. Uncited claims standing next to cited ones inherit unearned credibility from their neighbors.

The mechanical check catches the fabricated-reference class of failure completely: you cannot render a quote that isn't there. What it can't catch is a real quote supporting the wrong claim. For that we needed the second layer.

Validating claims as a graph: JSON-LD → RDF → SHACL

The insight that unlocked semantic validation: a grounded answer is secretly a small knowledge graph. Claims, evidence, documents, pages — nodes and edges with rules about what a well-formed graph looks like. So we made that structure explicit. The pipeline emits claims as JSON-LD:

{
  "@context": { "ev": "https://example.org/evidence#" },
  "@id": "ev:claim-42",
  "@type": "ev:Claim",
  "ev:text": "The device operates between 0 and 40 degrees Celsius.",
  "ev:supportedBy": {
    "@type": "ev:Evidence",
    "ev:sourceDocument": { "@id": "ev:doc_8f3a91" },
    "ev:pageNumber": 12,
    "ev:quote": "The unit operates in ambient temperatures from 0 to 40 °C."
  }
}

JSON-LD converts losslessly to RDF, and RDF has a W3C-standard constraint language: SHACL. Shapes declare what every claim must structurally satisfy, and a conformance check either passes or produces a machine-readable violation report:

ev:ClaimShape a sh:NodeShape ;
  sh:targetClass ev:Claim ;
  sh:property [
    sh:path ev:supportedBy ;
    sh:minCount 1 ;                # every claim carries evidence — no orphans
    sh:node ev:EvidenceShape ;
  ] .
 
ev:EvidenceShape a sh:NodeShape ;
  sh:property [ sh:path ev:sourceDocument ; sh:minCount 1 ; sh:nodeKind sh:IRI ] ;
  sh:property [ sh:path ev:pageNumber ; sh:datatype xsd:integer ; sh:minInclusive 1 ] ;
  sh:property [ sh:path ev:quote ; sh:datatype xsd:string ; sh:minLength 1 ] .

(These shapes are illustrative, not the production ontology — the pattern is the point.)

What does this buy over ad-hoc JSON checks? Three things I didn't fully appreciate until they paid off. Declarative constraints live outside code — domain rules tighten without redeploying the pipeline, and the shapes file is the documentation of what "well-formed evidence" means. Violation reports are structured data — a failed check says which node violated which shape at which path, which feeds scoring and debugging rather than a boolean "invalid." The ontology composes — when claims later reference typed entities, the same machinery validates that an entity of one type isn't cited as evidence for a property it can't have. That's the beginning of catching the "real quote, wrong claim" failure structurally instead of statistically.

SHACL validates the shape of the evidence graph; the quote-on-page check validates its content. Neither alone is sufficient. Together they mean a rendered footnote has survived both a semantic-structure check and a mechanical ground-truth check — which is as close to "prove it" as I know how to get without a human in the loop.

The honest "no evidence" state

The score stage looks at what survived validation and makes a product decision: is there enough evidence to answer?

When there isn't, the system says so. Not a hedged paragraph that answers anyway; a distinct terminal state, rendered differently in the UI: no supporting evidence was found in the provided sources for this question.

This was debated, because the no-evidence state feels like failure and demos badly next to a competitor that always answers. I've come to see it as the single most important feature of the trust layer. An always-answering system trains users to stop checking footnotes — the diligent behavior decays precisely because it's never rewarded. A system that visibly declines when evidence is absent teaches users that when it does answer, the footnotes are load-bearing. Trust compounds from honest refusals. Every fabricated citation you ship spends trust you cannot buy back with a hundred correct ones — the denominator users remember is "times it lied to me," not "overall accuracy."

The no-evidence state is also only possible because of the pipeline. A system without validation can't know it has no evidence — it just generates. Being able to say "nothing survived validation" is a capability, not a shortfall.

Streaming citations: NDJSON vs SSE

Grounded answers still have to stream — users won't stare at a spinner while four pipeline stages run. The gateway speaks two streaming dialects, chosen by consumer.

NDJSON — newline-delimited JSON over a plain chunked HTTP response — for service-to-service consumers: trivially parseable in any language, no event framing, pipes cleanly through queues and logs.

{"type":"delta","text":"The device operates between 0 and 40"}
{"type":"delta","text":" degrees Celsius."}
{"type":"citation","footnote":1,"doc_id":"doc_8f3a91","page":12}
{"type":"done","state":"grounded"}

SSE — Server-Sent Events — for browsers: native EventSource support, automatic reconnection with Last-Event-ID, named event types, and it plays well with proxies that dislike long-lived exotic connections. SSE also carries progress for the async batch jobs, so a long extraction run streams stage-by-stage status updates to the UI over the same mechanism as answers.

The interesting design problem is where validation sits in the stream. Text deltas arrive token by token, but a citation is only render-safe after the quote-on-page check. So the stream is two-phase by design: text streams immediately with placeholder footnote markers, and citation events arrive as each anchor clears validation, patching the placeholders live. If a citation fails validation mid-stream, a retraction event removes the marker and its sentence before the answer finalizes. The client contract is explicit: an answer is not grounded until the terminal event says so — everything before that is provisional rendering, and honest UIs style it that way.

The thin-client payoff

None of this would be maintainable if every client embedded provider SDKs. The decisive architectural move was a thin-client decoupling: all model and SDK work lives behind the gateway, and clients speak only the gateway's versioned APIs. The backend grew to roughly 120 endpoints across those versioned APIs — answers, document intelligence, async batch jobs, SSE progress — and that sounds heavy until you compare it to the alternative: four SDKs, their citation formats, and their streaming quirks re-implemented in every client, drifting independently.

Instead, provider churn is absorbed at exactly one layer. When a provider changes its citation span format, one adapter changes. When a new model earns its capability row, every client gains it simultaneously. The facade isn't indirection for its own sake — it's where the citation contract is enforced, and a contract enforced in one place is the only kind that holds.

What's hard, and what's next

Honest ledger of what still hurts:

• Cross-provider citation parity is unsolved. Native grounding differs in granularity and reliability across the four providers, and normalizing to one internal citation shape means the weakest provider defines edge-case behavior. Capability gating hides models that can't cite at all; it can't yet express "cites, but coarsely."
• Streaming retractions are honest but jarring. Watching a sentence disappear because its citation failed validation is the truthful behavior — and users find it unsettling. Buffering claim-plus-citation into joint units would trade latency for stability; we haven't found the right point on that curve.
• SHACL coverage is only as good as the ontology. Structural validation catches malformed evidence reliably; catching a well-formed citation attached to a subtly wrong claim needs richer typed relations than we've modeled. Widening that coverage is slow, deliberate work — every shape is a commitment.
• PDF extraction is the ceiling. Page-anchored citations are only as trustworthy as the vision-LLM extraction underneath. Complex tables and multi-column layouts still produce anchors that pass string checks while subtly misrepresenting structure. Ground truth has the same problem as everything else in this system: it has to be earned, not assumed.
• No numbers yet, on purpose. I haven't published precision figures for the validation layers because we don't yet have an evaluation harness I'd defend adversarially. Consistent with everything above: no claim without evidence — including claims about the evidence system itself.

The through-line of the whole project is that "explainable AI" stopped being a property of the model and became a property of the pipeline. The model was never going to prove anything. The system around it can.