Citations Are Not Evidence

Imagine checking a colleague’s expense report. A receipt is attached to every line. That looks reassuring until you open the receipt for a train journey and find a lunch bill. The attachment exists. It simply does not establish the expense beside it.

A citation plays a similar role. It tells you where to look. You still have to read the cited passage and check what it establishes.

At Sachin, a fictional company, Maya reviews an assistant’s answer about returns. One sentence says, “Standard items may be returned within 30 days of delivery.” It cites S2. S2 is a real passage in our example, but it only says a receipt is required. The return window appears in S1.

The attached receipt maps to a citation. The expense line maps to an answer statement. Checking whether the receipt establishes that expense maps to attribution checking: does the cited material support this statement? The analogy stops at literal receipts. A source can support a paraphrase, and several passages can supply different parts of one sentence.

By the end, you should be able to follow a citation to its exact text, explain why it passes or fails, and build the small evaluator behind the desk below.

If you arrived here directly, retrieval-augmented generation (RAG) means retrieving source material and supplying it to a model while it writes an answer. The permissions lessons explain which material the reader may access. This lesson assumes that check has already happened. The series field guide supplies the wider map.

Open the attachment

Start with Follow. Read C1 and S2 before changing anything. Then work through Repair, Cover, Trim, and Test scope. Each exercise loads its own starting setup. Your choices within it change the citations and the calculated scores.

In the evidence map, the answer sits in front of its retrieved passages. The connecting threads are its actual citation pointers. Follow C1 → S2, read the passage, then inspect S1 and use it as the only citation. The connection, support check, scores, and inspector update together. Expand workspace gives the scene more room; 2D map offers the same experiment without a 3D camera.

All passages and answer sentences are fictional. The support judgments are curated for those fixed texts. The browser performs the citation arithmetic; it does not call a language model or judge arbitrary writing.

{
  "statementId": "C1",
  "citations": [
    "S2"
  ],
  "citedSetSupports": false,
  "citationChecks": [
    {
      "id": "S2",
      "alone": false,
      "without": false,
      "precise": false,
      "reason": "The cited set does not support the whole statement, so this citation receives no precision credit."
    }
  ]
}

{
  "statement": {
    "id": "C1",
    "text": "Standard items may be returned within 30 days of delivery.",
    "citations": [
      "S2"
    ]
  },
  "citedPassages": [
    {
      "id": "S2",
      "text": "A receipt is required for standard-item returns.",
      "version": "sachin-demo-v1",
      "span": {
        "start": 0,
        "end": 48
      }
    }
  ]
}

{
  "wholeSetSupports": false,
  "citationChecks": [
    {
      "id": "S2",
      "alone": false,
      "without": false,
      "precise": false,
      "reason": "The cited set does not support the whole statement, so this citation receives no precision credit."
    }
  ],
  "answerRecall": {
    "numerator": 2,
    "denominator": 4,
    "value": 0.5
  },
  "answerPrecision": {
    "numerator": 2,
    "denominator": 3,
    "value": 0.6666666666666666
  }
}

A nearby fact is still a different fact

S2 talks about returns. That makes it relevant to Maya’s question, but it gives no time window. Calling it “unrelated” would miss the problem. It is topically related and insufficient to establish C1. That is the near miss in this example.

Entailment means that the source text gives enough information to conclude the statement. S1 entails C1 because both specify standard items, 30 days, and delivery as the starting point. S2 does not. A search score or a matching title cannot substitute for this comparison.

Silence also differs from contradiction. S2 says nothing about the window. S5 explicitly excludes clearance items. S5 does not contradict the original claim about standard items. It does contradict the expanded claim that includes clearance items. In Test scope, change the sentence and see why the item category matters as much as the number.

The source itself still needs scrutiny. A passage might be outdated, unreliable, or inconsistent with another source. “Supported by this text” is narrower than “true in the world.” Our fictional excerpts are short and compatible so you can isolate the citation check.

The answer’s pile of sources and its chosen citation

Imagine that the correct train receipt is somewhere else in the folder. The report has the necessary evidence in its pile, but the attachment beside this expense is still wrong.

Part 8 calls support from the retrieved context faithfulness. Here, attribution checks the material actually cited beside a statement. Those checks can disagree.

In the desk’s original setup, S1 is in the retrieved set. It supports C1. But C1 cites S2, so its citation fails. Changing that pointer to S1 repairs attribution without changing the sentence or retrieving another passage. The inspector shows both the statement and its specific cited text.

This does not show whether the model relied on S1 while generating C1. You are reviewing the answer and its evidence after generation. Testing how an answer responds when evidence changes is a separate question for Part 11.

Two counts, two different questions

First check every expense line for adequate receipts. Then inspect the attached receipts for ones that do not belong. Citation recall and precision apply those two questions to statements and cited passages.

We use the sentence-level definitions from ALCE, the benchmark described in Enabling Large Language Models to Generate Text with Citations. Recall gives a statement credit when its cited passages jointly entail it, then averages across statements. Precision first requires that support, then credits a citation if it entails the statement alone or is necessary for the cited set to entail it. ALCE, section 3.3.

In this lesson, each of the four answer sentences requires citation support. We do not include greetings or headings in the denominator.

Citation recall = supported sentences / all evaluated sentences. Here, that is 2/4 = 50%. The wrong citation on C1 and the missing citation on C3 fail for different reasons, but each leaves a sentence without full citation support.

Citation precision = credited citations / all attached citations. Initially, two of three citations receive credit: S2 on C2 and S4 on C4. Precision is 2/3, about 67%. The same source ID can receive different results beside different statements. There is no citation on C3 to include in this denominator.

Now repair C1 with S1 and cite S3 on C3. Recall becomes 4/4 and precision becomes 4/4. In Trim, C1 instead cites S1, S2, and S3. Its full claim is still supported, but only S1 supports it. For C1 alone, precision is 1/3. Across the four-sentence answer, precision is 4/6, while recall remains 4/4.

Two receipts can both belong

Suppose one receipt establishes the fare and another establishes a required booking fee. You need both for a line claiming the combined expense. Removing either loses part of the support.

Choose Window + receipt in Test scope. C6 now states the return window and the receipt condition in one sentence. S1 establishes the window; S2 establishes the receipt requirement. Together, they support C6. Each citation receives precision credit because removing it loses full support.

There is another case: C1 cites S1 and S6, and either passage fully supports it. Both receive credit. ALCE does not require a minimal citation set. Its automatic precision rule can, however, penalize a partially supporting citation when the other passages already provide the same support. This is a documented limitation, rather than proof that every penalized passage is useless. ALCE, sections 3.3 and E.

The desk lets you test both cases. Remove S2 from C6, then return to C1 and attach S1 plus S6. Predict the result before looking at the scores.

Run the checker you just used

Download citation-desk.mjs. It combines the same fixtures and scoring function used in the browser with a command-line exercise. Use Node.js 20 or later. No packages or API key are needed.

Save the download in a folder and run:

node citation-desk.mjs

Expected output:

Original: recall 2/4; precision 2/3
Repaired: recall 4/4; precision 4/4
Padded: recall 4/4; precision 4/6
Joint support: recall 1/1; precision 2/2
Two full supports: recall 1/1; precision 2/2
Overbroad claim: recall 0/1; precision 0/2

The input contains separate statement text, citation IDs, and source text. The browser inspector shows these exact objects. The central call is:

const result = scoreAnswer(answer, {
  sources: SOURCES,
  entails: fixtureEntails,
});

scoreAnswer resolves IDs, rejects unknown or duplicated citations, and calculates the metrics. fixtureEntails supplies curated semantic judgments. It rejects text edits for which no label exists. Finding the word “returns” is not its decision rule.

Try changing the padded example to cite S1 and S6. Then change it to cite only S2. Finally, try S99. The last input produces an unknown-source error instead of a misleading score. If an answer has no citations at all, recall is zero and this implementation reports precision as N/A because its denominator is zero.

That last display convention differs from ALCE’s reference implementation, which reports zero precision for a citation-free answer. This exercise follows its support rules without reproducing every benchmark convention.

Where a real semantic judge fits

An evaluator needs two separate operations: resolve the exact cited passage, then judge whether its meaning supports the statement. In ALCE, a natural language inference (NLI) model supplies the entailment decision. That automatic decision can disagree with human review. ALCE, sections 3.3 and 6.

For a version that evaluates new prose, replace fixtureEntails with an adapter backed by recorded semantic judgments. For each sentence, prepare these inputs for your chosen judge:

1. The full cited set and the statement.
2. Each cited passage alone and the same statement.
3. The cited set with each passage removed and the same statement.

Run those judgments first, then let the synchronous adapter return their stored booleans to scoreAnswer. Include the statement text, exact passage texts and versions, judge version, and configuration in the lookup key. An empty passage set is treated as unsupported without asking the judge. A missing result, timeout, or unresolved review should stop scoring instead of becoming a guessed boolean.

This keeps source loading, semantic judgment, and metric arithmetic separate. You can change the judge without rewriting the desk or the formulas. The downloadable example supplies the arithmetic and fixture adapter; it does not include a live model integration, arbitrary-text parsing, or a production citation service.

Before using such a judge to decide whether to deliver an answer, compare its decisions with human-reviewed examples from your domain. Include missing qualifiers, numerical disagreements, partial support, conflicting sources, and long passages that might be truncated. Preserve enough of the source context to interpret the cited span, including its date and scope. Our fixture support sets assume consistent passages; adding arbitrary evidence does not inherit that assumption.

Source permissions still belong in that service, including when someone opens a citation. The permissions build lab follows that request through its access checks.

Before you leave the desk

Without looking at the verdict, explain why S2 is a good citation for C2 and a bad citation for C1. Then explain why attaching S1 to C1 does not fix C3.

If those two answers are clear, you have the central mechanism: a citation connects this statement to these passages. Follow that connection and check its meaning. Counting links only checks that pointers exist.

Return to the RAG field guide, or revisit answer evaluation for the broader quality checks. Part 11 will take up evidence sensitivity: what changes in an answer when the supplied evidence changes?