When Does Graph Expansion Help Personal-Corpus Retrieval? A Query-Class-Stratified Evaluation over Human-Curated Wikilink Graphs: Protocol and First Execution with LLM-Judged Relevance (v0.3)

Anton Dziatkovskii · ORCID 0000-0001-7408-3054

Preprint, 3 October 2026. Version of record: Zenodo, DOI 10.5281/zenodo.23113066. License CC BY 4.0.

Full text (PDF, 7 pages, 296 KB)DOICode: sqlite-graph-memory

Abstract

Version 0.3 of a preprint on graph-augmented retrieval over human-curated wikilink graphs (personal knowledge bases). v0.2 (doi:10.5281/zenodo.22639718) described the system, a query-class-stratified evaluation protocol (entity, theme, bridge, compare, temporal) and N=10 pilot A/B pairs. v0.3 reports the first full execution of the protocol: 38 queries, 509 pooled documents, graded 0-2 under blinding.

The relevance labels were produced by an LLM judge (Claude Opus 5.5), not by a human annotator. Label reliability is bounded by a 14-query, 178-document subset re-graded by two LLM judges from other vendors (GPT-5.6, GLM-5.3): Cohen's kappa 0.32-0.42 unweighted (0.54-0.64 quadratic-weighted) between the primary judge and the others, 0.57 between the two external judges.

Under the primary labels, reranker-guarded 1-hop wikilink expansion is inert or slightly harmful: mean delta nDCG@12 = -0.036 over 38 queries (12 harmed, 6 helped, 20 unchanged); no stratum is significant after Holm correction; the v0.2 hypothesis that thematic queries benefit is not supported (theme: -0.020, 95% CI [-0.041, -0.002], n=16). The sign of the small per-query effects depends on the judge. Results are descriptive: one corpus, one user, 4-16 queries per stratum.

Keywords

LLM relevance judgments · benchmark · evaluation protocol · graph RAG · information retrieval · inter-annotator agreement · personal knowledge management · retrieval-augmented generation · wikilinks

How to cite

Dziatkovskii, A. (2026). When Does Graph Expansion Help Personal-Corpus Retrieval? A Query-Class-Stratified Evaluation over Human-Curated Wikilink Graphs: Protocol and First Execution with LLM-Judged Relevance (v0.3). Preprint. Zenodo. https://doi.org/10.5281/zenodo.23113066

@misc{dziatkovskii2026graphragbenc,
  author    = {Dziatkovskii, Anton},
  title     = {When Does Graph Expansion Help Personal-Corpus Retrieval? A Query-Class-Stratified Evaluation over Human-Curated Wikilink Graphs: Protocol and First Execution with LLM-Judged Relevance (v0.3)},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23113066},
  url       = {https://doi.org/10.5281/zenodo.23113066},
  note      = {Preprint, CC BY 4.0}
}

One text, one DOI: the PDF on this page is the Zenodo file byte for byte. Please cite the DOI.