Faithfulness Benchmark Probes If Open-Weight LLMs Override Injected Geographic Context
An arXiv preprint dated 30 September 2026 presents a network-based spatial context retrieval pipeline using open data and open-weight models to test whether LLMs reason from…
- Resistance to planted false premises varies more strongly by model family and generation than by scale across sixteen tested configurations.
- Brief-reading competence appears partly independent from conventional world-correctness scores in the faithfulness evaluation.
- The open pipeline and benchmark use pedestrian street networks, OpenStreetMap features and GHS-POP data for reproducible grounded geographic reasoning.
Core Research Question
The preprint abstract notes that large language models encode latent geographic knowledge yet reason poorly over space when queried from coordinates alone. Useful behavior appears only when structured spatial context is supplied.
The work examines whether models actually reason from the injected context or override it with training-derived recall. It introduces a pipeline that defines surroundings via the pedestrian street network reachable on foot.
Pipeline and Benchmark Design
The open pipeline retrieves features from OpenStreetMap and the GHS-POP population grid, computes indicators over the network catchment, and injects a compact spatial brief into prompts.
A faithfulness benchmark labels every model claim by source and correctness. It includes probes with planted false premises that the supplied brief refutes. This approach highlights limitations of standard world-correctness scores and single-shot evaluation.
Evaluation Scope
Sixteen open-weight model configurations from Qwen, Gemma (two generations) and Llama families were evaluated. These spanned four size classes and, where available, thinking and non-thinking modes.
Tests covered three contrasting cities with ten resampling seeds per case. Results indicate that resistance to false premises and brief-reading competence behave as partly separate dimensions. This evidence is limited to the preprint abstract; the full paper is not available here and no independent replication exists.
Released Resources
The authors release the implementation, spatial briefs, model outputs and claim-level labels as a reproducible workflow on GitHub. The work supports transparent evaluation of how open-weight models attend to provided context versus memorized knowledge.
