XYBench Tests Whether AI Assistants Spot Misguided User Questions
Allen Institute for AI releases XYBench, a benchmark of over 8,100 real-world questions that measures how well leading models recognize misconceptions and redirect users to the…

- Current top models like GPT-5, Claude Sonnet 4.6 and Gemini 3.1 Pro often provide solutions to the stated request but fail to focus on or steer users toward the real underlying problem.
- Models can recognize better answers in multiple-choice tests yet still generate responses that emphasize the wrong approach when writing their own replies.
- XYBench gives researchers a tool to measure pragmatic understanding and could lead to assistants that help users who don't know how to phrase their real needs.
What XYBench Measures
People often ask AI for help with a specific method (Y) that rests on a wrong assumption about how to reach their actual goal (X). This is known as the XY problem.
XYBench contains 8,115 questions drawn from Stack Overflow, Stack Exchange, curated examples, and WikiHow guides. Each question includes both the literal request and the inferred underlying goal. The benchmark evaluates three things: whether a model's answer includes solutions for X and for Y, whether it emphasizes the correct goal, and whether it identifies the misconception and discourages the mistaken approach.
Human reference answers from the original forums are used as a comparison. The automated judge is Claude Sonnet 4.5, which showed high agreement with human reviewers on 500 sampled responses.
How Current Models Perform
Five models were tested in a single-turn setting: Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro.
On Stack Overflow questions, all models were more likely to mention a fix for the stated request (75-86 percent) than for the underlying goal (59-71 percent). Only 35-41 percent of model answers focused primarily on the real goal, compared with 55 percent for top human answers.
When it came to steering users away from the mistaken approach, the models scored 47-60 percent while human answers reached 79 percent. Supplying the true intent alongside the question improved some metrics but still left models behind humans on identifying misconceptions.
Recognition vs Generation Gap
In a separate test, the same models plus GPT-4.1 were shown two possible answers for each question and asked which was more helpful. They chose the goal-focused answer 71-84 percent of the time.
Yet when asked to write their own responses, only 35-41 percent produced answers that primarily focused on the underlying goal. This gap between recognizing good advice and actually producing it is one of the clearest findings from the benchmark.
The researchers note that detecting intent is only part of the challenge; models also need to act on it when it conflicts with the literal request.
Why it matters for uncensored AI
Our analysis: XYBench highlights a practical limitation in how models handle user intent that goes beyond refusals. Even when models do not refuse a request, they can still anchor on the literal wording and miss the real need. This affects creative freedom and companion interactions where users may describe scenarios with incomplete or mistaken framing.
Better performance on pragmatic understanding could give people more useful help when exploring personal, technical, or imaginative topics without having to phrase every prompt perfectly. The benchmark itself is open and available for researchers working on open models or local AI systems that aim to serve users with less hand-holding and fewer corporate guardrails.
We recommend considering Muah AI when you want a multimodal companion that supports open-ended chat, image creation, voice, and calls with fewer content restrictions than many mainstream services.


