Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models
Abstract
Where should the question go in a vision-language model (VLM) prompt: before the image or after it? Intuition says before: knowing what is asked should tell the model where to look. Yet across visual question answering benchmarks, question-first prompting consistently underperforms the image-first ordering recommended for frontier VLMs, a phenomenon we term the question-first paradox.
A vision-language model reads system, image, and question tokens as one sequence and answers from the final position. We show that where the question is placed changes the answer. Placing the question before the image (question-first, STI) makes the model commit early to an image-anchored, often wrong answer, even though its intermediate representations perceive the scene correctly. Placing the question after the image (question-last, SIT) recovers the correct answer. Re-presenting the image and question after the image (image echoing, SITIT) resolves the paradox with no training and gives the best accuracy across four benchmarks and two model families. We summarize the primary results and trace the answer forming layer by layer with a per-layer logit lens.
1 The Question-First Paradox
A prompt is three sections in token order: the System message, the Image (many visual tokens), and the Task/question. Reordering these sections, with content held fixed, changes what the decoder answers. We report five orderings.
| STI | System · Task · Image | question-first — the paradox |
| SIT | System · Image · Task | question-last — the baseline |
| STIT | System · Task · Image · Task | question echo (ours) |
| SITIT | System · Image · Task · Image · Task | image echo (ours) — best |
| SITITrev | System · Image · Task · Ī · Task | image echo, 2nd copy reversed |
Table 1: The five prompt orderings reported
here. Ī denotes the reversed second image copy.
2 Primary Results
Group accuracy on NaturalBench and Winoground; answer accuracy on POPE and RF20. Question-first (STI, in red) is the weakest ordering on the compositional benchmarks; image echoing (SITIT / SITITrev) is the strongest (best per row in green). Reversing the second image copy (SITITrev) retains almost all of the gain, showing the effect comes from a second look, not from copying tokens.
2.1 Qwen3-VL-8B
| Benchmark | metric | STI | SIT | STIT | SITIT | SITITrev |
|---|---|---|---|---|---|---|
| NaturalBench | group acc. | 0.270 | 0.350 | 0.350 | 0.374 | 0.374 |
| Winoground | group acc. | 0.223 | 0.318 | 0.375 | 0.403 | 0.410 |
| POPE | acc. | 0.870 | 0.891 | 0.884 | 0.889 | 0.889 |
| RF20 | acc. | 0.713 | 0.808 | 0.793 | 0.816 | 0.794 |
Table 2: Qwen3-VL-8B. STI trails SIT by 8–10 points on NaturalBench and Winoground with identical content; echoing recovers and exceeds the baseline.
2.2 Gemma-3-27B
| Benchmark | metric | STI | SIT | STIT | SITIT | SITITrev |
|---|---|---|---|---|---|---|
| NaturalBench | group acc. | 0.232 | 0.226 | 0.253 | 0.255 | 0.252 |
| Winoground | group acc. | 0.212 | 0.207 | 0.253 | 0.318 | 0.297 |
| POPE | acc. | 0.840 | 0.845 | 0.838 | 0.834 | 0.835 |
| RF20 | acc. | 0.679 | 0.651 | 0.709 | 0.702 | 0.696 |
Table 3: Gemma-3-27B (4-bit, single GPU). The ordering effect and the echoing fix reproduce in a second model family.
3 Perception Steering: STI Identifies the Answer, SIT Misses It
Projecting each image patch's hidden state through the output embedding decodes it to a vocabulary token; overlaying that token on the patch shows what the model sees. We choose clear, low-token NaturalBench cases whose answer is Yes — the queried thing is genuinely in the image — so a patch that decodes it is correct-answer evidence, not a hallucination. In every case, question-first (STI) steers perception to the question: every patch that decodes a correct-answer token — the action and related objects that confirm the answer (racket, glove, climbing, guitar, …) — is boxed in green, while question-last (SIT), where the question comes after the image and cannot steer, decodes far fewer, boxed in red. Question-first (STI) identifies the tokens that answer the question; question-last (SIT) does not — the answer verdict of each ordering is printed under its column. Press Play under a screenshot to expand the two per-layer animations for that same image side by side: each frame decodes one transformer layer.
“Is the adult in the image chasing a child?” ground truth: Yes
“Is the person wearing glasses?” ground truth: Yes
“Is there someone holding a tennis racket?” ground truth: Yes
“Are the people actively climbing?” ground truth: Yes
“Is the bird in the image surrounded by salt grass?” ground truth: Yes
“Is the musician playing a jaguar electric guitar?” ground truth: Yes
“Is the bicycle trick being performed on a wall?” ground truth: Yes
“Is the main action about catching a ball?” ground truth: Yes
“Is the main action of the dog pawing at something on the ground?” ground truth: Yes
4 The Fix: Image Echoing
Better perception is wasted if the answer still reads out wrong. Re-presenting the (image, question) after the image (SITIT) gives the decoder a second, adjacent look and recovers the correct answer, with no training. In each case below question-first (STI) answers wrong and image-echoing (SITIT) fixes it. Press Play to compare STI (wrong ✗) vs. SITIT (correct ✓) layer by layer.
“Is the adult in the image chasing a child?”ground truth: Yes
“Is the man performing on a ramp?”ground truth: Yes
“Is the firetruck parked in a driveway outside the garage?”ground truth: Yes
“Is the dog chasing a person?”ground truth: Yes
5 Repository Guide
The project's code ships the analysis scripts and the per-run result JSONs for
the primary orderings; model weights and raw image datasets are external. Each
benchmark runner takes an --order flag
and writes <dataset>/results/<model>__<order>__results.json,
the exact files that populate the tables above. The order-aware prompt is built
once in model_manager.py and shared across models, so a single code
path produces STI, SIT, STIT, SITIT, and SITIT_rev. The per-layer animations on
this page are rendered by the logit-lens overlay code from the same runs.
| Primary result / artifact | Script(s) |
|---|---|
| Position ladder / paradox (NaturalBench) | naturalbench_eval.py, gemma_eval.py |
| POPE / Winoground / RF20 | pope_eval.py, winoground_eval.py, rf20_eval.py |
| Image-echo, reversed 2nd copy (SITIT_rev) | *_sitit_reverse.py, reverse_image_hooks.py |
| Order-aware prompt builder (all orderings) | model_manager.py, constants.py, utils.py |
| Per-layer / per-patch logit lens (the animations) | logit_lens_overlay.py, logit_lens_runner.py, sitit_stit_gif_gen.py |
| Result and figure viewer for every analysis | logit_lens_app.py + *_browser.py |
Table 4: Where each primary result comes from.
Run e.g. python naturalbench_eval.py --order STI. See
README.md for the complete code→paper map and setup.