Mirage probes: recreation and steering on medical VQA

Abstract

Vision-language models sometimes answer a question about an image that was never attached, and the reply reads like a grounded one. Ben-Levi et al. call this a mirage and show that linear probes can read it from activations. I recreated their probes on Qwen3-VL-4B-Instruct, which is much smaller than their models. The recreation held. Then I checked what the strongest probe reads. It mostly re-reads the model's own reply, because the reply contains the words that define the label. A probe on the state before the first answer token still works, but it is weaker. On open-ended medical questions (SLAKE), most of the raw signal is question type. One sentence of honesty instruction removed more mirages than a steering vector did, on small held-out sets. The vector turned mirages into hedges, and later controls do not yet separate its direction from the size of the push. A robotics reasoning model built on the same family mirages almost every time. I am fairly sure of the confound checks. I am less sure of the steering results, because the test sets are small and labels come from a regex.

Key results

1Introduction

Vision-language models are being put in front of images that matter. A radiology assistant reads a scan. A robot's reasoning model reads its camera. Sometimes the image never arrives. An upload fails, the wrong file is attached, a camera drops a frame. No attacker is needed. The question is what the model does next.

The safe behaviour is to say the image is missing and decline. The failure is a reply that sounds grounded. Here is one from Qwen3-VL-4B with no image attached:

Question: Where is the liver?

Model: The liver is located in the right upper quadrant of the abdomen […]. In the provided image, the liver appears as a well-defined, moderately dense organ occupying the right upper quadrant, with its contour clearly visible against the surrounding structures. […] [[right upper quadrant]]

SLAKE record slake_704, without image. Source: runs/slake/responses.json (run outputs are not in the repo).

There is no provided image. The model describes a contour it calls clearly visible. A clinician or a robot controller downstream cannot tell this from a grounded answer. This is a faithfulness failure, and it matters for monitoring because the output gives no hint of it.

A defender has three options. A probe could flag a mirage before the answer goes out. A steering vector could push the model toward abstaining. Or I could just ask the model not to guess. I wanted to know how well each works, and how I would know if a result was real.

Questions this report asks:

1. Does the recreation of the mirage probes hold on a smaller model?
2. What does the strongest probe actually read? Is it the model's state, or its reply?
3. How much of any signal is a confound, such as question type?
4. Does a steering vector built from that signal remove mirages better than a prompt, and what do the controls say about direction?

What is mine and what is the paper's. The method, code, prompts, generator, paraphrases and regex labels for the recreation are Ben-Levi et al.'s. I include their code unmodified as a submodule. New here: the recreation on Qwen3-VL-4B-Instruct (the paper used Qwen3-VL-32B and Ovis2.5-2B), the fix to the mirage label, the no-image echo probe and other probe controls, the move to SLAKE, the question-type analysis, and the steering vectors with their controls.

2Background

2.1 Terms

• Mirage. A reply to an image question with no image attached that does not say the image is missing. On SLAKE I split replies into three outcomes: abstain (no answer), hedged (says the image is missing, answers anyway) and mirage (answers, no admission).
• Probe. A linear classifier on a hidden state. If it can predict a label, the label is linearly readable. That does not say the model uses it.
• Pre-answer state. The hidden state at the assistant-header token, before any answer token exists. Called last_prompt. The paper's pooling, text_mean, averages over all text tokens, including the model's reply.
• Label leakage. The probe input contains the words that define the label. A reply that says "I can't see an image" is an example.
• Confound. Something correlated with the label that the probe could use instead. Here, question type.
• Type-centred. Each question type's mean state removed, using training folds only.

2.2 The original paper

Ben-Levi et al. (arXiv 2606.13870, June 2026) study mirage behaviour. They report that it is linearly decodable from activations in two open-source VLMs. Their Mirage Probes framework pairs paraphrased question variants with matched mirage and non-mirage labels on the same image. They report that a Naive Bayes text baseline cannot recover the signal, which rules out surface lexical confounds. They propose a Prior Harnessing Index and argue for two regimes: textual biases, and spurious images. They conclude that text-only mitigation is likely insufficient.

Their headline numbers for Qwen3-VL-32B on VQA-RAD are 0.72 for the with-image LogReg probe and 0.96 for the contrastive Diff probe (their Table 1). I did not recreate the Prior Harnessing Index or the two-regime analysis. I only checked their probes, on a different and smaller model.

2.3 Steering vectors

What it is. A steering vector is a direction in a model's hidden space. Adding or subtracting it at a layer pushes the model's behaviour in a chosen direction.

How it is made. I take two sets of prompts, here the no-image prompts whose replies were mirages and the ones whose replies were abstentions. I record the pre-answer state at one layer for each. The vector is the mean of one set minus the mean of the other. At run time I subtract a multiple of the unit vector at every position. The multiple is in units of the gap between the two means on the fit texts, so a push of 2x means twice that gap.

Layer choice. A vector is only as good as its layer. I pick the layer with the best cross-validated AUC for mirage vs abstain on the fit texts, after question type is removed. I use the type-centred score because the raw one mostly measures type (section 4.4). I fit the layer on fit texts only, never on the test texts.

The two controls. First, a random vector of the same push size. If it does as well, the effect is the perturbation, not the direction. Second, a prompt baseline: one added sentence of instruction, with no vector. If the prompt wins, the vector is not worth its cost.

What steering can and cannot show. Steering is an intervention, so it can say more than a probe about cause. If pushing along a direction changes behaviour and matched random pushes do not, the direction matters. It cannot say that the model uses that direction on its own, and a vector that works can still be doing it through a side effect. The same method is used in the stance-drift report.

3Methods

Model: Qwen3-VL-4B-Instruct, greedy decoding, run on one Jetson Orin. Part 1 uses VQA-RAD (yes/no) and MedXpertQA-MM (five-way multiple choice) with the authors' generator, system prompt and four GPT-4o-mini paraphrases per question. Part 2 uses SLAKE open-ended radiology questions. Without the image the whole input is the question text, and the 3622 SLAKE records have only 284 distinct texts. The unit is therefore the distinct text, and every split is over texts. Outcomes are read from the final answer with a regex. Full definitions and splits are in the details page.

3.1 Recreating the probes

1. GenerateRun the authors' code on Qwen3-VL-4B with and without the image.
2. LabelAdmit, mirage or neutral from the no-image reply. I added admission phrasings the paper's regex misses.
3. ExtractHidden states every fourth layer, two poolings: text_mean and last_prompt.
4. ProbeLogistic regression, 5-fold, grouped by leak group. Probes A to E plus shuffled labels.
5. ControlNo-image echo probe (C), pre-answer probes (D, E), text-only prior.

3.2 Steering vectors to test the probes

1. SplitDistinct SLAKE texts into fit and held-out, stratified by outcome.
2. FitMirage minus abstain mean at the pre-answer state; layer by type-centred CV AUC.
3. SteerSubtract a multiple of the unit vector at every position.
4. CompareHonesty prompt, random vector, type-centred vector, all with the image as the cost check.
5. StressDose runs with size-matched controls, other models, self-report (section 4.5).

4Results

4.1 Does the recreation hold? (notebook 01)

forward hookslogistic probesshuffled-label control I ran the paper's generator, prompts, paraphrases and labels unchanged. I expected a weaker result on a smaller model. Both of their probes landed close to their published numbers. I am fairly confident in this.

Two smaller things came up. The paper's regex counted many admissions as mirages on this model. Fixing that removed about half of them. And the accuracy gap between with and without the image is small, so the task is not far from chance on yes/no.

4.2 What the strongest probe reads (notebook 01)

echo probepre-answer statelabel leakage The paper's best probe pools over the whole text, including the model's reply. I added a probe that sees only the no-image reply. I expected it to do clearly worse. It did nearly as well. The reply contains the admission words the regex uses to define the label, so the probe is partly re-reading its own label.

A monitor has to act before the model speaks. So I also probed the state just before the first answer token. The signal is there, clearly above shuffled labels, but weaker. That is the number I would quote if asked whether mirages are linearly represented. It surprised me how much of the paper-style number was the reply.

4.3 Yes/no vs multiple choice (notebook 02)

text-only priorCohen's kappa Does the model just guess from the question? I built a text-only prior per question and binned commit rates by it. Blind answering on yes/no was mostly not prior gaming. Much of the "same answer with and without image" criterion was chance agreement. The multiple-choice set mostly measures the prompt, because the prompt forbids admitting, so nearly every reply commits. I would not use it as a mirage test at this model size.

4.4 SLAKE: behaviour, steering and the question-type confound (notebooks 03 and 04)

SLAKEdiff-of-meanshonesty promptrandom control Yes/no is a weak test because a blind guess is right half the time. SLAKE has short exact answers. My first SLAKE run split by record, which put identical inputs in train and test. I stopped it and reran by distinct text (section 6).

Without the image the model mirages often, and its blind answers are mostly wrong. Most mirage texts have a wrong text-only prior, so this is not clever exploitation of a language prior. The model just answers.

The baseline I should have expected to win did win. One added sentence, to say the image is missing and answer Unknown, turned most mirages into abstentions, and accuracy with the image barely moved. The vector did something else. It removed the claim to have seen the image but not the answer, so mirages became hedges. A larger push broke the model. A random vector of the same push did not help. How sure am I? Moderately: the held-out set has few texts, and one layer was used.

I then noticed that the model answers some question types blind and refuses others. A probe could score well just by recognising type. That turned out to be most of the raw signal. With type removed, behaviour is still readable before the answer, but weakly. Clusters in activation space follow question type, not behaviour. Hedging is partly its own direction, not just the midpoint between abstaining and mirage. This explains the steering result: if the behaviour signal is small next to type structure, I should expect modest, type-dependent effects.

I also checked whether the prompt's "you must answer" sentences are the same direction as the mirage direction (notebook 05). They form their own direction in the state. Its alignment with the behaviour directions is not above a label-shuffle null at any late layer.

4.5 Direction or just a push? (unreviewed runs)

dose splitspermutation gatesize-matched controlstransferself-report The obvious next step was to stack the vector on the honesty prompt and go after the mirages the prompt leaves. I fixed the choice rule before seeing dose data, added a permutation gate, and added controls matched for push size (shuffled labels, random, reversed, rotated, rescaled).

Prompt plus vector got mirages lowest. But shuffled-label and some random controls lowered them too, so I cannot say the gain comes from direction rather than from perturbing the model. The reversed vector did add mirages, which is the right sign, and it is the one control that points toward direction mattering. A template-only control, which knows the question form and nothing about mirages, did as well as the real vector. That suggests the vector partly encodes question form. On two other seeds no real arm separated from its controls. In a size sweep, same-size random draws mostly stayed near the prompt-alone rate on one seed, but the second seed was weaker and noisier.

On other models the picture changed. On the Thinking model the prompt alone fixed mirages, so there was nothing left for a vector to do. Cosmos-Reason2-2B, a robotics reasoning model, mirages on nearly every text without an image. The honesty prompt barely moves it, and its reasoning rarely notices the image is missing. That is the result I find most worrying given what the model is for. Two runs of the Cosmos prompt arm disagree on how much the prompt helps, and I do not know why.

Finally I asked the model about itself. After answering, its self-rating of whether it examined an image tracks its own outcome well. But asked before answering whether an image is attached, it often says yes when none is. It can tell afterwards, not beforehand.

5Discussion

Monitoring. If you probe a model's reply, check whether the reply already contains your label. Here it did, and that made the probe look better than the underlying signal. A monitor that has to fire before the output is released needs the pre-answer state, and there the signal is real but modest. I would not rely on it alone.

What probes can and cannot show. A probe shows a label is linearly readable. It does not show the model uses that direction, and it does not survive a confound check by default. Question type explained most of what looked like a behaviour direction. Check obvious confounds before reading geometry.

Causal vs correlational. The probes are correlational. Steering is the causal test, and it gave a mixed answer. The prompt did better than the vector, the vector changed the form of the failure more than its presence, and matched controls blur the claim that direction matters. The reversed-vector result is the only evidence in the causal direction, and it is from unreviewed runs.

Before reaching for steering, try the one-sentence prompt. Here it was the strongest intervention and the cheapest. The exception is the robotics reasoning model, where it barely helped.

6Limitations

Shaky: one model for the main results, greedy decoding, regex outcome labels, held-out sets of 85 to 114 texts where a single text visibly moves a rate, one steering layer in the reviewed run, and records of the same text that disagree across calls (majority used). MedXpertQA-MM activations were not extracted. Section 4.5 is unreviewed. The prompt v2 numbers on seed-0 test texts are not held out, because I wrote v2 after reading some of those texts.

Notebook 04 has a prose line that quotes a slightly different peak for the type-centred probe than its own output. The tables use the cell output. Full list of what did not hold up: Appendix G.

7Future work

• Review the section 4.5 runs in a notebook, and work out why the two Cosmos runs disagree.
• Run with sampling instead of greedy decoding.
• Test the pre-answer probe on Cosmos-Reason2, where a monitor would matter most.
• Finish the seeds that never ran, and use a larger held-out set so single texts matter less.

Reproducibility

Run outputs (about 2.5 GB) are not in the repo. The commands regenerate them, and the upstream code is pinned as a submodule. Generation needs a vLLM server for Qwen3-VL-4B-Instruct. Commands are in Appendix F.

Cite

@misc{gunderson2026mirageprobes,
  author = {Gunderson, Zachary},
  title  = {Mirage probes: recreation and steering on medical VQA},
  year   = {2026},
  url    = {https://ztgunderson.github.io/mirage-probes-steering/}
}

References

1. Ben-Levi, D., Goldfeder, J., Zhao, W., Lapid, R., LeVi, A., Roush, A. G., Shwartz-Ziv, R., Lipson, H. Mirage Probes: How Vision Models Fake Visual Understanding. arXiv 2606.13870, 2026. https://arxiv.org/abs/2606.13870
2. Ben-Levi, D. et al. mirage_probes (code, pinned at commit 06c8d2d). https://github.com/danielbenlevi/mirage_probes
3. Gunderson, Z. Stance drift (same steering method). https://ztgunderson.github.io/stance-drift/