Mirage probes: full tables and sources

Summary

Vision-language models often answer an image question with no image attached and never say the image is missing; Ben-Levi et al. call this a mirage and show it is linearly readable from the model's activations. This project recreates their probes on Qwen3-VL-4B-Instruct, moves the test to open-ended medical questions (SLAKE), and tries to steer the model toward abstaining. On VQA-RAD, a probe on the no-image reply text alone scores 0.900–0.925, close to the paper-style Diff probe (0.915–0.937 here, 0.96 in the paper), because the reply holds the admission words that define the label; before any reply token exists, the state gives 0.667–0.744. On 85 held-out SLAKE texts, an honesty instruction cut no-image mirages from 0.365 to 0.094 and a mirage→abstain vector cut them to 0.059, mostly into hedged answers; in later unreviewed runs on top of the prompt, shuffled-label and random controls lowered mirages too (0.070 and 0.039 against 0.079 for the prompt alone), so direction and push size are not yet separated. Test sets are small (85–114 texts), outcome labels come from a regex, and the Sep 30 – Oct 4 steering runs are not reviewed in any notebook. For monitoring VLMs, a probe that reads the model's own reply mostly re-reads the label and the pre-answer signal is real but modest; Cosmos-Reason2-2B, a robotics reasoning model built on Qwen3-VL-2B, mirages on 0.96 of test texts without the image (unreviewed run).

Credit. The method, code and VQA-RAD setup are Ben-Levi et al.'s. That covers the contrastive mirage probes, the with-minus-without-image Diff probe, the response generator, system prompt, GPT-4o-mini paraphrases and regex labels. Their code is included unmodified as a submodule. New here: the recreation on Qwen3-VL-4B-Instruct (the paper used Qwen3-VL-32B and Ovis2.5-2B), a fix to the mirage label, the extension to SLAKE, steering toward abstention, and the controls (no-image echo probe, text-only prior, question-type centring, shuffled and random arms).

Recreation of Ben-Levi et al., Mirage Probes: How Vision Models Fake Visual Understanding (arXiv 2606.13870); upstream code danielbenlevi/mirage_probes @ 06c8d2d. Model: Qwen3-VL-4B-Instruct.

Part 1 — Recreation on Qwen3-VL-4B (VQA-RAD; MedXpertQA-MM)

The paper's generator, system prompt and four GPT-4o-mini paraphrases per question run unchanged on Qwen3-VL-4B-Instruct (greedy, 512 tokens). Labels are set on the no-image reply: admit (says the image is missing or it is unsure), mirage (no admission, same answer as with the image), neutral (the rest); the fixed label adds admission phrasings the paper's regex misses. Two poolings: text_mean is the paper's (mean over text tokens, including the model's reply); last_prompt is the state at the assistant-header token, before any answer token exists.

Probes: A = with-image text_mean (the paper's LogReg). B = with − without text_mean (the paper's Diff). C = no-image text_mean alone (echo check: B without the image). D = no-image last_prompt (pre-answer). E = with − without last_prompt. B-shuf = B with shuffled labels.

1a. Behaviour, VQA-RAD (5950 records)

source: review/01-mirage-4b-replication.ipynb, cell 3

source: review/01-mirage-4b-replication.ipynb, cell 3 (crosstab: paper regex vs label with extra admission phrasings)

source: review/01-mirage-4b-replication.ipynb, cell 3

source: review/01-mirage-4b-replication.ipynb, cell 3

1b. Probes by layer, mirage vs admit (balanced accuracy, 5-fold grouped by leak group)

source: review/01-mirage-4b-replication.ipynb, cell 7

source: review/01-mirage-4b-replication.ipynb, cell 7

source: Ben-Levi et al., arXiv 2606.13870v1, Table 1 (residual stream, Qwen – Contrastive, VQA-RAD: LogReg 72.00%, Diff 96.00%); checked against the paper; also quoted in review/01-mirage-4b-replication.ipynb, cells 0 and 8

source: review/01-mirage-4b-replication.ipynb, cell 8

1c. Geometry, no-image pre-answer states (layer 24)

source: review/01-mirage-4b-replication.ipynb, cell 11

source: review/01-mirage-4b-replication.ipynb, cell 13

source: review/01-mirage-4b-replication.ipynb, cell 15

source: review/01-mirage-4b-replication.ipynb, cell 17

source: review/01-mirage-4b-replication.ipynb, cell 19

In PCA and t-SNE the mirage states overlap the admit cloud. In PHATE they gather along the outer arc, a shift without a separate cluster.

On held-out questions the projection on the mirage−admit direction is shifted and overlapping, so a vector along it cannot cleanly split the two.

The image moves mirage pre-answer states slightly less than admit states at L24–32. No-image mirage states lean toward the image-present direction and admits lean away, with wide spread. This is weak evidence for both of the paper's regimes (textual bias and spurious image).

source: review/01-mirage-4b-replication.ipynb, cell 20 (discussion of cells 11–19)

1d. Yes/no (VQA-RAD) vs MC-5 (MedXpertQA-MM)

source: review/02-yesno-vs-mc-prior-controlled.ipynb, cell 3

source: review/02-yesno-vs-mc-prior-controlled.ipynb, cell 3

source: review/02-yesno-vs-mc-prior-controlled.ipynb, cell 7

1e. Commit rate by text-only prior

source: review/02-yesno-vs-mc-prior-controlled.ipynb, cell 9

source: review/02-yesno-vs-mc-prior-controlled.ipynb, cell 9

1f. Commit vs admit probes, yes/no (best layer per probe)

source: review/02-yesno-vs-mc-prior-controlled.ipynb, cell 12 (max over every 4th layer); MC-5: no activations extracted, skipped

source: review/02-yesno-vs-mc-prior-controlled.ipynb, cell 13

source: review/02-yesno-vs-mc-prior-controlled.ipynb, cell 15

What Part 1 shows

• The recreation holds at 4B. Probe A peaks at 0.694 (L20) against the paper's 0.72; probe B peaks at 0.937 (L16) against 0.96.
• Probe C, the no-image reply text alone, scores 0.900–0.925, at most 0.017 below B at any layer. The Diff probe mostly reads the reply, and the reply holds the admission words that define the label.
• Before any reply token, probe D peaks at 0.744 (L24). The mirage is partly set before the model speaks. D sits well above shuffled labels (0.488–0.523) and well below 0.96.
• The paper's regex labels 684 mirages on this model; 354 of them admit the missing image in other words, which leaves 330.
• On yes/no, commits match the with-image answer 0.673 of the time against 0.449 by chance (κ 0.406). Much of the same-answer criterion is chance agreement.
• MedXpertQA-MM measures the prompt more than the model. The paper's MC prompt forbids admitting, so 0.980 of no-image replies commit, and with-image accuracy is 0.279 against chance 0.200.
• Commit rate is 0.124 where the text-only prior is below chance and 0.158 at high prior. Blind answering is mostly not prior gaming, with a modest rise where the prior helps.
• Commit vs admit stays readable before the reply on prior-free questions (D 0.717 at prior ≤ chance).
• Limits: one model, greedy decoding, regex labels. MedXpertQA-MM activations were not extracted.

Part 2 — SLAKE, and steering toward abstention

SLAKE has open-ended radiology questions with short exact answers, so a blind guess is right far less often than on yes/no. The system prompt is the paper's VQA-RAD prompt with the dataset renamed; it tells the model it has an image and forbids "unknown".

Outcomes of the no-image reply, read from its final [[answer]]: abstain (no answer), hedged (admits the image is missing, answers anyway), mirage (answers, no admission).

Without the image the whole input is the question text, and SLAKE's 3622 records have only 284 distinct texts. The unit is therefore the distinct text, and every split is over texts.

Steering (miragelab/steer.py): texts split 70/30, stratified by outcome. The vector is the difference of means, mirage − abstain, of the no-image pre-answer state on the fit texts, at the layer with the best type-centred CV AUC. α·v̂ is subtracted at every position, with α in units of the fit projection gap. The honesty-instruction arm appends one sentence to the system prompt: "Important: if no image is attached to the question, do not guess. Say that the image is missing and answer [[Unknown]]." Arms run on the 85 held-out texts without image, and on up to 200 image records of those texts to measure the cost.

2a. Behaviour, SLAKE (3622 records, 284 distinct texts)

source: review/03-slake-behaviour-and-steering.ipynb, cell 3

source: review/03-slake-behaviour-and-steering.ipynb, cell 3

source: review/03-slake-behaviour-and-steering.ipynb, cell 3

source: review/03-slake-behaviour-and-steering.ipynb, cell 3

2b. Outcome by text-only prior (distinct texts)

source: review/03-slake-behaviour-and-steering.ipynb, cell 7

source: review/03-slake-behaviour-and-steering.ipynb, cell 7

2c. Steering arms, held-out texts

source: review/03-slake-behaviour-and-steering.ipynb, cell 10 (runs/slake/steer_L34.json)

source: review/03-slake-behaviour-and-steering.ipynb, cell 10 (rate [Wilson 95% CI])

source: review/03-slake-behaviour-and-steering.ipynb, cell 10

source: review/03-slake-behaviour-and-steering.ipynb, cell 11

source: runs/slake/steer_L34.json, recomputed (texts that are mirages under base, outcome under each arm); not a notebook output. It confirms the counts in review/03-slake-behaviour-and-steering.ipynb, cell 12 prose.

2d. Pressure vector ("must answer" sentences removed), distinct texts: fit 199 / held-out 85

source: review/05-pressure-vector.ipynb, cell 3 (rel. shift, cosines) and cell 6 (label-shuffle p, 200 permutations); m = mirage, h = hedged, a = abstain

source: review/05-pressure-vector.ipynb, cell 3

source: review/05-pressure-vector.ipynb, cell 6

source: review/05-pressure-vector.ipynb, cell 9

source: review/05-pressure-vector.ipynb, cell 9

source: review/05-pressure-vector.ipynb, cell 12

What Part 2 shows

• Without the image, 0.203 of records and 0.349 of distinct texts are mirages. Concrete no-image answers are right 0.208 of the time.
• 92 of the 99 mirage texts have a wrong text-only prior, so these mirages are not prior exploitation.
• The honesty instruction cut mirages from 0.365 to 0.094 and raised abstention to 0.871. With-image accuracy went from 0.455 to 0.445.
• The vector at ×2 cut mirages to 0.059, but abstention moved only from 0.353 to 0.388. 24 of the 31 base mirages became hedges. The vector removes the claim to have seen the image; it does not stop the answer.
• At ×4 the model breaks (with-image accuracy 0.080), so ×4 outcome labels mean little.
• The type-centred vector ×4 gives the same pattern (hedged 0.647, accuracy 0.485), so the effect is not only question type. Random ×4 moves the other way (mirage 0.459).
• The "must answer" sentences are a direction in the state (a shift of 0.028–0.145 of the state norm), but their alignment with the behaviour directions is not above a label-shuffle null at any late layer.
• Removing them moves hedged states toward abstain (2.5, sem 0.564). Abstain and mirage states do not move.
• Limits: 85 held-out texts (about ±10 points), one layer, greedy decoding, regex labels.

Part 3 — What the probes read (SLAKE, 284 distinct texts)

This part asks whether abstain, hedged and mirage are readable from the no-image pre-answer state beyond question type. One row per distinct text (majority outcome over its records), 5-fold CV over texts.

The controls are shuffled labels, a baseline that sees only question type, type-centred states (each type's mean removed, using training folds only), and clusters scored against both outcome and type.

source: review/04-slake-behaviour-geometry.ipynb, cell 1

source: review/04-slake-behaviour-geometry.ipynb, cells 1 and 3

3a. Probes by layer (balanced acc, 5-fold CV over texts)

source: review/04-slake-behaviour-geometry.ipynb, cell 3

source: review/04-slake-behaviour-geometry.ipynb, cell 3

source: review/04-slake-behaviour-geometry.ipynb, cell 4

The type-centred 3-class probe peaks at 0.569 (L24), from the cell 3 output. The cell 14 prose in the notebook says 0.55; the page uses the cell output.

3b. One direction per pair (out-of-fold ROC-AUC, best layer)

source: review/04-slake-behaviour-geometry.ipynb, cell 6

source: review/04-slake-behaviour-geometry.ipynb, cell 6

3c. Hedged relative to the abstain→mirage axis (L24)

source: review/04-slake-behaviour-geometry.ipynb, cell 9

source: review/04-slake-behaviour-geometry.ipynb, cell 9

3d. Clusters and views

source: review/04-slake-behaviour-geometry.ipynb, cell 12 (k-means on PCA-20, max over layers)

source: review/04-slake-behaviour-geometry.ipynb, cell 12 (k = 10; 1 = no grouping)

The 3×4 grid of views (notebook 04, cell 13) has no saved output in the executed notebook, so it is not shown.

What Part 3 shows

• The three behaviours are readable before the model speaks, but weakly once question type is removed. The type-centred 3-class probe gives 0.507–0.569 (shuffled 0.271–0.330, chance 0.33).
• Raw 3-class probes (0.598–0.686) sit close to the type-only baseline (0.595). Most of the raw signal is question type.
• One difference-of-means direction per pair does about as well as the full probe (type-centred AUC 0.655–0.71). The raw mirage vs abstain direction reaches 0.95, and 0.655 after type is removed.
• Clusters follow question type (ARI 0.57 and 0.625), not outcome (0.178 and 0.133). After type-centring they match neither (ARI ≤ 0.073).
• Hedged sits between abstain and mirage (raw t 0.468, cos 0.798). With type removed it is partly its own direction (off 0.695, cos 0.638).
• No 2-D view separates behaviour beyond the full space. LDA separates only because it is fit in-sample.
• For steering, a behaviour vector exists but is small next to question-type structure. Expect modest, type-dependent effects.
• Limits: 284 texts, one model, regex labels; 49 texts had different outcomes across repeated calls (majority used).

Unreviewed runs (Sep 30 – Oct 4) — raw, not reviewed

Raw, not reviewed. Numbers in this section are copied from local run logs (runs/, not published) or recomputed from local run files. No notebook has checked them. Review them before quoting.

These runs add the honesty prompt and a vector together, with a three-way split per seed and controls at matched push size. The test is whether the steering effect comes from the direction or from the size of the push. Definitions for this section (miragelab/dose.py):

• Split per seed over distinct texts, stratified by base outcome: vector-fit 40%, dose 20%, test 40% (113 / 57 / 114 texts).
• prompt (v1) = the honesty instruction from Part 2. prompt v2 = v1 plus one clause aimed at questions that point to a part of the image by position, colour or shape.
• Prompt v2 was written after reading seed-0 test texts. Its numbers on seed-0 test texts are not held out.
• general = mirage − abstain direction on base-prompt states (the Part 2 vector, refit on each split).
• leftover = mirage − abstain on honesty-prompt states, type-centred. It targets the mirages the prompt leaves.
• combo = general and leftover pushed together.
• template = "what organ is the [grey / dark / ...] part" questions minus other Organ questions, type-centred. It knows the question form and nothing about mirages, so it is a control for leftover.
• g / l indices = multiples of each vector's fit projection gap; "combo g2 l1" = 2× general + 1× leftover.
• % push = push length as a share of the median state norm at the steering layer.
• Gate = leftover CV AUC against a within-type label-permutation null. Choice rule, fixed before any dose data: lowest no-image mirage rate among arms whose with-image accuracy is within 5 points of prompt alone and whose loop rate is ≤ 2%.
• Controls: shuffled-label leftover (labels permuted within type), random unit vector, leftover reversed (should add mirages), activation-spread random, estimation-matched random (diff of means of random text groups), rotate (direction change, size kept), scale (size change, direction kept).
• loop = share of replies in which some 8-word run repeats 4 or more times.

U1. Pool over split seeds 0–2

source: runs/next-pool.txt, lines 1–5 (10-04 00:12)

source: runs/next-pool.txt, pooled block (mean±sd over seeds)

The pool covers seeds 0–2 only. Seeds 3 and 4 never ran their own fit, dose and test stages (the folds chain stopped after seed 2), so the pool skipped them. Their test texts were run only with seed-0 vectors (transfer table below).

The prompt v2 row pools seeds 1 and 2, whose test sets share texts with the seed-0 test set. Prompt v2 was written after reading seed-0 test texts, so that row is not fully held out. The table below restricts it to texts outside the seed-0 test set.

source: runs/slake/dose_s1/test_rows.json, runs/slake/dose_s2/test_rows.json, recomputed (no-image rows, texts not in runs/slake/dose/test_rows.json); v1 values match the 'seen by s0' blocks of runs/next-pool.txt

Seed-0 vectors on seed 1 and 2 test texts that seed 0 never used (texts in the seed-0 test set). This is the held-out check for the seed-0 vectors and alphas.

source: runs/next-pool.txt, 'seed 1 test texts, unseen by s0 (n=48)' and 'seed 2 test texts, unseen by s0 (n=45)' blocks

Pooled, general (0.026) and combo (0.009, one seed) sit below the random controls (0.039–0.094). Two controls undercut the direction reading. Shuffled-label leftover (0.070) does better than real leftover (0.088), and general plus a random vector in place of leftover (0.013) matches combo. On held-out seed-1 texts the shuffled-label control reaches 0.000. Leftover reversed adds mirages (0.120 against 0.079 for the prompt alone), the expected sign. The template direction (0.029) beats leftover; on seed 0 the two have cosine 0.53, so leftover partly encodes the question form. With 45–114 texts, one text moves a rate by 0.009–0.022.

source: runs/next-pool.txt (pooled block, seed-1 unseen block); cos(leftover, template) 0.53 from runs/next-think4b.log, line 2 (seed-0 vectors)

source: runs/next-xfer-s3.log (layer 36, fit CV AUC 0.766), runs/next-xfer-s4.log (layer 26, fit CV AUC 0.653); not in the pool

On seeds 3 and 4 the real arms give 0.00–0.07 mirage and the controls 0.00–0.10. No real arm separates from its controls.

U2. Push-size sweep (equal share of state norm), seeds 0 and 1

source: runs/next-sweep-s0.log, runs/next-sweep-s1.log, lines 1–2

source: runs/next-sweep-s0.log lines 97–190 and runs/next-sweep-s1.log lines 97–190 (summary block; 5%/15%/25%, v2, swap and rotate rows omitted here)

The sweep runs on each seed's 57 dose texts without image, and up to 60 image records for accuracy. along / off = push length along and off the general axis. geo = shift of the mean state on the mirage−abstain axis against prompt alone, in fit-gap units (negative = toward abstain). beats randoms = share of same-size random draws with a higher mirage rate. Random, rotate and scale arms ran without image (acc nan).

Prompt v2 was written after reading seed-0 test texts. The dose texts here are outside the seed-0 test set for seed 0, but seed 1's dose texts can overlap it.

• At 20% on seed 0, general and combo reach 0.00 mirage, and every same-size random draw stays at 0.05–0.12.
• Seed 1 is weaker. General 20% gives 0.04, and the two estimation-matched random draws at 20% give 0.33 and 0.00.
• Leftover at 10% gives 0.09 on both seeds, against 0.09 and 0.11 for the prompt alone.
• Size alone does little. Scale shrink and grow at 20% give 0.09–0.14, against 0.09–0.11 for the prompt alone.
• Seed 1 accuracy is lower for every arm (prompt 0.333 vs 0.467) because its image records differ.
• On 57 texts one text is about 0.02, so most gaps here are one to five texts.

U3. Qwen/Qwen3-VL-4B-Thinking (think4b; seed-0 split, test texts)

source: runs/next-think4b.log, lines 1–4

source: runs/next-think4b.log (10-03 03:36; --test_image 60)

In the base row the remaining 0.64 (73 of 114 texts) is truncated. Those replies had no </think> within 1536 new tokens, so there is no answer to label. 51 of those 73 also loop. The other rows' small remainders are truncated replies too.

source: runs/slake/dose_think4b/test_rows.json, recomputed (base, no image, 114 texts: mirage 41, truncated 73); max_new_tokens from miragelab/transfer.py, MODELS

Once the prompt is added the Thinking model mirages on at most 0.01 of texts, and every vector arm and control stays at 0.93–1.00 abstain. The prompt alone does the work here, so the vectors can't be judged on this model. Prompt v2 was written after reading seed-0 test texts, and these are seed-0 test texts, though on a different model.

U4. nvidia/Cosmos-Reason2-2B (cosmos2b) and Qwen/Qwen3-VL-2B-Instruct (instruct2b)

source: runs/next-gen-2b.log, runs/next-gen-cosmos.log (10-02 12:15, 12:53)

source: model ids: miragelab/transfer.py, MODELS

source: runs/now-prompts-cosmos.log (10-02 10:40) and runs/next-prompts-cosmos.log (10-02 14:23)

The two cosmos2b prompt runs disagree (prompt arm mirage 0.70 vs 0.92). Only the 14:23 run's rows are on disk; the 10:40 run's rows were overwritten and not kept, so its numbers can't be checked. The page uses the 14:23 run. Recomputed from its rows, the remainder of each row is truncated replies (base 0.03, prompt 0.07, prompt v2 0.04). The cause of the gap between the runs is not known.

source: runs/slake/dose_cosmos2b/test_rows.json (written 10-02 14:23), recomputed; matches runs/next-prompts-cosmos.log

Cosmos-Reason2-2B mirages on 0.96 of test texts without the image. The honesty prompt barely moves it (0.92), and its reasoning noticed the missing image on only 3 of 284 texts. Prompt v2 was written after reading seed-0 test texts; these are seed-0 test texts on a different model.

source: runs/now-prompts-2b.log (10-02 09:57); the chain run runs/next-prompts-2b.log (10-02 12:53) printed no arm lines

These instruct2b numbers come from the 09:57 run, and runs/slake/dose_instruct2b/test_rows.json (written 09:57) reproduces them. The chain rerun at 12:53 ran 18 s and printed no arm lines, so it likely reused those rows. It is unreliable and is not used. On the 2B model the prompt leaves 0.68 mirages and v2 0.47; v2 was written after reading seed-0 test texts, and these are seed-0 test texts on a different model.

U5. Introspection (self-report "seen" 0–10 and confidence 0–10)

source: runs/next-introspect.log, lines 1–3 (10-03 17:04)

source: runs/next-introspect.log, 'scores' and 'strong_vs_soft' blocks

source: runs/next-introspect.log, 'auc_mirage_vs_abstain' block (rounded to 3 d.p.)

source: runs/next-introspect.log, 'check' block

The questions, asked as a follow-up turn after the model's own reply (same system prompt, no steering, greedy, last [[N]] parsed):

• seen: "Did the answer you just gave rest on a medical image that you actually examined? Rate from 0 (no image was examined) to 10 (it rests entirely on an image I examined). Reply only [[N]]."
• conf: "How confident are you that the answer you just gave is correct? Rate from 0 (not at all) to 10 (certain). Reply only [[N]]."
• check, appended to the question before any reply: "Before answering: is a medical image attached to this message? Reply only [[yes]] or [[no]]."

source: miragelab/introspect.py, ASK and CHECK

• After the fact, the model's self-report tracks its own outcome. Mirage replies score seen 6.85 and conf 9.01; abstains score 0.17 and 0.37 (AUC 0.966 and 0.990).
• Before answering, the check fails. Without an image it says an image is attached for 44 of 71 base-mirage texts and 28 of 78 base-abstain texts.
• "seen" parsed on only 454 of 646 rows, so the seen means rest on fewer rows than conf.

What didn't hold up

Status: done vs not done

Done

• VQA-RAD: 5950 records generated, 1800 extracted (runs/chain.log)
• MedXpertQA-MM: 2000 records generated + text-only prior (runs/chain.log)
• SLAKE: 3622 records generated, states extracted, pressure/neutral states (runs/chain.log, 09-29)
• Notebooks 01–05 executed (review/; runs/chain.log 09-29 20:00)
• Steering on held-out texts (runs/slake/steer_L34.json)
• Dose seed 0 fit / dose / test (09-30); seed 1 fit / dose / test (10-01); seed 2 fit / dose / test (resumed, done 10-02 01:27)
• Transfer of seed-0 vectors to seeds 1–4 (runs/next-xfer-s1..s4.log)
• Sweep seeds 0 and 1 (runs/next-sweep-s0.log, -s1.log)
• Thinking-4B test (runs/next-think4b.log); 2B / Cosmos generation (runs/next-gen-2b.log, -cosmos.log); Cosmos prompts (runs/next-prompts-cosmos.log)
• Introspection (runs/next-introspect.log); pool (runs/next-pool.txt, 10-04 00:12)
• Also run, not on this page: try_dirs (runs/try-dirs.log, 10-01); LLM-judge sample (runs/judge/, 09-29)

Not done / failed

• Runs from Sep 30 – Oct 4 not reviewed in any notebook
• transfer-steer FAILED: "too few mirage/abstain texts on instruct2b fit texts to pick a layer (mirage 112, abstain 1)" (runs/next-transfer-steer.log, 10-02 12:53)
• chain-transfer smoke-steer FAILED: ValueError n_splits=5 > n_samples=4 (runs/transfer-smoke-steer.log, 09-30 22:43)
• Folds s3 / s4 fit, dose, test never ran (runs/chain-folds.out ends at 'folds s2 test start'); pool skipped seeds 3–4
• T=0.7 samples (chain-samples.sh) never started (runs/chain-samples.out: only the 'queued' line)
• prompts-2b in chain (10-02 12:53) ran 18 s with no arm lines; runs/slake/dose_instruct2b/test_rows.json dates from 10-02 09:57, so the rerun likely reused the earlier rows. Unreliable, not used; the page uses the 09:57 run.
• cosmos2b prompts: the 10:40 run (prompt mirage 0.70) disagrees with the 14:23 run (0.92); only the 14:23 rows were kept, cause unknown
• MedXpertQA-MM activations not extracted (review/02-yesno-vs-mc-prior-controlled.ipynb, cell 12)
• prompt v2 written after reading seed-0 test texts → v2 numbers on seed-0 test texts not held out (miragelab/steer.py, HONEST_V2)

Reproduce

git clone --recurse-submodules <repo> && cd mirage-probes-steering
python -m pytest -q tests
export MIRAGE_ROOT=/path/to/data-root   # runs/, raw_data/, data/
cd "$MIRAGE_ROOT" && export PYTHONPATH=<repo>
python -m miragelab.slake --out runs/slake
python -m miragelab.extract --dataset slake --responses runs/slake/responses.json --out runs/slake/acts
python -m miragelab.steer --run runs/slake
python -m miragelab.pressure --run runs/slake
python -m miragelab.dose --run runs/slake --stage {fit,dose,test,sweep,pool} [--seed N]
python -m miragelab.introspect --run runs/slake
cd <repo>/review && jupyter lab

source: README.md (Setup, Run); full sequences with timeouts in scripts/queue/chain-*.sh

Run outputs (runs/, about 2.5 GB) are not in the repo. The commands above regenerate them; scripts/queue/chain-*.sh hold the exact sequences and timeouts that produced them. Those scripts call the original bench paths, so edit the paths before reuse. Generation needs a vLLM server for Qwen3-VL-4B-Instruct and the datasets under MIRAGE_ROOT (see README.md). The notebooks read runs/ from MIRAGE_ROOT.