The Illusion of Visual Understanding: Mirages in Vision-Language Models

Based on: Asadi et al., "Mirage: The Illusion of Visual Understanding in Vision-Language Models," arXiv:2603.21687v3, April 2026

Duration: ~15 minutes Prerequisites: CLIP, LLaVA, image tokenization, visual reasoning over video (Chain-of-Steps)

Learning Objectives

By the end of this lecture, students will be able to:

1. Define the mirage effect and distinguish it from hallucination in VLMs

2. Explain how standard VLM benchmarks can be solved without visual input

3. Critically evaluate VLM benchmark results using the B-Clean framework

4. Connect mirage findings to the broader question of whether models genuinely reason over visual input

Lecture Outline

Section 1: Motivation — A Provocative Question (~2 min)

Slide 1: Title slide Slide 2: The core question

• We've studied how models align vision and language (CLIP), answer questions about images (LLaVA), and even reason visually in video (Chain-of-Steps).

• But here's an unsettling question: What if models can answer visual questions without actually looking at the image?

• Introduce the paper's central finding: frontier VLMs generate confident, detailed descriptions of images that were never shown to them — and they do so without expressing uncertainty.

Section 2: Defining the Mirage Effect (~2 min)

Slide 3: Mirage vs. hallucination

• Hallucination: Model sees an image but generates incorrect details about it (wrong color, missing object, invented text). The epistemic frame is valid — the model is grounding in visual input — but the details are wrong.

• Mirage: Model receives no image at all but generates a confident, coherent response as if it did. The entire epistemic frame is false — the model fabricates the act of seeing.

• Key distinction: hallucination is a detail-level failure; mirage is a frame-level failure.

• Why this matters: if a model can answer benchmark questions without images, high benchmark scores don't prove visual understanding.

Section 3: The Phantom-0 Benchmark (~3 min)

Slide 4: Phantom-0 design and Figure 1 heatmap Slide 5: Key results

• Phantom-0 design: Take 20 standard VLM benchmark categories (VQA, medical imaging, OCR, chart reading, etc.). Remove the images entirely. Ask models to answer anyway.

• Mirage rate: Fraction of questions a model answers confidently (without expressing uncertainty) when no image is present.

• Walk through Figure 1 heatmap: all frontier models show >60% mirage rates across most categories.

• Standout results: Gemini 2.5 Pro = 90.5%, Claude Sonnet 4.5 = 86%, GPT-5.1 = 81%.

• Pathology bias in medical mirages: Models don't just guess — they systematically assume pathology is present, creating dangerous false positives in clinical settings.

Section 4: The Super-Guesser Experiment (~3 min)

Slide 6: Mirage-scores on standard benchmarks Slide 7: The super-guesser

• Mirage-score: Run standard benchmarks (MicroVQA, ReXVQA, etc.) but with images removed. Models retain 70–80% of their original accuracy.

• This means most of what benchmarks measure can be recovered from text alone.

• The super-guesser: Fine-tune a tiny 3B text-only model (Qwen-2.5) on image-less ReXVQA data. Result: it outperforms all frontier VLMs AND human radiologists by >10%.

• Implication: benchmark performance ≠ visual understanding. A text-only model shouldn't beat radiologists on a "visual" medical benchmark.

Section 5: Why This Happens — Mirage Enablers (~2 min)

Slide 8: Mirage-mode vs. guess-mode

• Mirage-enablers: Features of benchmark questions that allow text-only answering — answer leakage in question text, domain priors, statistical regularities.

• These can't be detected manually (inter-annotator agreement is near chance).

• Mirage-mode vs. guess-mode: When models are told "the image is missing, please guess," performance drops. But when images are silently removed (mirage-mode), models perform better — they adopt a confident epistemic frame that happens to exploit text shortcuts.

• The model's confidence isn't a sign of understanding; it's a sign of exploiting priors.

Section 6: B-Clean Mitigation Framework (~1.5 min)

Slide 9: B-Clean pipeline and impact

• B-Clean: Post-hoc benchmark cleaning framework. Run each question through a text-only model; if it can be answered without the image, remove it.

• Results: B-Clean removes 74–77% of questions from standard benchmarks.

• After cleaning: MicroVQA accuracy drops from ~65% to ~15–23% for frontier models.

• The "real" visual understanding captured by current benchmarks is a fraction of what scores suggest.

Section 7: Connecting the Threads (~1.5 min)

Slide 10: Synthesis and discussion

• Contrast with video reasoning: In the previous lecture, we saw genuine emergent reasoning in video models (Chain-of-Steps). The mirage paper shows that image-language models may not be reasoning visually at all — they may be exploiting text shortcuts.

• The evaluation crisis: If 75% of benchmark questions are mirage-answerable, how do we know when models truly "see"?

• Going forward: Need benchmarks that are provably vision-dependent, adversarial to text shortcuts, and test for genuine visual grounding.

• Discussion questions: How would you design a vision benchmark that can't be gamed by a text-only model? Is the mirage effect also present in video reasoning benchmarks?

Slide Summary