What Is Multimodal AI? The Day Text-Only AI Opened Its Eyes — How It Works and How to Use It in 2026

AI became able to "see" images not because it got better, but because the very structure that handles input changed. And this change altered how we ask questions before it altered model scores.

---

Background / Why This Topic Now

Part 1 of this series covered the problem of AI being wrong (hallucination), and Part 2 covered the problem of AI being inconsistent (non-determinism). This installment is a story on a slightly different level. It is not about what the model outputs, but about what it can take as input— that is what changed. 

We no longer retype an error message; we just capture the screen and drop it in. Instead of summarizing a contract in words, we snap a photo and upload it. This seemingly trivial shift is actually one of the biggest turning points in how AI is used. From the era of describing and asking to the era of showing and asking — because that is the line we have crossed.

But misconceptions grew alongside it. The phrase "AI has finally opened its eyes" is only half right.

As we will see below, AI in 2026 can interpret college-major-level diagrams and yet cannot reliably read the time on a wall clock. Understanding what lies behind this gap is the goal of this article.

---

Key Data & Current State

• Distance to expert level: On MMMU, a college-major-level multimodal benchmark (11.5K questions across 6 disciplines, 30 subjects, 183 subfields, and 30 image types), the human expert ceiling is 88.6% (Yue et al., CVPR 2024)
• A leap in three years: On the same benchmark, GPT-4V scored **56.8%** in November 2023, and as of 2026 frontier models are reported at the high 70s to around 80%(compiled from the MMMU leaderboard and vendor materials, 2026)
• Raise the difficulty and the gap reopens: On MMMU-Pro, which expands the options from 4 to 10 and embeds the question itself inside the image, performance compared with the original drops by 16.8–26.9 percentage points (Artificial Analysis tally)
• An unexpected blind spot: In an analog-clock-reading experiment, all four multimodal models tested performed poorly; training on 5,000 synthetic clock images improved them, but scores plunged again on clocks with unfamiliar designs (Conde et al., IEEE Internet Computing, 2025)

? Interpretation: Put the first two lines next to the last one and the shape of today's AI vision becomes clear. It gets nearly 80% on college-major exams, yet fails at reading a clock, something an elementary schooler can do. If human visual ability is stacked from easy to hard, AI visual ability is stacked in the order in which training data was abundant. It is data distribution, not difficulty, that shapes the terrain of skill.

---

In-Depth Analysis

1) Exactly When Was "the Day It Opened Its Eyes"? Attached Eyes vs. Native Eyes

Multimodality did not arrive as a single event but in two stages. And that difference is the heart of this topic.

Generation 1 — the pipeline-stitching approach. The early approach stitched together separate models. A speech-recognition model converted audio to text, an OCR or image-recognition model converted images into descriptions, and only that text went into the language model. In other words, the AI had never seen the image. It had only read a summary someone else wrote for it.

The problem with this structure is that information is discarded at the translation step. The moment speech becomes text, the tremor in a voice, the intonation, the spot where words trailed off, and the sounds in the background all vanish. The moment a photo becomes a description, only what the describer judged important survives.

Generation 2 — native multimodal. The approach that followed does not translate images and sound into text; instead, they are learned together in the same representation space from the start. The model that made this shift widely known was GPT-4o, released in May 2024. The "o" stands for "omni": unlike earlier models that depended on a separate model to handle audio, it adopted a structure that processes speech directly without converting it to text.

Author's view: This is where the real meaning of "AI opened its eyes" lies. Its eyesight did not improve; the interpreter in the middle who used to read on its behalf disappeared. With the interpreter gone, information that used to be thrown away in interpretation survived, and so suddenly much more seemed possible.

2) What Changed Before Performance — the Form of the Question

A bigger change for users than the technical one is that the cost of composing a questiondisappeared.

To ask about an error, you used to have to copy the message off the screen and reconstruct the situation in sentences. Information is already lost in that process, because the user has to decide first what matters and pick it out. Tossing in a single screenshot lets you defer that judgment . Being able to ask even when you do not know what matters: that is the biggest change felt in practice.

3) So Why Can't It Read a Clock?

A multimodal model's abilities are arranged in an order that differs from human intuition.

In the study cited above, models repeatedly failed to read analog clocks. The researchers pointed to two causes: errors occur both at the stage of distinguishing the hands and at the stage of converting hand angles into directional information, and errors from the first stage are amplified in the second. Models were especially vulnerable to unusual hand designs such as arrow shapes, and even after training on 5,000 images raised their scores, they collapsed again in front of designs they had never seen(Conde et al., 2025).

The reason is simple. The internet has almost no images labeled "this clock shows 3:47." Academic diagrams and charts, by contrast, exist in bulk alongside explanatory text. Counting objects, judging precise spatial relationships, and determining temporal order in video are weak for the same reason.

The key is the difference between recognition and reading. The model reminds us that knowing "this is a clock" and "calculating what time it shows" are entirely different tasks. The ability to recognize what is on a screen and the ability to extract exact values from it develop separately.

4) How Far Has the Korean-Language Environment Come?

In Korea, too, multimodal models took off in earnest across 2025–2026.

• LG AI Research's EXAONE 4.5 (released April 2026): A vision-language model that understands images and text together, aimed at interpreting complex documents such as contracts, technical drawings, and financial statements. It has 33 billion parameters, was announced as scoring 62.2 on ChartQA Pro, and was released as open weights on Hugging Face (LG AI Research, 2026).
• Naver's HyperCLOVA X OMNI (released December 2025): Heading toward an omni model that handles text, images, and speech on both the input and output sides, Naver released two tracks together: one that adds speech to its existing vision-language model, and one that trains all three modalities jointly from the start (Naver CLOVA, 2025).

Author's view: It is worth noting that Korean models are focused on "documents." A large share of Korean corporate work involves unstructured Korean-language documentssuch as scanned contracts, hand-annotated drawings, and government forms, and here specialization in Korean documents may translate into results more directly than a global model's general performance. If a model is released as open weights, it also opens the option of processing documents that cannot leave the premises internally.

---

In Practice — 7 Principles for Showing and Asking

① Don't describe it, capture it and show it

Error messages, settings screens, receipts, hand-drawn diagrams: anything that loses information when put into sentences is better simply captured and uploaded. You don't need to know what matters. Handing that judgment to the AI is multimodality's biggest advantage.

② Don't just toss in an image; give it a role and a purpose

"What's this?" is the most wasted question. The same image calls for looking at completely different places depending on the purpose.

"The attached screen is an error that appeared at the payment step. Tell me separately what the user should press next, and what information should be included when passing this to a developer."

③ For tables and charts, have it "transcribe the numbers first"

When you have a chart or table image interpreted, demanding a conclusion right away leaves no way to catch misreadings. Have it output the values it read as a table first , then request the interpretation, so you can catch errors at the intermediate step.

"First, organize the values you read from the graph into a table by year and output it. Then continue with the interpretation on the next line."

④ Treat counts, times, and precise positions with suspicion by default

How many, what time, and what is to the left of what: these three are the representative weaknesses of current models. If you get an answer, the safe rule is to count it yourself. The verification principles from Part 1 apply here unchanged.

⑤ Crop before uploading

An image cropped to the region of interest is more accurate than a single full screenshot. With limited resolution, feeding in the whole screen smears exactly the text that matters. Splitting into several images and uploading them in orderis also a good approach.

⑥ If failure is predictable, reshoot

Tilted photos, shadowed documents, and low-resolution handwriting will not improve no matter how much you polish the prompt. Cases where reshooting is faster than prompt engineeringare far more common in practice. Documents should be shot head-on, without shadows, one page per image.

⑦ For audio and video, demand "quotes," not a "summary"

When dealing with long meeting recordings or video, a summary alone cannot be verified. Require it to cite the supporting statements together with their position (timestamp) and the output becomes checkable. Judging temporal order in video is still a weak area, so this habit matters especially.

---

Implications

The area that benefits most from multimodality is likely not flashy image generation but document processing. Korean corporate work has piles of material that cannot be extracted as text: scanned contracts, drawings with handwritten notes, financial data frozen into PDFs. Until now, the only way to handle them was for a person to read them by eye and transcribe.

Korean models putting contract, drawing, and financial-statement interpretation front and center appears aimed at exactly this point. When evaluating adoption, however, results from testing directly on your own documentsmatter far more than benchmark scores. As the clock example showed, a model's skill follows the training data distribution rather than difficulty. For a document format of which only a few hundred exist in the world, such as an internal company form, public benchmark scores can hardly be more than a reference.

---

Outlook & Variables to Watch

• Improvement variable — agents that look at screens: As AI expands toward looking at and operating screens directly, visual recognition is shifting in status from an "add-on feature" to a "precondition for execution." In that case the bar for required accuracy rises with it.
• Risk variable — rising verification difficulty: A text answer can be read by a person who senses something is off, but the claim "I read this dimension from this drawing" cannot be confirmed without checking against the original. The richer the input, the higher the verification cost.
• Checkpoints: ① The size of improvement on benchmarks like MMMU-Pro that block shortcuts ② Whether a public evaluation set specialized for unstructured Korean documents emerges ③ Accumulation of on-premises adoption cases for open-weight Korean models

---

Conclusion

1. It did not open its eyes; the interpreter disappeared. With the intermediate step that translated images and sound into text gone, information that used to be discarded began reaching the model.
2. AI's visual ability follows data distribution, not difficulty. It reads college-major diagrams but not wall clocks. Predict with human intuition about hard and easy, and you will be wrong every time.
3. The key to using it is "show instead of describe" and "demand the intermediate step." Don't transcribe, show it as is, but have it output the values it read first so the result is verifiable. These two are the minimum conditions for using multimodality safely in practice.

References

• MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI (Yue et al., CVPR 2024)
• MMMU-Pro Benchmark Leaderboard — Artificial Analysis
• MMMU Benchmark: 11,500 Multimodal Questions, Frontier Above 70%
• Large Language Models Struggle With Reading Clocks — IEEE Spectrum (2025)
• GPT-4o — Wikipedia
• "Integrated Image-Text Reasoning"… LG AI Research Unveils EXAONE 4.5 — ZDNet Korea (2026)
• HyperCLOVA X OMNI: Korea's National AI, the Journey Toward an Omni Model — Naver CLOVA Tech Blog (2025)