Verified, then logged.
Sun 22 Nov 2026 · 00:17 KST / 15:17 UTC한국어 | ENSearch /

01 / AI · Analysis

20 Atari games expose vision AI's limits in following motion

The latest vision-language models often missed object motion and leaned on written history over new screens, a study found.

On a lab bench, a square ball streaks past a misplaced paddle on a small CRT screen while an unattended joystick sits in front, wired to a plain desktop computer. AI-generated image

AtariBench, a benchmark described in a paper posted on arXiv on 9 Nov 2026, tested vision-language models on 20 Atari 2600 games and reported substantial limitations in the latest systems. Vision-language models are AI models that handle images and language together.

The paper said these models achieved strong performance on benchmarks of static image understanding. Their ability to act in moving environments is less well understood.

The benchmark shows models gameplay video at high temporal resolution and requires a series of decisions. It separates game speed from the time a model takes to infer. The test therefore sets response speed aside and asks whether a model can read changes on screen and act on them.

Models that miss an object's direction

The paper said the latest vision-language models often failed to infer the motion and direction of objects. Their sequential decisions also broke down easily. The abstract gives no scores by model.

The paper wrote that the models "over-rely on textual interaction history" even when new visual feedback is added. They lean too heavily on the accumulated written record after the screen has changed.

Sorting the game scores that can be trusted

The researchers also presented a separability analysis, a procedure for identifying which in-game scores reliably show a model's competence. The analysis rests on the premise that not all game scores carry the same signal.

The abstract does not list the games whose scores were classed as reliable.

Visual skill that does not carry into action

The researchers said they also used the benchmark to study how memory design and prompting strategies affect performance. A prompt is the instruction given to a model. The abstract does not say which design raised scores or by how much.

The paper concluded that "a persistent gap" separates passive visual perception from active decision-making. The abstract does not say how this result carries over to devices that must see and then move.

The researchers presented AtariBench as a testbed for models that reason and act in dynamic environments. The gap the paper describes is the one such models have to close.