MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models

Cohen, Vanya; Mooney, Raymond

Can vision-language models track a changing world?

Agents need to understand how actions change the world and track the state of entities over time. Across Minecraft, Chess, and Shell Game, VLMs generally track entity state less accurately from visual inputs than from text. MET-Bench measures this gap using parallel text and image versions of entity-tracking tasks.

MET-Bench overview: predict the final state after chess moves, shell swaps, or a Minecraft action, using text or image representations. — Chess and Shell Game vary the representation of actions; Minecraft varies the representation of the initial state and candidate next states.

VLMs track entity state better from text than images.

TextImage

Minecraft accuracy with text versus image states: Gemma 3 27B IT 51.6 versus 27.8; Qwen3-VL 8B 49.4 versus 28.4; Qwen3-VL 235B 73.0 versus 34.2; GPT-4o 71.0 versus 37.0; Claude Sonnet 4.6 88.4 versus 41.8 percent.

Chess accuracy with text versus image actions: GPT-4o 94.2 versus 83.4; GPT-4.1 98.1 versus 75.3; Gemini 2.5 Flash 77.0 versus 44.6; Llama 4 Maverick 82.4 versus 61.9; Claude 3.7 Sonnet 99.5 versus 96.2 percent.

Shell Game accuracy with text versus image actions: GPT-4o 98.2 versus 36.6; GPT-4.1 99.8 versus 37.6; Gemini 2.5 Flash 94.0 versus 34.0; Llama 4 Maverick 77.0 versus 34.6; Claude 3.7 Sonnet 100.0 versus 77.4 percent.

Selected models with chain-of-thought prompting. Error bars show 95% confidence intervals. Full results →

Datasets

Minecraft

Predict the next state from an initial state and a text action. Select from four candidates, represented as game-state text or first-person images.

Chess

Track a board from its initial FEN through a sequence of moves represented as text or images. Return the final board state as FEN.

Shell Game

Track a ball through a sequence of swaps represented as text or images. Return its final position.

🤗 Raw Minecraft trajectories: 462 trajectories containing 462,235 observations with screenshots, game-state telemetry, and recorded controls and actions.

Evaluation

Run all three domains with text and image inputs using lmms-eval. See the setup and evaluation instructions to get started.

Alternatively, the standalone evaluator supports OpenAI-compatible APIs and local Hugging Face models. Both options use ten-action sequences for Chess and Shell Game and next-state prediction tasks for Minecraft.

Citation

@inproceedings{cohen2026metbench,
  title={MET-Bench: Multimodal Entity Tracking for Evaluating the Limitations of Vision-Language and Reasoning Models},
  author={Cohen, Vanya and Mooney, Raymond},
  booktitle={International Conference on Machine Learning},
  year={2026},
  url={https://arxiv.org/abs/2502.10886}
}