Verified, then logged.
Sun 22 Nov 2026 · 00:33 KST / 15:33 UTC한국어 | ENSearch /

Breaking01 / AI

Google launches benchmark for visual document tasks

MKA grades outputs in a shared execution environment while keeping its evaluation tasks private.

Image from Google
Image: Google

Google DeepMind announced the Multimodal Knowledge Agent (MKA) benchmark on 10 Nov 2026, reporting evaluations on 200 expert-designed tasks (n=200, private evaluation set) involving visually complex files; its announcement presents no independent replication results.

Google describes MKA as an evaluation of agents that use images and document layouts to complete work. All models run through Stirrup, an open-source agent execution tool, with the same tools, isolated environment and limits on interaction turns. An automated grader based on a non-Google model inspects output files and spreadsheet formulas.

Google reported agreement between domain experts and the automated grader of 96% (n=undisclosed, comparison of judgements on model outputs). This measures agreement on grading, rather than how often agents complete tasks successfully.

According to Google, Artificial Analysis keeps the evaluation tasks private to prevent them from entering training data or being found through search. A separate set of public tasks is available for inspecting the format and testing agents.

Google argues that many existing knowledge-work benchmarks can be solved without examining images or page layouts. MKA instead focuses on visual work such as cross-referencing blueprints and engineering schematics, and creating presentations and spreadsheets.

Google has not disclosed the comparison sample size behind the grading agreement figure. Users can explore the benchmark and evaluate models through Artificial Analysis, while the separate public tasks provide a way to try the format.