Can Frontier Models See Past a Decoy? Elorian Tests Visual Reasoning

Judgment

Enterprise buyers comparing visual-reasoning models should not treat Glimpse’s 50.48% as a like-for-like win: it is a single internal run, while the paper’s frontier scores are three-run averages under reported settings, so procurement should demand matched, repeated DecoyCount results before paying.

Basis

Elorian CEO Andrew Dai told Latent Space that Glimpse scored 50.48% on DecoyCount in an internal test of 1,660 items, a single run without an agent or tool harness, and said evaluation images and text were withheld from training with no overlap found in the checked data. The accompanying paper, co-released with Mercor, reports frontier models below 50% accuracy; the highest was GPT-6 Astra at 43.17%, then Gemini-3.8-flash at 32.71%, Qwen3.8-max at 28.31%, and Claude-Fable-5.1 at 25.22%, each averaged across three runs under reported settings. Dai said the benchmark tests vision and reasoning together and exposes interpretable counting failures across seven mechanisms, because final-number-only tests cannot separate a vision miss from a reasoning error. The paper concludes that stronger general reasoning does not by itself generalize to aligned visual reasoning.