Visual Cipher Bench Finds Frontier AI Agents Struggle With Hidden Rules

Einsia AI's Navers Lab has launched Visual Cipher Bench, a benchmark for testing whether AI agents can discover hidden rules from visual clues and solve long-horizon tasks. Across 3,480 attempts involving eight frontier models and 29 reasoning configurations, no configuration exceeded 30% accuracy.
- Visual Cipher Bench contains 40 puzzles across four categories.
- GPT-6 Astra (max) led the evaluation with 26.7% accuracy.
- Rule induction accuracy was 0.6% across 3,480 attempts.
- Providing core rules increased overall accuracy from 12.2% to 51.1%.
A benchmark without explicit instructions
Visual Cipher Bench is designed to test a capability that conventional visual question-answering benchmarks generally do not measure: solving problems when the instructions and rules are hidden in images. Agents receive one instruction, examine the images, and write a final answer to a file.
The benchmark contains 40 original visual puzzles represented by 67 black-and-white illustrations. To complete them, an agent must identify examples, understand the target problem, infer the required output, and solve each task without explicit rules or immediate correctness feedback.
The puzzles are divided into four categories: spatial operations such as folding, rotation, projection, and geometric transformations; logical planning; rule induction; and knowledge decoding involving Morse code, Braille, and word puzzles.
GPT-6 Astra leads, but rule induction remains weak
GPT-6 Astra (max) achieved the highest result, with 26.7% accuracy and an average of 10.7 correct answers per 40-puzzle run. GPT-6.1 Sol (max) and Claude Opus 5.5 (xhigh) each scored 21.7%.
GPT-6 Luna and Claude Sonnet 5.5 each scored 0.8%, solving one puzzle across 120 attempts. DeepSeek V4.1 Flash, Qwen 3.8 Max, and Kimi K3.1 Pro recorded zero correct answers.
Spatial operations were the strongest category, at 22.8% accuracy across all attempts. Knowledge decoding reached 3.3%, logical planning reached 2.4%, and rule induction reached only 0.6%. Models solved five of 870 rule-induction attempts. GPT-6 Astra (max) reached 80% on spatial operations, while no configuration exceeded 13.3% in the other three categories.
Einsia AI said some correct answers appeared to result from thematic guesses rather than systematic rule discovery, further separating answer execution from independent reasoning.
Rule hints sharply improve execution
An additional experiment tested 10 puzzles with GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna. When researchers supplied the core rules, without providing intermediate reasoning or answers, overall accuracy rose from 12.2% to 51.1%. Rule-induction accuracy increased from 0% to 85.2%.
GPT-6 Astra improved from 23.3% to 70.0%, while GPT-6.1 Sol rose from 13.3% to 66.7%. Timeout occurrences fell from 51 to 29 across 90 attempts. A control experiment that supplied information about task-state representation, but not the core rules, produced 16.7% accuracy.
The researchers identified premature conclusions and inefficient search as recurring failure patterns. Agents sometimes accepted rules that explained only part of the evidence, or used extensive tool calls and combinations until they timed out. The findings suggest that discovering, validating, and transferring hidden rules remains a major obstacle for agents expected to handle less structured real-world tasks.

