Google sets a new benchmark for testing what AI engineers can do

The test goes beyond textbook questions

Google's AI & Infrastructure team created Physical-Engineering's Last Exam, or PLE, in collaboration with Mercor. The benchmark is designed to measure how well leading models can perform work needed to build the physical world.

Google says existing tests, which are largely based on textbook questions, do not show whether an AI agent can produce a safe, standards-compliant and cost-effective design under the supervision of a licensed engineer. PLE aims to address this gap.

The full benchmark contains 85 long-horizon workflows. Of these, 30 are mechanical engineering tasks, 27 are electrical engineering tasks, 18 are civil engineering tasks and 10 involve structural design. The public release currently includes 29 tasks, their evaluation scripts and the evaluation container.

Agents must work in engineering project folders

The tasks are not built around answering a single question. Agents work in containerized project folders based on Mercor's Archipelago environment, which Google says simulates what a practicing engineer might receive for a real-world project.

Each task contains an average of 61 files, while the full benchmark includes 5170 files. These include PDFs, spreadsheets, emails, two-dimensional CAD drawings, three-dimensional BIM models and document versions. The agent must identify the applicable building codes and physical constraints, perform calculations, and then produce final materials that a qualified reviewer can inspect and approve.

Practicing engineers created 44 of the tasks, including the project folders, prompts and evaluation criteria. For a further 41 tasks, engineers designed the initial assignment, while an agent workflow assembled the environment and evaluator based on licensable engineering standards. Engineers reviewed and made each of these tasks more difficult.

None of the tested models reached 50 percent

Google tested 14 leading models with seven publicly available agent frameworks. The evaluation included Antigravity, Claude Code, Codex, Hermes, OpenCode, OpenHands and Pi.

PLE awards no points for a design that fails any of its strict safety checks. In Google's baseline evaluation, every model scored below 50 percent on the private test set. The top-performing model achieved an average score of 48.3 percent in its own framework.

The best agent configuration satisfied 91 percent of the task-specific constraints in the co-authored tasks, but only 41.3 percent of the submitted materials passed every strict safety and feasibility check. According to Google's data, model choice produced a difference of up to 50 percentage points, while the median difference between frameworks for the same model was 3 points. The framework also had a significant impact on the cost of running the system.

The researchers have also launched an open competition. Participants can submit their own agent configuration for the private test set at ple-bench.com, where 13 Google-supported models run free of charge on Google's servers. Details of the prizes will be announced later. The technical paper is available on arXiv, while the public tasks and evaluation tools are available on GitHub.