Open-source benchmark of 1,632 tasks across five domains and 19 simulation skills that tests whether LLMs can accurately simulate RL environments by predicting next-state responses given agent trajectories. GPT-6 Astra achieves 49.57% success; no other of 20 models exceeds 30%. Tasks, harness, model outputs and judge decisions released on GitHub and Hugging Face.
GlobeBench evaluates whether language models can predict an environment’s next response from an agent’s action-and-observation history. Its 1,632 execution-grounded tasks cover five domains and 19 simulation skills, with predictions compared against reference outcomes produced by executing actions in each task’s backend. Patronus AI reports results for 20 general-purpose and specialized models; the benchmark paper describes a hybrid grader that combines deterministic checks with an LLM judge. The project releases its benchmark items, evaluation harness, model outputs and judge records for inspection and rescoring.
