Benchmarks

Roblox shipped a Studio plugin for writing AI evals

September 8, 2026

On August 28, 2026, Roblox added an Eval Runner plugin to the OpenGameEval repository. It is a Studio plugin that finds eval tasks in your place, runs their checks in edit mode, then starts play mode automatically to run the runtime checks. One button runs both phases and combines the results.

No new leaderboard rows shipped with it. The scores are still the June 10 numbers. What changed is that the eval format stopped being an internal Roblox artifact and became something you can author and run yourself.

What the Eval Runner plugin actually does

An OpenGameEval task is three things: a prompt, a reference solution, and a set of checks. The benchmark measures whether a model can reach the same end state your reference does. Before this plugin, writing one of those tasks meant running the harness outside Studio and guessing whether your checks were correct.

The plugin closes that loop. Clicking Run does four things:

1. Runs setup(), then reference(), then check_scene() in edit mode
2. Injects server and client scripts if the eval has runtime checks
3. Starts play mode through StudioTestService and waits for results
4. Stops play mode, cleans up injected instances, and merges both sets of results

Results appear per check with a pass or fail marker and the message you wrote. Server results are prefixed [server] and client results [client:N]. An Export button copies everything to the clipboard as JSON.

Multiplayer checks come free

The part most Roblox developers will care about is the client handling. The plugin counts the entries in runConfig.clientChecks. One entry starts Play Solo. Two or more call StudioTestService:ExecuteMultiplayerTestAsync(N) and launch a server plus that many simulated clients, capped at eight. Each client gets an index from the server and runs only its own check function.

Roblox's own example uses this to test per-player state. Client 1 walks onto a coin and asserts its score went up. Client 2 stands still and asserts its score stayed at zero. That is a replication bug test, and it runs without touching the Test tab.

Client scripts drive input with UserInputService:CreateVirtualInput(), so a check can walk a character across a level rather than teleporting it and hoping the physics agree.

Reset only removes what the eval created

Before an eval runs, the plugin snapshots the contents of Workspace, ServerScriptService, ServerStorage, ReplicatedStorage, Lighting, StarterPlayer, StarterGui, StarterPack, SoundService, and Teams. Reset compares the current tree against that snapshot and destroys only what the eval created.

That means you can iterate on an eval inside a place you are also editing by hand. It is the difference between a tool you try once and a tool you keep open.

The limits are stated plainly

Roblox lists four, and they matter before you plan around the plugin:

• Eight simulated clients maximum per test
• Assets listed in includeAssets must be inserted manually
• The plugin never invokes the Roblox Assistant, so it validates evals rather than scoring models
• You open the target place file yourself; the place field is informational

The third one is the important caveat. This is not a way to benchmark Claude, Gemini, or GPT inside your own game. It is a way to make sure the test you would benchmark them against is correct.

Why this matters even if you never write an eval

The eval format is a good template for AI acceptance criteria. Every task it defines has a starting scene, a target end state, an assertion in edit mode, and an assertion at runtime. That is the structure that makes agentic work verifiable.

We covered the same idea from the model side in what OpenGameEval says about writing better Roblox AI tasks: bounded tasks with a named target and a checkable outcome produce the largest gains. The plugin is that finding expressed as tooling.

You can borrow the shape without the plugin. When you ask BloxBot to build a coin system, describe the check alongside the request. “Two players should have independent scores, and player two's score should stay at zero when player one collects a coin” gives the agent something it can verify. A vague request gives it nothing to verify against.

What else moved in the repository

The July 28 annotations are metadata only. Roblox is explicit that they are not shown to the model and do not affect grading. The July 30 removal drops one task from the code generation set, which is one more reason to read the June leaderboard as a snapshot rather than a scoreboard.

Frequently asked questions

Sources

• Eval Runner plugin user guide
• Studio plugin for eval running (#56)
• OpenGameEval leaderboard

Back to the guide