What an Evaluation Harness Decides

A multimodal model does not play Atari by itself. An evaluation harness turns pixels into a prompt, text into an action, and environment transitions into a score. When I built the harness for Atari-GPT, each conversion required a choice. Those choices determined the system we actually tested.

The model was only one component. The observation transform, prompt history, output parser, recovery rule, environment clock, and aggregation procedure all stood between the model and the reported result. Calling that code “plumbing” hides part of the experiment.

Define the tested system

In the final arXiv version, each 210-by-160 Atari frame was resized to 512 by 512. The model received the current frame plus two previous frame-response pairs, a game-specific list of legal actions, and a request for a JSON object containing an explanation and a numerical action. The environment used a frame skip of eight.

We ran four 1,000-timestep rollouts for each model and game. If the environment reached a terminal state, it reset while reward continued accumulating within the same rollout. The experimental unit was therefore a fixed interaction budget, not one episode or one life.

None of these choices changed the model weights. Every one changed the task. A useful result should identify this full system boundary as plainly as it identifies the model.

History and demonstrations are different variables

In the final protocol, the two previous frame-response pairs gave an otherwise stateless API short-term memory. They could reveal motion between frames, but they also exposed the model to its own earlier explanations and decisions.

In-context demonstrations supplied a different kind of information. The first arXiv version records the experiments I ran: the prompt prepended a curated set of four human-annotated frames with reasoning and action labels, while game play used the previous and current frames. That difference from the final context window is itself a reason to version the harness with the result. History answers “what just happened in this rollout?” A demonstration answers “what action did a person choose in an example selected before this rollout?” They should not be merged into a single notion of context.

A more diagnostic follow-up would vary them independently: neither, history only, demonstrations only, and both. The model version and decoding settings, observation sampling, action vocabulary, starting conditions, and rollout budget would otherwise stay fixed. Demonstration identity and order should be recorded as experimental inputs.

The parser changes the policy

Asking for JSON creates a second task alongside playing Atari: obeying the interface. A response may express a sensible action while missing a brace, using the wrong field, or returning an action outside the legal set. The harness must decide whether that is a failed decision, a recoverable formatting error, or text it is willing to interpret.

Atari-GPT sent a corrective follow-up when the model returned an invalid action. That is a defensible policy, but it measures a model paired with a repair mechanism and spends another call. Rejecting the response or coercing it into a legal action would produce different behavior from the same raw output.

I should move left to avoid the incoming shot.
Action: 3

Reward alone conceals these distinctions. At minimum, a decision record should preserve the raw response, validation result, number of repairs, executed action, and time spent recovering. Invalid-output and repair rates belong beside the task score.

Choose the clock

The APIs in our experiments took roughly two to seven seconds to return an action, depending on the model, and provider rate limits added delay. An emulator score is indexed by simulated game steps. A deployed controller must also obey wall-clock time.

In the first public harness snapshot, the model call completed before the environment stepped. The emulator did not advance while the provider answered. Reward therefore measured action choice in paused simulated time; latency remained a separate control limitation.

A real-time loop may instead act on an observation that is already stale. A corrective retry may cost nothing in simulated time while making the controller slower in real time. The two systems can receive the same game score and have very different control performance.

The harness should state whether the environment advances during inference and whether one action persists across skipped frames. Return and end-to-end decision latency should be reported separately; latency should include parsing, retries, and provider failures rather than only the nominal model call.

Make the denominator visible

“One rollout” is not a universal unit. It may mean one episode, a fixed number of decisions, a fixed number of emulator frames, or a wall-clock budget. Reset rules, seeds, failed calls, truncated runs, and the order of averaging can all change a final number.

For a fixed-horizon study like Atari-GPT, one row per rollout, the reference baselines, and a versioned aggregation formula should make the final score reconstructible. Each row should include its initial condition, total reward, resets, invalid responses, repairs, failed calls, and a latency summary. The decision-level trace can then distinguish a bad action from a protocol failure or a service failure.

Publish the measurement contract

Before publishing a closed-loop score, five questions should have short answers:

• Observation: What did the model see, and what history was retained?
• Action: Which outputs counted as legal environment actions?
• Recovery: What happened after malformed output or a failed call?
• Clock: Which time advanced while the model was deciding?
• Aggregation: How did steps, resets, and rollouts become one score?

There is no single correct answer to all five. The point is to make the answers inspectable and version them with the result. A harness is an executable measurement contract. Once that contract is visible, readers can tell whether an improvement came from the model, the interface around it, or both.

Related: Atari-GPT overview → · arXiv v1 ↗ · harness snapshot ↗