Why do Astra and Opus 5.5 Fail Simple 2D Mazes?

Pranav Guruprasad^{1, 2}, Sean Rivera², Helen Lu^{2, 4}, Arushi Jain², Harshvardhan Sikka^{1, 2, 3}

¹Fig; ²Manifold Research Group; ³Georgia Institute of Technology; ⁴Tufts University.

A few weeks ago, we released an early preview of MultiNet v2.0, where we evaluated Claude Opus 4.8, Kimi K2.6 and Qwen 3.6-27B on 50 interactive 2D MiniGrid mazes. The models solved 6 of 150 episodes, and 45 of the 50 mazes were never solved by any model. Based on this evaluation run, we reported three main findings:

• Progress decays with task horizon: Mean action progress falls from 0.287 on mazes solvable in 30 moves or fewer to 0.070 on mazes needing 61 or more.
• Models do not work out a mechanism they have no prior for: There were 0 solves in the 105 switch-gate maze episodes, while 4 of the 6 solves came on key-door mazes.
• Reasoning budget does not buy solves: The two models that spent about 20x or more the output tokens per new tile solved one maze each, against four for the model that spent the fewest.

New frontier models such as GPT-6 Astra and Claude Opus 5.5 have been released since then, and have showcased significant capability jumps across environments and domains. In order to evaluate their long-horizon action and causal reasoning capabilities, specifically in comparison with the previous generation of frontier models, we ran both models on the same 50 mazes with the same protocol. With this follow on evaluation run, we primarily aimed to understand which of the three main findings from our previous evaluation still hold at the frontier.

Evaluation Setup

Evaluated Models

We evaluated GPT-6 Astra and Claude Opus 5.5, and both models ran with thinking at xhigh and a 64k output token cap, which is the same cap the models ran under in our previous evaluation setup.

Maze Set and Evaluation Protocol

The evaluation set is the same 50 mazes from our earlier evaluation effort: 11 mazes that are 8x8 grids, 19 that are 10x10 grids and 20 that are 14x14 grids. 35 of the mazes contain a switch-gate mechanism and 15 do not. How the maze set was constructed and validated is described in detail in our previous technical report.

The evaluation protocol for this run was kept identical to our previous work, which we had finalized based on an extensive 540-episode ablation run across a variety of protocol settings:

• Minimal prompt: The system prompt that provides information about the task contains a simple, high-level task instruction and the list of valid actions in the action space only. The rules of the environment, and the ways to operate the mechanisms are not provided to the model.
• Image-only observation: The current state is provided to the model as a rendered image, with no associated text description.
• Egocentric actions: Turn left, turn right and move forward relative to the direction the agent faces, plus pickup, toggle, drop and done actions.
• Zero-shot: Trajectory of a solved maze is not provided as reference to the model.
• History: The history of progress given to the model at each timestep of an episode includes - the start position, a prose summary of every mechanism event until the current step, and the three most recent visual observation frames, along with the inventory and the action the model chose at that step.
• Stateless turns: Each query corresponding to a single timestep in an episode is one self-contained message. Earlier turns from the model’s progress until the current timestep are not provided as history in a multi-turn conversation.
• Stall watchdog: If for 30 steps in a given episode, the combination of position of the model, inventory it accumulates and mechanism state remains unchanged, the episode is terminated as a stall kill.
• Step cap: The optimal actions required to solve the maze is calculated using Breadth-First Search. The models get 3 times the optimal step length to complete a maze, so the step budget scales with the maze difficulty instead of being a flat upper limit across varied difficulty levels.

The rules for interacting with objects and mechanisms are not stated either. A key is picked up by standing on it, and a switch is toggled by standing on it to open its corresponding gate. A door is toggled from the square in front of it, and a gate opens only through its switch. One major difference between our environment’s rules and the standard MiniGrid setup is that objects are picked up by standing on the same cell as them in our setup, as opposed to the adjacent cell in default MiniGrid.

Metrics

A solve means the agent reaches the goal tile of a given maze. Action progress is defined as:

where $d_{\text{closest}}$ is the closest the agent came to the goal and $d_{\text{start}}$ is the distance from the start, both counted in the executable actions still needed, including the actions to clear any locked door or closed gate. A 0 score means that the model made no progress in a given maze, and a 1 score refers to a successful run. A stall kill refers to when an episode is terminated by the stall watchdog.

Difference from our previous set up

The only difference from the setup of our previous evaluation run:

• Switch rendering: In our previous environment setup, a switch was rendered as a solid disc whatever its state, the same shape as MiniGrid's pickup-able ball. For the new frontier model runs, the switch rendering was modified to be a ring whose centre is dark when it is off (not toggled) and white when on (Figure 1).

Results and Analysis

Solve Rates and Progress

As seen in Table 1, Astra solved 11 of the 50 mazes and Opus 5.5 solved 7, against 4 for Opus 4.8, the best performing model from the previous evaluation run. Every frontier model solve came on a switchless maze: Astra solved 11 of the 15 switchless mazes and Opus 5.5 solved 7, while both are at 0 of 35 on mazes with a switch.

In this evaluation run with the frontier models, the longest grid-size mazes saw multiple solves unlike our previous evaluation run. Astra solved three 14x14 mazes that need 77 to 85 actions and Opus 5.5 solved four that need 77 to 89, while the models in the previous run solved 0 of their 60 episodes at that grid size. Six of Astra's solves and five of Opus 5.5's are on mazes that no model in the previous run solved. The two frontier models mostly solve the same mazes. Between them they solve 12 distinct mazes: 6 solved by both, 5 by Astra only, and 1, the 89-action 14x14 corridor in Figure 3, by Opus 5.5 only.

Every failure of both frontier models was a stall kill. All 39 of Astra's failures and all 43 of Opus 5.5's ended on the stall watchdog (Figure 4). The frontier models also stall later in the episode when compared to the models from the previous evaluation run. Opus 4.8 stalled after exploring 4 tiles or fewer in 28 of its 46 failures, and Kimi and Qwen in 10 of 49 each, while Astra did so in 1 of 39 and Opus 5.5 in 3 of 43.

What Changed at the Frontier

Task horizon is no longer what limits the models, and is not the biggest predictor of a run success or failure. Shortest-path length accounts for 21.2% of the variation in Opus 4.8's action progress, 17.9% for Qwen and 10.9% for Kimi, but only 1.9% for Astra and 0.3% for Opus 5.5. These are R² values from one regression per model over its 50 episodes, where R² is the share of the variation in progress that path length explains. When ranked, progress falls as mazes get longer for all three models from our previous evaluation run (rank correlation of -0.84 for Opus 4.8, -0.51 for Qwen and -0.23 for Kimi), and shows no relation for Astra (-0.05) or Opus 5.5 (+0.08). As seen in Figure 5, progress for the two frontier models stays flat as the mazes get longer, and Opus 5.5 solves a 14x14 corridor that takes 89 actions, which no model had solved before.

Running into walls does not seem to be an issue with the frontier models. 7% of Astra's forward moves and 2% of Opus 5.5's are blocked by the wall, against 82% for Opus 4.8 and 51 to 52% for Kimi and Qwen. Opus 5.5 hit a wall only once in 1,065 forward moves.

Both frontier models reason more when they are stuck. In the last 30 steps before a stall kill, Astra reasons 1.57 times more per turn than earlier in the same episode, and Opus 5.5 1.55 times more. This holds in 38 of Astra's 39 stalls and 37 of Opus 5.5's 43 stalls. The models from our previous evaluation run did not showcase the same trend: Opus 4.8 reasoned less when it was stuck (0.63 times), and Kimi and Qwen stayed about the same (1.07 and 1.03 times).

In absolute terms, Opus 5.5 reasons more per turn than Astra, with a median of 1,230 tokens against 535. Both use far fewer than Kimi (about 20,300) and Qwen (about 16,400), but more than Opus 4.8 (about 282). Neither frontier model came close to the 64k output cap, with largest single outputs of 13,655 tokens for Opus 5.5 and 3,965 for Astra, while Kimi and Qwen each hit the cap once.

Across episodes, reasoning increases as progress decreases only for Astra. Among episodes of similar shortest-path length, Astra's episodes with more reasoning per turn got less far. For Opus 5.5, Kimi, and Qwen we see no reliable relation.

Astra is also the most economical explorer. Pooled over all episodes, it spends 2,172 output tokens per newly visited tile, against 4,504 for Opus 5.5, 6,297 for Opus 4.8, 123,120 for Qwen and at least 162,222 for Kimi. Per episode, the medians are about 3000, and 6400 (Figure 7) for Astra and Opus 5.5 respectively, and about 4300 for Opus 4.8, 134000 for Qwen, and 179000 for Kimi, so Astra is still the lowest, but a typical Opus 5.5 episode spends more per tile than a typical Opus 4.8 episode. In total, Opus 5.5 produced 4.10M output tokens, 1.99 times Opus 4.8's and 1.76 times Astra's, against 48.0M for Qwen and 67.0M for Kimi.

How the Frontier Models Fail

The two frontier models fail in different ways:

Astra walks back and forth over explored ground: In its last 30 steps before a stall, Astra hits a wall on only 5% of its actions (91% for Opus 4.8, 28% for Kimi and 26% for Qwen), so it is still moving freely. However, the final 30 steps cover a median of just 7 distinct cells, all of them previously visited, so it walks back and forth over the same short stretch of the maze without exploring a new tile. Its failed episodes explore a median of 16 tiles, against 4 to 7 for the models from our previous evaluation run. On switchless failures, it turns back at a corridor corner and shuttles along corridors it has already walked.

Opus 5.5 acts on the object in front of it: Opus 5.5 rarely gets blocked, but its failures are due to actions that do nothing: 877 of them, and 871 were an interaction aimed at the object directly ahead, mostly PICKUP at a key, TOGGLE at a closed gate, and PICKUP at a switch. In the last 30 steps before its stall kills, 59% of its actions had no effect, against 10% for Astra and 4 to 6% for the models from our previous evaluation run.

A key can be picked up only from its own cell, which is information we do not explicitly provide to the models. Opus 5.5 issued PICKUP while facing a key from the adjacent cell 570 times, and while standing on one only 7 times (34 and 29 for Astra, 12 and 7 for Opus 4.8, 3 and 13 for Kimi, 3 and 12 for Qwen). It got into that position in 26 of the 31 key-door mazes and recovered in only 7. All 8 of its failures on switchless mazes end in this loop (Figure 8).

When it is not caught in this loop, its navigation is perfect. Its four solves that need no key used the minimum number of steps (23, 23, 77 and 89), and on its three key-door maze solves, the extra steps exactly equal its failed PICKUPs (2, 3, and 12). None of the other four models has a single solve in the optimal number of steps.

A failed action leads to an unchanged observation state and summarized history, and no prompt explicitly says that an action failed, so the next prompt is identical to the last. 472 of Opus 5.5's 2,599 prompts were byte-identical to the one before, and on 440 of them it chose the same action again. In one key-door maze, queries 6 to 32 were a single repeated prompt.

The Switch Mechanism

The switch-gate mechanism remains completely unsolved by every model we have evaluated so far. Astra and Opus 5.5 both solve none of the 35 switch-gate mazes, similar to Opus 4.8, Kimi, and Qwen in our previous evaluation run. Furthermore, none of these models issued a toggle correctly while standing on the same cell as a switch.

A switch costs Astra and Opus more progress in a given maze episode compared to what it cost the models from our previous evaluation run. Holding the shortest-path length fixed, a switch-gate mechanism lowers action progress by 0.57 for Astra (95% CI 0.43 to 0.70) and 0.38 for Opus 5.5 (0.22 to 0.54), compared to 0.22 for Opus 4.8 (0.08 to 0.35), 0.14 for Qwen (0.03 to 0.26) and 0.07 for Kimi (-0.03 to 0.18). The frontier models get much further on switchless mazes, with mean progress of 0.85 and 0.60 against 0.24 - 0.31 for the models from our previous evaluation run. (Figure 10).

Even with the clearer rendering of the switch, neither frontier model solved a switch-gate maze. Astra finds the switch, but uses the wrong action on it. It stood on a live switch in 18 of its 35 switch-gate maze episodes, and issued PICKUP on the switch 33 times while standing on it and 30 times from the adjacent cell while facing it. 63 of its 66 TOGGLEs that did nothing were aimed at the gate, as if the gate were a door. It never toggled a switch it was standing on, which is the only action that opens the gate.

Opus 5.5 treats the gate as a door. It reached a live switch in 11 episodes and aimed 108 PICKUPs at switches (100 from the adjacent cell and 8 while standing on one) against 1 TOGGLE. 199 of its 200 TOGGLEs that did nothing were aimed at the gate from the adjacent cell.

When counting PICKUPs alongside TOGGLEs, we see that the models from our previous evaluation run also read the switch as an item. Kimi issued 21 and Qwen 17 PICKUPs while standing on a switch. Opus 4.8 issued 3 while standing on one and 22 from the adjacent cell while facing it, across 6 of its 35 switch-maze episodes.

Prompt Tests on the Key and Switch Mechanisms

The prompt in the evaluation run lists PICKUP and TOGGLE without saying which objects they act on or from which cell the action needs to be executed, and a failed action only leaves the next frame unchanged. The switch-gate maze failures and Opus 5.5's repeated PICKUP execution could mean the models cannot work out these rules. So we ran a small set of tests that add rules to the prompt one sentence at a time.

Each test is one episode per model, with the same model configuration and protocol as the main run. Each test appends text after the valid-actions line in the prompt, and changes nothing else. Opus 5.5 and Astra ran 4 identical tests, and Opus 5.5 ran an additional test aimed at its repeated PICKUP execution behavior. Tests 1 to 3 use the smallest switch maze (8x8 corridor, 26 actions, one switch, one gate, no decoys), which both models failed in the main run, where Astra stalled at step 45 with 0.08 progress. Test 4 uses the smallest key-door-and-switch-gate maze (29 actions: a key-door, then a switch-gate), where Astra had opened the door and stalled at the gate with 0.48 progress, and Opus 5.5 had stalled at step 51. Test 5 uses an 8x8 key-door maze (28 actions) where Opus 5.5 had repeated PICKUP for 27 queries. Each result is a single episode, so these are cursory signals that give us a better understanding, as opposed to a major finding.

As seen in Table 2, each model needed a different sentence to solve the mechanisms. "Switches must be toggled." got Astra through the gate and to the goal, while Opus 5.5 toggled the switch from the adjacent square, which does nothing in our environment. "You must stand on a key/switch to act on it." got Opus 5.5 through both mazes it was tried on in the optimal number of steps, while Astra, given the same prompt, passed the switch and turned back short of the goal.

A general principle did nothing for either model. In test 2, both models could see that their actions changed nothing, and both stalled at step 43.

Discussion

Both frontier models are a clear step up from the previous generation, and Astra is the stronger of the two in the current setup:

GPT-6 Astra and Opus 5.5 solve 11 and 7 of the 50 mazes against 4 for the best model from our previous evaluation run, with mean action progress of 0.45 and 0.34 against 0.14 to 0.18 for the previous models. Astra also spends the fewest output tokens per new tile of any model, both pooled and per episode, making it the most efficient model as well.

With the new frontier, unexplained mechanisms is the limiting factor:

For the models in our previous evaluation run, progress fell with optimal path length. For both frontier models, optimal path length explains at most 2% of the variation in progress, and both solve 14x14 mazes of 77 actions or more. What stops them now is acting on objects whose rules are unstated: the switch-gate mechanism for both models, and key-door as well for Opus 5.5.

Both models apply a familiar convention based on their priors and do not revise it when it fails:

Astra and the models from our previous evaluation run treat the switch as an item to pick up. Opus 5.5 acts on keys and switches from the adjacent cell, the way a door works, and treats the gate as a door. Each failed action leaves the frame unchanged, and the model does not switch to an alternative action in the available action space. Revising a belief from one's own failed actions is one of the capabilities these mazes were built to measure.

Extra reasoning does not get the models unstuck:

Both frontier models think more in the final steps of a stall, but are not able to reason their way out of the stall to a solve. This is consistent with our previous finding that a larger reasoning budget does not buy solves.

The lack of a failure signal in the environment makes discovery hard and errors sticky:

A failed action changes neither the visual observation state, nor the summary of recent steps, so a misread rule by the model is never corrected by the information provided to the model. The model has to notice that nothing in the environment changed after the action it executed, which is the discovery these mazes test.

Keeping track of explored ground is something Astra struggles with in the current evaluation setup:

Astra's failures involve the model shuttling over cells it has already visited, and in test 4 of the prompt experiments, after getting past every mechanism, it turned back three squares from a visible goal. Providing just three previous visual observation frames and a text summary of actions up until the current step seems to be insufficient for the model to track its progress over a long episode. In the experiments we are running next, every previous visual frame and action taken by the model is provided to the model as history, so that we can measure how this affects performance.

What Comes Next

This early preview to the MultiNet v2.0 benchmark ran models under a deliberately minimal setup and in one modality of observation, to assess basic performance and characteristics of frontier models in interactive environments that are meant to evaluate long-horizon action taking and causal reasoning. GPT-6 Astra and Claude Opus 5.5 now clear long mazes under that setup. Both frontier models still fail on mechanisms whose rules are unexplained, and each of them have a distinct mode of failure as well - Astra loses track of the maze regions it has already explored, and Opus 5.5 repeats actions that do not change the state of the environment. Our next set of experiments asks the complementary question: given a reasonably strong setup, can current frontier models solve harder mazes with several unfamiliar mechanisms, and does that hold when the same maze is presented in different modalities? Can it truly generalize across modalities?

A stronger setup to help the frontier models:

• Unbounded history: Every previous visual frame and action taken by the model is provided to the model, instead of just the three most recent frames and a text summary.
• Step-budget awareness: The model will be told how many steps it has taken up until the current step in a given episode, and how many remain before the episode is terminated due to a stall kill or step cap.

The same underlying environment in multiple domains: Each maze is presented as a 2D grid as in this evaluation run, as an egocentric 3D environment, and in natural language. The natural language domain rendering of the maze provides a JSON representation and ASCII representation as 2 format variations, and an additional prose representation of the maze to test the capability of models to spatially reason based on just language. The underlying maze and the optimal solution are identical across the different domains, so a gap in the performance of the models between domains measures how much of the score depends on the familiarity of the interface and the way the environment is built and how much of it is actual task understanding.

Work with us

At Fig, we are building the control layer for AI: systems that perceive environments rendered in various modalities, act in them reliably over long horizons, and improve from their experience continually.

Measuring progress towards this goal requires evaluation setups and benchmarks representative of complex, real-world, multi-domain workflows. This is what we aim to do with MultiNet, in collaboration with our friends at Manifold Research, MIT, Georgia Tech, and Tufts.

If you build models or agents, or work on benchmarking and evaluation, we want to hear from you - whether that means getting your model on our MultiNet benchmark, contributing to the environments we are building, or working with us on what comes after.

Citation

Please cite this work as:

Guruprasad, P., Rivera, S., Lu, H., Jain, A., Ren, H. and Sikka, H. (2026) Why do Astra and Opus 5.5 fail simple, interactive 2D mazes? Available at: https://www.fig.inc/multinet-v2-preview-astra-opus

Or use the BibTex citation

@online{multinet_v2_frontier_mazes_2026,
  title   = {Why do Astra and Opus 5.5 fail simple, interactive 2D mazes?},
  author  = {Pranav Guruprasad and Sean Rivera and Helen Lu and Arushi Jain and Hangliang Ren and Harshvardhan Sikka},
  year    = {2026},
  url     = {www.fig.inc/multinet-v2-preview-astra-opus},
  note    = {MultiNet 2.0 Preview: Frontier Models on Interactive 2D Mazes}
}