Goal Progress Decays with Task Horizon for Frontier VLMs in Interactive 2D Environments

Pranav Guruprasad^{1, 2}, Sean Rivera², Helen Lu^{2, 4}, Arushi Jain², Hangliang Ren², Harshvardhan Sikka^{1, 2, 3}

¹Fig; ²Manifold Research Group; ³Georgia Institute of Technology; ⁴Tufts University.

We evaluated 3 highly capable Vision-Language Models (VLMs) on 50 2D mazes. Across the 3 models, only 6 total mazes were solved, and 45 of the 50 mazes were not solved by any model.

The size of the mazes ranges from 8x8 to 14x14 grids. An agent has to navigate through walls, avoid decoys and distractors, and operate objects that are to be operated in a specific order (a key that opens a door, a switch that toggles open a gate), in order to reach a goal tile and solve the maze. The action space consists of six actions: turn left, turn right, move forward, pickup, toggle, and done. Apart from the action space and the instruction to complete the maze by reaching the goal tile, the models are not given any other information about the environment. They have to figure out what to do by exploring the environment, acting in it, and observing the changes in the environment.

When a benchmark like the one we present records a zero for a model failing an episode, there are multiple possible reasons for the failure:

• The model never worked out what an object in the environment does.
• It worked out what the objects are, but could not operate them in the right order.
• It executed the wrong actions from the action space.
• It made mistakes and could not recover from them to get back on the right path.

All of these are completely different failure modes, and each of them requires a very different fix. Going down the wrong path in a scenario such as this is expensive in both time and resources spent, and the cost of not identifying the right failure mode is high: a survey of agentic benchmark construction found flaws that mis-estimate performance by up to 100% [1].

Separating the modes of failure means changing the task in different ways: running a maze without the switch-gate mechanism, running it with the optimal path length controlled, and so on. Whichever change moves the score helps narrow down on the failure mode of the model or agent.

In this report, we describe an environment built to evaluate long-horizon action and causal reasoning capabilities in models, which by the nature of its design also helps separate the different possible failure modes in models.

Designing an environment that can attribute failure

When a model has to act over a long horizon in an environment whose underlying rules it has not been told, where, how, and why does it break?

Through our evaluation in this work, we find that models:

• Solve 6 out of 150 episodes. 45 of the 50 mazes were not solved by any model, and outside the six solves no episode ended within three executable actions of the goal.
• Fail on mazes with a longer path length to the solution. Mean action progress falls from 0.287 on mazes solvable in 30 moves or fewer to 0.070 on mazes needing 61 or more.
• Are unable to learn how a mechanism works without prior knowledge. Across the 105 episodes containing a switch and a gate there were no solves at all, while key-door mechanisms account for four of the six solves.
• Fail to solve a majority of the mazes despite reasoning with an extended thinking token budget. The two models that spent more than 20x the output tokens solved one episode each, against four for the model that spent the fewest.

In order to identify and separate these different failure modes, we believe that four properties are needed in an environment:

1. Knowledge-free: No pre-required domain knowledge. A failure cannot be explained away as an unfamiliar interface and a human baseline requires no training in order to be an appropriate comparison.
2. Generative: If the environment is built as a composition of multiple parameters, rather than entirely designed by hand, there is no static answer key that could get contaminated by frontier model pre-training, and the difficulty scales with the models instead of saturating
3. Projectable: Because the environment structure carries no domain content, the underlying task can be rendered in language or in 3D or any other modality. Measuring performance on the same underlying task across modalities is a good measurement of capability instead of familiarity with an interface.
4. Verifiable: A method to obtain the exact optimal action sequence from any reachable state yields an objective difficulty scale and the ability to provide partial credit to a trajectory [2].

Mazes with causal mechanisms are a simple representative structure carrying the 4 properties discussed above. In MultiNet v2.0-Gridworld, we run evaluations on this substrate in a very basic rendering.

The Evaluation Environment

The underlying task we present in this evaluation is a model in a 2D Minigrid [3] maze with an action space containing 6 valid actions: turn left, turn right, move forward, pickup, toggle, and done. A model is required to navigate corridors bounded by walls, decoys and distractors, and mechanisms that need to be understood and operated in a specific order, with an end goal of reaching a target tile. Nothing about the environment is explicitly explained, and needs to be figured out by the model exploring the environment.

We chose to keep the environment and its components unexplained because prior domain knowledge is a confounding variable. Measuring capability requires controlling for priors [4], and with this design, a failure cannot be attributed to an unfamiliar API, library or interface, because there are none.

At each turn, the model is provided with persistent information on where it started in the maze, a prose activity summary of every mechanism event so far, and up to four rendered observation frames: the 3 most recent steps with inventory and the action taken at each, then the current frame with a request for the next action. The model receives no progress signal and no explicit verbal information that a move it took failed.

In order to succeed, the model needs to navigate to the goal tile - this is considered a solve. Progress is another metric we track, and is defined as:

1 − d_remaining / d_start

where d_remaining is the remaining oracle distance from where the model ends the episode, and d_start is the total oracle distance from the start state as calculated by a BFS implementation. This is computed over two ways: over tiles, which ignores the barriers, and over executable actions which accounts for them. The action variant makes more sense to rely on because the tile variant credits the model for being close to the goal even if it is sealed behind a door it never opened.

We implement 2 termination conditions other than the solve - a step cap, and a stall watchdog. The step cap is an upper limit on the number of steps the model is allowed to take in a given episode, and is defined as 3x the optimal BFS step count. The stall watchdog is implemented to prevent models wandering aimlessly and terminates a model’s episode if there is no novelty in position, inventory, and mechanism state together for 30 steps. This makes the step budget for a given episode relative to the difficulty of the maze instead of a flat action step cap. We observe in our evaluation results that the watchdog terminates almost every episode. Only 11 out of 150 episodes were terminated due to a success or the model hitting the step cap.

What does each mechanism isolate?

Solving each of the mazes in our evaluation set requires 4 things in sequence - work out how the mechanisms in the environment work, reason about the order in which the mechanisms need to be operated, execute the right actions, and recover in case something goes wrong. Each mechanism is included in the environment because it tests one of the 4, and because each can be added or removed independent of the others - failure modes can be classified accurately.

• Keys and doors test whether a model can understand and operate a mechanism it has prior knowledge about. All frontier models have seen keys being used to open doors in their training data and do not have to explore the environment to understand how they work.
• Switches and gates test whether a model can figure out how to operate a mechanism they are not familiar with. A switch is rendered as a colored circle on a cell, while a gate looks like a blocked cell. A model needs to explore and understand how the switches and the gates are associated and what action needs to be executed at what exact cell in the maze in order to be able to resolve the mechanism and make forward progress in the maze.
• Dependency chains test whether models are able to reason about the ordering of actions they need to take to resolve mechanisms in a sequence and make forward progress in the maze. For example - a gate that opens only once a switch is toggled can be behind a door that opens only once its key is carried.
• Distractors test error recovery - decoy keys, inactive switches, and dead-end branches cost moves to the model without moving them forward in the maze towards the goal. Upon interacting with these distractors, the models need to understand that they play no role in helping them reach their goal, plan the steps to get back on track, and execute the actions in the right order

Evaluation Setup

Designing the complete evaluation required several decisions regarding the configurations of each parameter of the environment as well as the protocol of a model’s interaction with the environment. In order to finalize these decisions, we ran a set of structured ablation experiments across 540 episodes, 12 different settings, on 15 held out mazes.

Evaluated Models

We evaluated 3 frontier vision-language models: Claude Opus 4.8 with adaptive thinking at the xhigh setting, Kimi K2.6 with thinking, and Qwen 3.6 27B with thinking. Claude and Kimi were queried via the native API, and we hosted Qwen 3.6 in parallel on two A100 80GB Virtual Machines to optimize runtime.

Ablation-based Evaluation Protocol

As seen in Table 1 below, the evaluation protocol sweep holds a baseline set of conditions fixed and varies 12 conditions across 7 different axes, each experiment changing exactly one field. We used 15 mazes that are held out from the 50 that were used for the final evaluation in order to ensure that protocol tuning would not leak into the benchmarking scores.

The standard baseline evaluation protocol gives the model a prompt stating the task, the action space which includes actions relative to the direction the agent is facing, and the locations of the mechanisms in the maze, but does not give the model information about how to solve the mechanisms. The observation of the current timestep is provided as an image accompanied by the text description of it. To provide historical context on the model’s progress up until the current timestep, the images of the previous 3 states are provided to the model as a part of the observation. Given all the visual and text context, the model is prompted to produce the immediate next action and each query is a single message with no previous turns from its history included. To guide the model on how to complete a maze, one in-context example is given as reference.

As seen in Figure 3, not providing text information about the current timestep’s observation significantly reduces performance. On the other hand, varying the amount of detail in the prompt barely seems to affect model performance.

Based on this extensive sweep, the evaluation protocol was finalized with the following settings:

• Thinking setting on for all the models, with an equal 64k token budget
• Minimal prompt setting since models do similarly performance-wise on the different prompt variations and also incentivize models to explore the environment and reason in order to succeed
• Image as the only modality of perception of the current state. This was picked intentionally to make the benchmark more challenging and also incentivize the models to learn about the consequences of their actions in the environment by observing and understanding the difference between 2 states - pre and post action, as often needs to be done in real-world workflows

Task Pool and the Evaluation Set

The initial pool of mazes that we created using a combination of hand-design and LLM-based generation included 214 mazes that were all confirmed solvable. The shortest path required to solve a maze ranged from 9 to 106 executable actions across all the mazes. From this initial pool, we picked a set of 50 mazes such that they were balanced and stratified across 5 path-length bands - 2 mazes requiring 20-25 moves, 12 mazes requiring 26-39 moves, 14 mazes requiring 40-59 moves, 8 mazes requiring 60-79 moves, and 14 mazes requiring more than 80 moves. The grid sizes for the selected maze set are 8x8 for 11 mazes, 10x10 for 19 mazes, and 14x14 for 20 mazes.

The mazes are grouped into 3 different categories - Scale, Mechanism, and Distractor. The evaluation set included 3 scale mazes - which are mazes that require only navigation, 37 mechanism mazes - which include keys, doors, switches, gates in a dependency order, and 10 distractor mazes which includes decoys on top of mechanisms - such as a wrong key, a switch that does nothing, or a dead-end branch.

When generating the initial pool of mazes we varied grid size, topology, dependency chain pattern, mechanism types, mechanism counts, and distractor types and counts. Every maze instance was confirmed reachable by an exhaustive search under a 500,000-state cap, and the optimal action count is computed with a BFS implementation that calculates the actions required to solve the mechanisms as well. We also did 3 further checks to thoroughly validate each maze:

• Mechanism necessity - reports any mechanism whose removal does not affect the solvability of the maze
• Chain ordering - confirms that each mechanism is unreachable until the prior mechanism in the dependency ordering has been solved
• Distractor safety - confirms that no single distractor interaction can make the task unsolvable

Results and Analysis

Solve Rates and Progress

As seen in Figure 4, only 6 out of 150 mazes were solved totally by all 3 models. Claude solved 4 of its 50, Kimi 1, and Qwen 1. 45 out of the 50 mazes were not solved by any of the models. We also observed that all the solves came on mazes with no switches, no distractor objects, and an optimal path length of 45 steps or lesser.

By grid size, we see that models solved 3 out of 33 8x8 mazes, 3 out of 57 10x10 mazes, and none of the 60 14x14 mazes.

Figure 5 shows that none of the 3 models came close to the goal tile without actually solving the maze. The median closest approach is 47.5 executable actions from the goal tile, and the only episodes arriving within the 3 actions of the goal tile are the six solves.

Difficulty Axes

Since we chose and designed a maze as our evaluation environment, several components determine how difficult it is for a model to succeed. How long the optimal path is, the grid size, how many mechanisms need to be cleared and in what order, and how many distractors are present, etc. We tested the various components that we thought made a given maze difficult and computed its influence on the action progress score of the models we evaluated.

We found that optimal path length is the largest influence on the difficulty of a maze when compared to the other parameters that make up the environment. When tested one property at a time against the progress, with the model that ran the episode held constant, path length shows an R² value of 0.171 - more than gates that showed 0.122, switches that showed 0.104, and distractor count that showed 0.030.

Each property is tested with one regression that predicts an episode's action progress from that property and from which model ran it. R² in simple terms is the share of the variation in progress that this regression accounts for, 1 − SS_res / SS_tot, where SS_res is the total squared gap between actual and predicted progress and SS_tot is the total squared gap between actual and average progress. An R² of 0.171 means the fit removes about 17% of the spread.

We see in Figure 6 that performance falls as the optimal path length required to solve the mazes increases, which agrees with the regression calculation above. When grouped by shortest path, mazes of 30 moves or fewer gave 5 solves in 39 episodes at a mean action progress of 0.287 (95% CI 0.163 to 0.423), 31 to 45 moves gave 1 in 30 at 0.240 (0.164 to 0.333), 46 to 60 gave 0 in 15 at 0.141 (0.089 to 0.205), and 61 or more gave 0 in 66 at 0.070 (0.058 to 0.084).

When restricted to the 69 episodes on the mazes with optimal solution path lengths of 45 moves or fewer, the influence of gates rise from an R² value of 0.122 to 0.255 and switches from 0.104 to 0.247. Meanwhile the influence of path length on this restricted subset falls from 0.171 to 0.012, which shows that when the range of optimal path length is restricted, the mechanisms have a relatively significant impact on the difficulty of the maze.

Failure Modes

Our evaluation runs resulted in 0 solves across the 105 episodes containing a switch-maze mechanism. Across all 3 models, despite 132 toggle actions being issued, only 2 of these actions resulted in flipping a switch. 15 of these toggles resulted in successful door opens, but the remaining 115 resulted in nothing.

We found it interesting that a model stood on a live switch in 38 out of the 105 mazes containing the switch-gate mechanisms, which means that the models had no issue getting to the tile with the switch, but they were not able to figure out how the mechanism worked in association with a gate.

On the contrary to how the models were confounded by the switch-gate mechanism, we observed that the models produced 15 successful door opens. Moreover, 4 of the 6 total solves came on the key-door mazes. One Kimi solve even required a key, a door, a second key, and a second door in sequence. What separates the key-door mechanism from the switch-gate mechanism is the fact that a frontier model is able to associate a key with a door purely based on visual perception due to its prior knowledge, whereas the functioning of a switch which is represented as a circle on a tile needs to be understood by exploring the environment and trying the various actions on a switch cell to observe how the environment changes.

Despite relative success with the key-door mechanism, the models interact with both mechanisms inefficiently. We observed that they issued 220 pickup actions which in turn produced only 27 successful key pickups. Similarly, with the toggle action, the models issued 132 of them, of which only 15 actually opened a door, and 2 flipped a switch. Models reach for interaction verbs far more than there is anything for them to interact with in the current state.

The switch-gate mechanism failure mode was pretty consistent across all the 3 models. However, we also observed that the models displayed a variety of other failure modes in their evaluations as well.

As seen in Figure 7 and Figure 8, Claude fails early in most of its episodes. Additionally, we saw in the output traces of our results that in 91% of Claude’s last 30 actions before it fails an episode, it pushes forward into a wall. In 41 of its 46 stalled episodes, wall collisions make up for more than half of the final 30 actions. As the steps in a given episode increase, the share of actions leading to wall collision increases, while successful moves fall.

In Kimi’s traces we observed that it retreads visited ground quite often. The revisits climb from 2.7% of its actions in the first quarter of steps in an episode to 21.2% in its last.

For Qwen, the percentage of moves that were turns climbs from 44.5% to a peak of 59.6% in the third quarter of its steps in an episode, while the share of steps reaching an unseen tile collapses from 24.5% to 2.4% indicating that it produces a lot of unfruitful turn actions as it progresses in the maze.

Test-time Compute

As we can see in Figure 9, Kimi K2.6 and Qwen 3.6 27B reason significantly more and utilize more than 20x the amount of output tokens as Claude to make more progress in the maze. However, as we saw in the Results section, Kimi and Qwen only have 1 solve each compared to Claude’s 4. This is an indication that more reasoning and tokens spent may increase the proportion of the maze explored by Kimi and Qwen, but does not help with the baseline success rate. Despite all 3 models being given the same output token budget of 64k, we see a clear disparity in the amount of tokens spent across the models.

Another observation we made as we looked into the model traces was that Claude’s thinking contracts as it makes progress in an episode. Interestingly, the thinking contracts further when it produces a move that achieves no progress in the maze. Median thinking tokens after a move that achieved nothing is 234 across 1301 turns, whereas it is 579 across 299 turns following a move that worked.

Discussion

Difficulty of 2D mazes:

The results and analysis from our evaluation in this work showcase that 2D mazes with mechanisms that require causal reasoning to solve, are a very challenging environment for highly capable VLMs. By increasing just one parameter of the environment - optimal path length to solve a maze, the task becomes nearly impossible for frontier models to solve. The difficulty of this underlying substrate will only increase as the mechanisms and distractors are made more challenging - we see this when we restrict the mazes to a certain path length and compute the influence of mechanisms such as switches and gates. Since we believe that this evaluation substrate is a a highly simplified representation of real-world workflows that requires models to take actions over several steps, while encountering novel scenarios, and learning how a new interface must be operated, we think that there is a lot of room for improvement in performance of models and agents engaging in long-horizon workflows.

Inability to learn from exploration:

We also see that frontier VLMs lack the ability to explore, understand, and learn from the environment: As we see in the case of the switch-gate mechanism, models are not able to causally reason in order to associate a toggle action on a cell containing a switch with the gate opening and unlocking more of the maze. This is an important finding as models and agents utilized in real-world workflows constantly come up against interfaces, objects, and scenarios that they have never seen before. Being able to figure out how something needs to be operated without any prior knowledge is an important trait to improve the reliability and capability frontier of a model.

Protection from contamination and saturation:

Due to the nature of the task and environment, we were able to isolate parameters of the environment to understand the effects of each one of them on the difficulty of the maze in the form of structured ablations. The ability to turn any of these knobs to increase difficulty ensures that our benchmark stays free of saturation, and due to the fact that each maze is created as a combination of hand-design and LLM generation - it is entirely new data, thus protecting our benchmark from contamination-related issues. With MultiNet v2.0, we aim to take a step towards solving these 2 important issues that plague most benchmarks that exist today.

Limitations

Inability of the benchmark to rank the frontier models: Since we only see 6 out of 150 solves, with the ranking inverting when we choose action progress rate as the main metric, the evaluation results on this version of the benchmark are not sufficient to differentiate between the capabilities of the frontier models

The discovery finding relies on a single mechanism: Currently, we conclude that the inability of the models to understand and operate switch-gate mechanisms, while they are significantly better at solving key-door mechanisms, indicates their inability to successfully operate a mechanism that does not rely on prior knowledge. However, this claim could be made significantly stronger with more mechanisms that do not rely on prior knowledge.

Visual history limited to 3 previous steps: In order to implement a minimal harness, understand the true capability of the model in isolation, and maintain a reasonable budget for input token related costs, we restricted the visual history provided to the model in a given timestep to 3 previous frames. In this study, we do not evaluate how the performance of a model changes if the visual history window is increased.

No human baseline: We did not run an official human baseline to measure the difference between human performance and model performance and truly understand whether the task is easy for humans of various backgrounds and age-groups.

No information provided to the model on steps remaining: We did not run an ablation to observe whether models tend to perform better when they are told how many steps they have available at a given point, thus providing them with a budget that could enable more cautious movement.

Conclusion and Next Steps

In this release, we build a 2D maze with obstacles as an environment to benchmark the long-horizon action taking and causal reasoning capabilities of frontier vision-language models. We design the environment in a structured manner by running ablation experiments to thoroughly understand the influence of each parameter that makes up the environment.

While models performed really poorly on the final evaluation set, we learned a lot about the different failure modes of each of these models. We learned how frontier models find it hard to figure out how to operate a mechanism that they are not familiar with, and also observed how each model had its own way of failing this benchmark. While Claude constantly jammed into walls, Kimi kept visiting previously seen portions of the maze, and Qwen wandered aimlessly through the maze barely getting close to the goal tile. The results also showed how extended reasoning and a large output token budget does not help improve the performance of these models in tasks such as this one.

The motivation behind this benchmark was to build an environment representative of real-world workflows where models and agents have to navigate novel interfaces and scenarios to achieve goals that require several steps aided by reasoning and error recovery. While this version of the benchmark already gives us a good idea of the capability frontier, we aim to assess a more realistic version of workflows in the coming versions of the benchmark - those involving multiple domains. Due to the projectable nature of the underlying substrate in this task, we can build an identical environment and task in various domains such as 3D simulation and pure language. Evaluating a frontier model across all projected domains, gives us a quantified understanding of how well models generalize to new interfaces, modes of perception, and action spaces, which is highly essential to solve and automate high-value trajectories in the real world.

Work with us

At Fig, we are building the control layer for AI: systems that perceive an environment, act in it reliably, and improve from the experience. Perception is largely solved. Agency is the open problem: acting dependably in environments nobody explained, recovering from mistakes, and compounding progress over many steps. As this report shows, it will not come from scaling reasoning budget.

Measuring progress towards this goal requires evaluation setups and benchmarks representative of complex, real-world workflows. This is what we aim to do with MultiNet, in collaboration with our friends at Manifold Research, MIT, Georgia Tech, and Tufts.

If you build models or agents, or work on benchmarking and evaluation, we want to hear from you - whether that means getting your model on our MultiNet v2.0 benchmark, contributing to the environments we are building, or working with us on what comes after. Please fill out this form to get involved, and we will follow up.

Citation

Please cite this work as:

Guruprasad, P., Rivera, S., Lu, H., Jain, A., Ren, H. and Sikka, H. (2026) MultiNet 2.0 Preview: Goal Progress Decays with Task Horizon for Frontier VLMs in Interactive 2D Environments. Available at: https://www.fig.inc/multinet-v2-preview

Or use the BibTex citation

@online{multinet_v2_preview_technical_report_2026,
  title   = {MultiNet 2.0 Preview: Goal Progress Decays with Task Horizon
             for Frontier VLMs in Interactive 2D Environments},
  author  = {Pranav Guruprasad and Sean Rivera and Helen Lu and Arushi Jain and Hangliang Ren and Harshvardhan Sikka},
  year    = {2026},
  url     = {www.fig.inc/multinet-v2-preview},
  note    = {MultiNet 2.0 Preview: Interactive 2D Mazes}
}

Acknowledgements

We thank Victor Barres for feedback on the design of this environment and benchmark, and for his insights on the evaluation space more broadly. We also thank Greg Kamradt and Yuansheng Ni for early comments on the direction and framing of this work.

References

1. Y. Zhu et al., "Establishing Best Practices for Building Rigorous Agentic Benchmarks," Jul. 03, 2025, arXiv. doi: 10.48550/arXiv.2507.02825.
2. P. Anderson et al., "On Evaluation of Embodied Navigation Agents," Jul. 18, 2018, arXiv. doi: 10.48550/arXiv.1807.06757.
3. M. Chevalier-Boisvert et al., "Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks," Jun. 24, 2023, arXiv. doi: 10.48550/arXiv.2306.13831.
4. F. Chollet, "On the Measure of Intelligence," Nov. 05, 2019, arXiv. doi: 10.48550/arXiv.1911.01547.