Understanding Open-ended Work from a Data Perspective

At Fig, we believe the next step for computer-use agents is the ability to reliably advance open-ended work: work without a set of instructions or tasks, or a rubric for when it's done. Previously, we described the bottleneck behind the data needed to get there. The capabilities on the other side of that bottleneck include picking up work where it was left off, advancing several goals at once, and carrying context across projects that run for hours and days, beyond completing a single task.

Over the past several months, we've been capturing the signals behind open-ended work through hundreds of hours of real people doing their own work on their own computers. Unlike most computer-use data, sessions are unscripted and recorded continuously, which has enabled our team to capture the tasks, context, and actions that surround them.

Ahead of sharing the data next week, here are two things we observed in an 11-hour session that make open-ended work hard to learn from:

• Work is resumed, not restarted: prototype tasks were picked back up after gaps of 40 minutes to four hours, each time on a changed desktop.
• Task boundaries go unmarked: tasks were a median 27 seconds apart, and 11% of task transitions overlapped.

Real work runs continuously

The bounded structure of scripted computer-use tasks makes them learnable. Each one begins from a staged desktop with a clear goal that's checked to confirm the result. Tasks are run in loops with the same starting state reproduced and the outcomes graded. Even OSWorld 2.0, which stages open files and messages and injects new ones mid-task, starts each task from a controlled workspace and scores the final state for one goal.

Real work is different. It is made up of sequential tasks within longer, continuous workflows, and interrupted work that resumes from the partially completed state it was left in. Both take place on evolving desktops, shaped by the work that came before them and changed by other work interleaved while they're in progress.

Across the hundreds of hours we’ve recorded and analyzed, we’ve observed similar patterns in how work is suspended and resumed, and how one task transitions into the next. The breakdown below is taken from a single continuous 11-hour session, segmented into 64 tasks, in which one person built a prototype, did background research, and wrote a blog post. Next week, we’re sharing the larger set of recordings it comes from.

Work is resumed, not restarted

Behind this window, there's a good chance your desktop is full of work in progress: drafts open, threads unread, and a few tabs being saved for later. When you close this post, you'll likely pick one of them back up without much thought.

Open-ended work takes place on desktops like yours, filled by earlier work and partially completed tasks. People sort through that context without much thought, deciding what needs to be prioritized, picked back up, and what can wait.¹ Agents working alongside us would see the same desktops we do, and our expectation is that they interpret them the same way. That means identifying task-relevant context among everything else that's open, inferring goal progress to determine where to resume, and maintaining that state awareness as the desktop changes from the other work moving forward in the same space.

In the first hour and a half of the session, work on the prototype in Figma Make was interleaved with 23 writing and research tasks, suspended and resumed around each one. Over the following nine hours, each time a prototype task was picked back up, it followed a gap of 40 minutes to four hours. Each gap changed the desktop. Five hours into the session, five windows were open, and at hour 11 there were nine. A model resuming the prototype at any of those points would have to continue from a different state each time.

Scripted tasks start with a staged state that's resettable and graded against a single goal. Each reset removes the evolving context captured in the session above: the suspended workflows, evolving desktop, and other concurrent work. Open-ended work requires an agent to interpret and track context as it changes over long horizons. Each resumption we capture pairs the desktop as it was found with the actions that picked the work back up.

Task boundaries go unmarked

Think back to the last hour of your work and try to map where each task began and ended. A few boundaries are easy to draw. Most aren't. A reply to a message leads to a search for a number, which ends up in a slide and then in an update to a colleague. Each task flows into the next, and you decided where one stopped and the next began in the moment, often without noticing.

Each unmarked boundary holds a signal that agents have to learn from: whether a task is sufficiently complete and what state it was left off in, in the event it needs to be picked back up. Because real work is continuous, the ground truth for both is never labeled. Instead, we infer both from the work that surrounds each transition and record the result as task segments alongside the actions. The boundaries below are that segmentation.

Over the 11 hours, the median time between tasks was 27 seconds. Half of the tasks overlapped a neighboring task or came within 10 seconds of one, and 11% of task transitions overlapped, with the next task starting before the previous one ended. App switches rarely coincided with the start of a new task. Most task transitions happened within the same app, while most tasks spanned two or more apps.²

Scripted tasks predefine the boundaries for an agent. The instruction defines where a task starts, and task completion or a step limit signals the end. Agents working alongside us won't have predefined boundaries for the work we hand them. They have to recognize when their work is sufficiently complete, which goal it advances, and how far it has moved the larger workflow forward. Learning those capabilities depends on signal from the unmarked boundaries in open-ended work.

Work that can't be reset

As models take on longer and more open-ended work, we expect them to operate reliably in the same shared environments we do. Those environments are constantly evolving and can't be paused or reset. While tasks are in progress, messages arrive, notifications flash, meetings start, and other work moves forward on the same desktop. Learning signals for resuming work and recognizing task boundaries are largely missing from computer-use data today, which is the bottleneck we described in our post last month. These signals come from real work: open-ended, multi-objective sessions on desktops that are never reset. Which of these signals helps a model most is the question we're working on now.

What comes next

At Fig, we've been developing entirely new ways to turn complex, open ended real work into high signal data and environments models can train on.

Next week, we're sharing a SoTA dataset that contains signals extracted from real world, open ended work.

@article{fig_understanding_open_ended_work_from_a_data_perspective_2026,
  title       = {Understanding Open-ended Workd from a Data Perspective},
  author      = {Joseph Hakim and {Fig AI Team}},
  year        = {2026},
  institution = {Fig},
  url         = {https://www.fig.inc/blog/understanding-open-ended-work-from-a-data-perspective/}
}