RealCUA: Computer-Use Training Data from Real, Open-Ended Work

Earlier this month, we shared our thoughts on the gap between today’s computer-use training data and the learning signals needed for agents to tackle the complexity of real, open-ended work. Part of that complexity is how people interleave work across applications, moving between drafts, conversations, and browser tabs throughout a session. We examined this movement by capturing real work performed by people on their own computers. Across more than 27,000 observed app transitions, we found that 93% were returns to apps used earlier in the same recording.

Today, we're sharing RealCUA v0.1, the first computer-use dataset of open-ended work in evolving environments.

Independent interwoven workflows spanning multiple apps, contexts, and thousands of steps. Each unscripted trajectory preserves the actions taken alongside the surrounding desktop context as operators progress several goals.

RealCUA v0.1 spans over 600 hours, 274k actions, 27k app transitions, and 35 applications.

What does real computer work look like?

Real, open-ended work does not follow a prescribed sequence of assigned tasks. We make progress toward goals by deciding what needs attention, moving work forward, and adapting as new context emerges. Continuous observation captures what happens before, after, and between actions, as well as the workspace, which is continually reshaped by earlier activity and other work in progress.

Our learnings below cover the patterns visible in RealCUA v0.1 and the limits of what these measurements can tell us today.

Work happens in long sessions

On average, a recording contains about 537 non-idle steps. The upper end runs much longer, with the 90th and 99th percentiles at 1,432 and 4,455 steps, respectively. Across the dataset, 87% of hours are attributable to recordings ≥ 60 minutes long, with the longest individual recording lasting over 12 hours.

In RealCUA, step boundaries are tied to the length of each recorded session, unlike scripted task demonstrations, which are bounded by the start and end of the defined task.

While longer recordings preserve more of the surrounding context, length alone does not establish which signals will improve an agent. Structuring that context for training and evaluation remains part of the work.

Tasks inherit complex workspaces

Each task begins, resumes, and ends in a desktop shaped by every earlier action and work in progress. When a new task begins, partially finished documents, open windows, and countless browser tabs crowd the workspace (in multi-monitor recordings, that crowding is extended across an even larger surface area). While some context may be relevant to the current objective, others belong to separate workflows or contain incorrect or outdated information.

Continuous recording across tasks enables the preservation of the evolving desktop state in parallel to the actions that changed it. Each task or workflow segment begins from a different point in the session with a unique environment state and the preceding activities provide provenance behind state changes.

These complexities provide ample headroom for us to study how relevant information is identified, context is carried between tasks, and work in progress is resumed. How these patterns inform the context provided to a model remains part of future work.

Interleaved apps are revisited, not abandoned

Each hour of recording includes over 45 app transitions, on average, with the 90th percentile of recordings reaching 139 transitions per hour. Across the 27,700 observed app transitions in the corpus, 93.3% represent returns to apps used previously in the same recording.

Application transitions occur both within a workflow and as operators move between independent goals. The completion of a task may require the use of several applications, and while loading or processing, we often shift attention away to progress elsewhere before returning. Notifications and new information also influence urgency around what needs attention, prompting shifts away from active workflows.

Preserving application transitions alongside changes in desktop state and the surrounding actions allows us to examine how work unfolds across multiple open-ended goals. These sequences provide the foundation for creating learning signals around how tasks span applications, how other workflows are progressed during delays, and how events change what gets prioritized.

Intervals between actions

We label intervals without inputs as idle states throughout the corpus. Idle states account for 79,785 or 22.5% of RealCUA’s 354,769 total steps. The median idle state is 1.1 seconds and 139 seconds at the 99th percentile.

Idle states can arise from both endogenous factors, such as the operator reading, thinking, or stepping away from the computer, and exogenous factors, such as system latency, loading, or processing delays.

Continuous video recording ensures that during these periods without input, the entire environment remains visible. That context helps us examine the underlying category of each idle state and create learning signals that inform whether a page was loading, information was being read, or attention had moved elsewhere.

Open-ended work vs. scripted task datasets

What each trajectory contains

Our capture software is purpose-built for continuous recording of real work across workflows spanning thousands of steps. RealCUA preserves the visible environment alongside the inputs and state changes needed to examine an action in context.

You can explore trajectories from the dataset, including an 11-hour continuous trajectory and a 2-hour multi-monitor sample through our data viewer. The viewer includes video, raw input events, reduced step-level actions, and the available task breakdown. Idle states are excluded from the step-level action sequences across both samples.

Each trajectory includes:

• Video: Continuous H.264 full-desktop video, recorded at 30 frames per second, with support for multiple displays. 26 unique video resolutions exist in the dataset.
• Inputs and actions: Timestamped raw mouse and keyboard events synchronized to video with discrete JSON step reduction into actions and idle states for state → action → next-state inspection.
• Window and interface state: Foreground application and window metadata paired with accessibility trees, HTML and DOM snapshots where available to support the visible interface.
• Task and workflow segments: Derived segmented workflows and tasks from the continuous recordings.

You can view the full RealCUA v0.1 dataset card here.

How teams can use RealCUA

We're providing teams with access to the growing RealCUA dataset, so they can apply it to advance their own work.

The corpus can be used across:

1. Train: Pair recorded actions with the preceding desktop states and interaction history to train models to choose their next action in context.
2. Evaluate: Use recorded workflows to design tasks that test whether agents can recover context after interruptions and successfully complete long-horizon tasks.
3. Study: Examine natural actions, application switches, and pauses together to understand how people move between goals and return to unfinished work.

What comes next

Continued progress depends on providing each new generation of models harder and more realistic tasks to learn from. The length of software tasks frontier agents can complete at 50% success has doubled roughly every seven months (METR, 2025). Scaling those tasks is bottlenecked by our ability to write them. Today, someone defines the objective, stages the environment, and writes the verifiers that determine completion by hand. As complexity grows, fewer people are able to write and check them, driving up cost and creation time relative to the capability frontier being pushed.

Real work already contains the components of those tasks. The 11-hour trajectory we shared spans 64 tasks and 3 workflows, each with a distinct start state and completion path. Once the starting state can be reconstructed and the outcome can be checked, those inputs can be mapped to a training environment.

Difficulty can then be set directly by varying and chaining verified tasks together until sufficiently difficult for frontier models to hillclimb. The open-ended nature of real work also enables continued scaling of difficulty along those axis as workflows grow in length from minutes to hours to days to weeks. We are working towards each hour of work producing the inputs to build a verified environment:

• Scaling collection: We can already generate thousands of hours a month and are scaling infrastructure and tooling towards tens of thousands.
• Task and workflow hierarchy: Mapping tasks to hierarchical workflows that carry context of the work surrounding them.
• Failures: Identifying corrections, rejected outputs, and interruptions signaling where agents fail on work that no program can check today.
• Recordings to environments: Combining verified pieces into environments with capabilities that sit just past the frontier.

Raising that yield is the work ahead of us.

@article{fig_realcua_600_hours_of_real_open_ended_work_2026,
  title       = {RealCUA: 600 Hours of Real, Open-Ended Computer Work},
  author      = {Joseph Hakim and {Fig AI Team}},
  year        = {2026},
  institution = {Fig},
  url         = {https://www.fig.inc/blog/realcua-600-hours-of-real-open-ended-computer-work/}
}