cu-bench: Professional Desktop Work from Screenshots, Mouse and Keyboard

Abstract

Can an agent do real professional work in Linux desktop applications from screenshots and mouse and keyboard alone? cu-bench asks agents to route circuit boards in KiCad, model parts in FreeCAD, edit scenes in Godot, slice prints in PrusaSlicer and work in 16 more applications, with real data and specs delivered as documents. File-state verifiers grade every deliverable, and each task has a pure-GUI track and a track that allows the application's own scripting. On the frozen release of 2026-10-16, Claude Opus 5.5 reaches 83.5% strict success on 158 paired task × track runs and gpt-6-astra 22.2%.

What the study covers

• The test split holds 290 variants of 165 tasks across 20 applications in three difficulty tiers (19 T3, 86 T4 and 60 T5 tasks); each task is tagged with its primary difficulty source, from perception and precision to long-horizon work and diagnosis.
• A run passes only when the verifier passes every required check, the transcript audit finds no rule violation and the agent stays within the task's step budget; an auxiliary composite and blind model judge never change a strict result.
• Curated replays below show what the agents actually did, frame by frame, alongside the full per-task matrix for both models and tracks.

The benchmark

Paired runs

158 task × track pairs with 72 gui and 86 code runs

Test split

290 variants of 165 tasks across 20 apps

Tiers

The split contains 19 T3 tasks alongside 86 T4 and 60 T5 tasks.

Budget

max_steps = ceil10(3 × est. GUI actions)

See the ranking on the leaderboard →

Strict success by difficulty dimension

How it is scored

• Strict success is the headline: a run passes only when the rule verifier passes every required rubric check, the transcript audit finds no rule violation (gui track: no code typed into the application) and the agent stays within the task's step budget. Otherwise the reward is 0.
• Both models ran the same 158 task × track pairs from fresh desktops, one attempt each (pass@1), from the frozen release of 2026-10-16. gpt-6-astra results are the raw protocol, without the one-time "keep going" nudge (it converted 0 of 14 runs).
• The composite (0–100) is an auxiliary training signal weighted over correctness, robustness, collateral damage, efficiency and process (plus code quality on the code track); it never changes the strict reward.
• Judge scores come from an auxiliary blind model judge (gpt-6-sol, prompt v2) over all valid test runs that were judged: visual quality of exported deliverables on 71 Claude and 45 gpt-6-astra runs, reasoning on 50 and 23. A two-judge, both-agree audit of passing runs found 0 justified extra failures, so the judge never changes a strict result.

Environment

Each run gets a fresh Fly.io machine (Ubuntu 24.04, Xvfb, openbox, Mesa llvmpipe, no GPU) with the application already open; the agent acts only through the cu screenshot, click, drag, type, key, scroll and zoom commands. Hidden tests are uploaded only for verification. On EnvPlatform the benchmark runs as the official envloop/cu-bench Runner over private datasets (envloop/cu-desktop-apps-tracks-test, 288 members in 0.2.0).