UI-Mate-democua

UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations

Show the workflow once. Let the agent adapt it to the task at hand.

Tencent HY Frontier

Project Page · GitHub · arXiv

Overview

UI-Mate-democua-27B is the demonstration-guided checkpoint of UI-Mate. It observes live screenshots, reasons over the visible state, and produces structured keyboard and mouse actions, and it can additionally take one recorded workflow as input and carry that procedure over to a new task.

It starts from the general reinforcement-learning checkpoint and is further trained with supervised fine-tuning on a mixture of general computer-use data and demonstration-augmented data. Keeping general data in the mixture is what preserves instruction-only competence: the model still runs a task from an instruction alone, and gains the ability to use a demonstration when one is supplied.

A demonstration is treated as guidance rather than a fixed action script. Recorded coordinates are never replayed, and the model re-plans from the live interface whenever the content, layout, or application state differs.

Model Details

• Parameters: 27B
• Base model: Qwen3.6-27B
• Input: task instruction, screenshots, interaction history, and an optional demonstration workflow
• Output: reasoning, a concise action description, and structured computer-use tool calls
• Action space: mouse, keyboard, scrolling, waiting, user interaction, subtask completion, and task completion
• Training: supervised fine-tuning, online reinforcement learning in executable GUI environments, then supervised fine-tuning on mixed general and demonstration-augmented computer-use data
• License: Apache-2.0

UI-Mate is an agent checkpoint rather than a standalone visual-chat model. We recommend using the official prompt, response parser, and interaction harness from the UI-Mate repository.

Highlights

• One-shot procedural learning from a single demonstration.
• Instruction-only execution retained through mixed general and demonstration training data.
• Subtask-level guidance rather than whole-workflow injection.
• Live-screen grounding instead of coordinate replay.
• Trained on incomplete workflows, so milestones must be connected by reading the screen.
• Structured actions compatible with pyautogui, served through an OpenAI-compatible interface.

Demonstration-Guided Execution

A demonstration is a recorded successful execution: every keyboard and pointer action, with screenshots immediately before and after each one. It may be recorded by a human, or taken from a successful rollout of a stronger GUI agent. The raw trace is then:

1. normalized into a consistent action-and-frame representation;
2. annotated by a vision-language model along four axes — screen state, intent, the action taken, and how its target was located visually;
3. segmented into named subtasks, each with a short goal and an explicitly checkable completion criterion; and
4. supplied at inference time as a compact view of the active subtask.

Three blocks are placed before the instruction: listing all subtasks as done, current, or upcoming; carrying the sub-instruction and its completion criterion; and `` holding the ordered steps of that subtask only. The tool schema gains a subtask_complete action, and the model reports it to advance the pointer.

Training deliberately withholds part of this guidance. The full trajectory remains the supervision target, but the workflow shown to the model keeps only key actions — focus clicks, scrolling, and popup dismissal are omitted — so the model cannot reduce the task to copying the next line and must infer the missing steps from the screenshot. The training mixture also covers three guidance-to-screen relationships: full alignment, partial misalignment where the model corrects mismatched steps from the screenshot, and irrelevance where it ignores the workflow entirely. Full alignment stays the majority case so that guidance remains useful.

At inference the complete action sequence of the current subtask is passed instead, with no key-action extraction. Once the model treats the demonstration as a fallible reference and arbitrates in favour of the screenshot, fuller guidance is simply more informative, and dropping the extraction step removes one model call and a source of silent corruption.

Evaluation

Paired evaluation in the self-demo setting, where each target is paired with a successful rollout of that same task from a stronger agent. Task instructions, initial environment states, interaction budgets, and evaluators are identical across the two conditions; only the demonstration differs. Episodes are allowed up to 1,000 interaction steps.

Results are averaged over three runs per target on OSWorkerBench-Subset and five runs elsewhere.

OSWorkerBench-Subset (33 tasks) contains multi-application office tasks requiring coordinated interaction across three to five applications, with repeated subtask patterns and branching execution paths. Demonstrations improve 28 of the 33 tasks, and the number