01 →
Observe
The model receives the given view of the Isaac Sim environment, including the monitor.
ROSWORLD: Robotic Computer Use for OSWorld Tasks
Release date: November 12, 2026 (AoE)
Caoyuan Ma1,2,* · Wendi Liu1,*
Wenqi Shao2 · Yinqiang Zheng1
Wenqi Shao2 · Yinqiang Zheng1
1 UTokyo 2 JD Explore
* Equal contribution (co-first authors).
A living benchmark · Embodied computer use
Digital tasks.
Physical actions.
A long-term, evolving benchmark for embodied computer use. Simulated robots operate real keyboard and mouse signals to complete OSWorld tasks. As models advance, we continuously introduce new tasks and update the leaderboard.
OPEN SOURCE PLANNED · BY END OF NOV 2026
ROSWORLD / ENVIRONMENT DEMOISAAC SIM × VIRTUAL MACHINE
01 / A simulated embodiment. A real computer interaction loop.PROJECT DEMONSTRATION
Preliminary demo. This video presents an early-stage demonstration and does not represent the final evaluation protocol. The viewpoint, robotic arm, and other assets may change before release. Evaluations will follow the protocols published with the released results.
Real input signalsKeyboard & mouse → virtual machine
Visual feedbackVM display → simulated monitor
Evolving evaluationNew models, tasks, and rankings
01 / THE APPROACH
Simulation meets
real interaction.
ROSWORLD combines simulated embodiment with a real virtual-machine interface. Models act from a given viewpoint, controlling a robotic arm or dexterous hand instead of directly issuing computer commands.
02 →
Act
A robotic arm or dexterous hand physically operates the simulated keyboard and mouse.
03 →
Execute
Triggered keyboard and mouse signals are sent to the virtual machine running OSWorld tasks.
04 ↩
See the result
The live VM display is rendered onto the simulated monitor, closing the perception–action loop.
DISPLAY FEEDBACK → NEXT OBSERVATION ↩
Consistent benchmark protocols
Comparative evaluations will use fixed protocols, including fixed robotic arm, dexterous hand, and humanoid robot models. Results will be presented with their corresponding evaluation setup.
Diverse assets for open research
The planned open-source repository will include diverse assets so researchers can evaluate their models across different embodiments.
02 / THE LIVING BENCHMARK
Built to evolve
with the models.
This is a long-term project. As model capabilities develop, we will introduce new tasks and keep the leaderboard updated. The table shows progress on the current evaluation.
40-task evaluation · in progressNOV 3, 2026 · 10:12 JST
| MODEL | SCORED / PLANNED | FULL-SCORE TASKS | OTHER SCORES |
|---|---|---|---|
| GPT-6 Astra | 16 / 40 | 9 | 6 × 0; 1 × 0.90649 |
| GPT-6.1 Sol | 16 / 40 | 8 | 7 × 0; 1 × 0.90649 |
| GLM-5.3-Flash | 1 / 40 | 1 | — |
| Kimi-K3-for-GameDev | 10 / 40 | 0 | 10 × 0 |
Snapshot from the ongoing 40-task evaluation, with up to 100 model decision rounds per task. Finished reports with numeric scores are shown; unfinished and infrastructure-error runs are excluded. Kimi includes two zero-score runs marked invalid in their original reports, classified as failures by the progress dashboard because the model ended normally. Partial scores are shown separately.
03 / WHAT'S NEXT
An open, ongoing effort.
BEFORE NOV 20, 2026 · EXPECTED
Initial model evaluations
The first round will evaluate gpt-6-astra, gpt-6.1-sol, claude-fable-5, claude-opus-5, and deepseek-vision-exp. Results are expected before November 20, 2026.
BY END OF NOV 2026 · PLANNED
Open source & arXiv report
Release the project repository with diverse assets for evaluation under different embodiments. The accompanying technical report is planned for release on arXiv before the end of November 2026. The official code repository is currently private and will become public by the planned release date. The report link will be added when released.
ONGOING
New tasks. Updated rankings.
Continuously expand the task set as models advance, evaluate new models, and refresh the leaderboard with the latest results.
JOIN THE EVALUATION
Your model.
Our next evaluation.
Have a model you would like evaluated in ROSWORLD? Email us to discuss API access and evaluation arrangements.
Contact the team ↗woshimacaoyuan@gmail.comCITATION
Cite ROSWORLD
Provisional project citation. The project website is available at miles629.github.io/ROSWORLD/.
@misc{rosworld2026,
author = {Ma, Caoyuan and Liu, Wendi and Shao, Wenqi and Zheng, Yinqiang},
title = {{ROSWORLD}: Robotic Computer Use for {OSWorld} Tasks},
year = {2026},
url = {https://miles629.github.io/ROSWORLD/},
note = {Project website; open-source release planned for November 2026}
}