Astra has successfully completed one task.
ROSWORLD: Robotic Computer Use for OSWorld Tasks
Release date: October 12, 2026 (AoE)
Caoyuan Ma1,2,* · Wendi Liu1,*
Wenqi Shao2 · Yinqiang Zheng1
Wenqi Shao2 · Yinqiang Zheng1
1 UTokyo 2 JD Explore
* Equal contribution (co-first authors).
A living benchmark · Embodied computer use
Digital tasks.
Physical actions.
A long-term, evolving benchmark for embodied computer use. Simulated robots operate real keyboard and mouse signals to complete OSWorld tasks. As models advance, we continuously introduce new tasks and update the leaderboard.
OPEN SOURCE PLANNED · BY END OF OCT 2026
PROJECT UPDATES
Latest progress
Astra has successfully pressed multiple physical keys and is making progress toward the goal. Fable has not yet successfully touched a physical key.
ROSWORLD / ENVIRONMENT DEMOISAAC SIM × VIRTUAL MACHINE
01 / A simulated embodiment. A real computer interaction loop.PROJECT DEMONSTRATION
Preliminary demo. This video presents an early-stage demonstration and does not represent the final evaluation protocol. The viewpoint, robotic arm, and other assets may change before release. Evaluations will follow the protocols published with the released results.
Real input signalsKeyboard & mouse → virtual machine
Visual feedbackVM display → simulated monitor
Evolving evaluationNew models, tasks, and rankings
01 / THE APPROACH
Simulation meets
real interaction.
ROSWORLD combines simulated embodiment with a real virtual-machine interface. Models act from a given viewpoint, controlling a robotic arm or dexterous hand instead of directly issuing computer commands.
01 →
Observe
The model receives the given view of the Isaac Sim environment, including the monitor.
02 →
Act
A robotic arm or dexterous hand physically operates the simulated keyboard and mouse.
03 →
Execute
Triggered keyboard and mouse signals are sent to the virtual machine running OSWorld tasks.
04 ↩
See the result
The live VM display is rendered onto the simulated monitor, closing the perception–action loop.
DISPLAY FEEDBACK → NEXT OBSERVATION ↩
Consistent benchmark protocols
Comparative evaluations will use fixed protocols, including fixed robotic arm, dexterous hand, and humanoid robot models. Results will be presented with their corresponding evaluation setup.
Diverse assets for open research
The planned open-source repository will include diverse assets so researchers can evaluate their models across different embodiments.
02 / THE LIVING BENCHMARK
Built to evolve
with the models.
This is a long-term project. As model capabilities develop, we will introduce new tasks and keep the leaderboard updated. Results from the initial evaluation are expected before October 20, 2026.
Leaderboard · continuous updatesRESULTS FORTHCOMING
| MODEL | EVALUATION TRACK | STATUS |
|---|---|---|
| gpt-6-astra | Initial evaluation | Under evaluation |
| gpt-6.1-sol | Initial evaluation | Pending evaluation |
| claude-fable-5 | Initial evaluation | Under evaluation |
| claude-opus-5 | Initial evaluation | Pending evaluation |
| deepseek-vision-exp | Initial evaluation | Pending evaluation |
The leaderboard will be updated as new models and tasks are evaluated. No scores have been published yet; the listed entries describe the initial evaluation scope. Released results will include specific model versions and evaluation protocols.
03 / WHAT'S NEXT
An open, ongoing effort.
BEFORE OCT 20, 2026 · EXPECTED
Initial model evaluations
The first round will evaluate gpt-6-astra, gpt-6.1-sol, claude-fable-5, claude-opus-5, and deepseek-vision-exp. Results are expected before October 20, 2026.
BY END OF OCT 2026 · PLANNED
Open source & arXiv report
Release the project repository with diverse assets for evaluation under different embodiments. The accompanying technical report is planned for release on arXiv before the end of October 2026. The official code repository is currently private and will become public by the planned release date. The report link will be added when released.
ONGOING
New tasks. Updated rankings.
Continuously expand the task set as models advance, evaluate new models, and refresh the leaderboard with the latest results.
JOIN THE EVALUATION
Your model.
Our next evaluation.
Have a model you would like evaluated in ROSWORLD? Email us to discuss API access and evaluation arrangements.
Contact the team ↗woshimacaoyuan@gmail.comCITATION
Cite ROSWORLD
Provisional project citation. The project website is available at miles629.github.io/ROSWORLD/.
@misc{rosworld2026,
author = {Ma, Caoyuan and Liu, Wendi and Shao, Wenqi and Zheng, Yinqiang},
title = {{ROSWORLD}: Robotic Computer Use for {OSWorld} Tasks},
year = {2026},
url = {https://miles629.github.io/ROSWORLD/},
note = {Project website; open-source release planned for October 2026}
}