🐶 Labomaru’s Quick Take & Specs
“MirroS has bridged real-world video understanding and physical simulation by turning raw video frames directly into executable MuJoCo physics code (
scene.json) through an agentic self-verification loop! 🐶⚡”
- 🚀 Tool Type: Open-Source Vision-Language Physics Agent / Local vLLM Pipeline
- 💰 Cost & Pricing: 100% Free Open-Source (Apache 2.0 License)
- 💻 System Requirements: CUDA GPU (16GB+ VRAM recommended for 9B model), Python 3.10/3.11, vLLM
- 🎯 Best For: AI Robotics Researchers, Physics Simulation Engineers, Synthetic Dataset Creators
- ✨ Key Benefit: Automates 3D physics scene generation from standard videos with zero manual 3D modeling!
1. Key Takeaways & Real-World Impact (Before vs. After)
Traditional workflows for reconstructing physical 3D simulations from real-world video clips require arduous manual effort. Engineers had to manually measure geometry, estimate mass, friction, and joint limits, and write custom simulation scripts for engines like MuJoCo or PyBullet. Generative video-to-3D approaches often yield non-executable point clouds or mesh representations that lack underlying physical constraints.
Code-as-World by MirroS changes this paradigm entirely. By fine-tuning Qwen3.5-4B and Qwen3.5-9B into specialized vision-language models (Code-as-World-VL), the model generates structured, executable MuJoCo scene definitions (scene.json). Coupled with an agentic feedback loop of up to 5 self-correction rounds, Code-as-World validates rendered simulation outputs against source input frames, correcting physical parameters iteratively.
| Workflow Dimension | Traditional Manual Reconstruction | Vision-to-3D Mesh Models | Code-as-World (MirroS Agentic Loop) |
|---|---|---|---|
| Output Format | Manual XML/JSON Code | 3D Mesh / Point Cloud | Executable MuJoCo JSON Code |
| Physics Fidelity | High (Human Tuning Required) | None (Visual Only) | High (Agentic Closed-Loop Tuning) |
| Setup Time | Hours to Days per Scene | Minutes (Non-interactive) | Automated in Minutes |
| Editability | High (Scripted Code) | Low (Static Asset) | Complete (Parametric Code) |
| License / Cost | High Labor Cost | Proprietary API Costs | 100% Free Open Source (Apache 2.0) |
2. Quickstart Setup & Code Snippets
Setting up Code-as-World requires a CUDA environment running Python 3.10 or 3.11. The model can be served locally via vLLM with OpenAI-compatible endpoints.
Repository & Environment Preparation
git clone https://github.com/MirroS-Lab/Code-as-World.git
cd Code-as-World
python -m venv .venv
source .venv/bin/activate
pip install -r requirements/inference.txt
Downloading Model Weights & Serving via vLLM
# Download 4B or 9B weights from Hugging Face
hf download MirroS-Lab/Code-as-World-VL-4B --local-dir weights/4b
hf download MirroS-Lab/Code-as-World-VL-9B --local-dir weights/9b
# Serve the 4B model using vLLM on GPU 0
CUDA_VISIBLE_DEVICES=0 vllm serve weights/4b \
--served-model-name code-as-world-4b \
--max-model-len 4608
Executing the Agentic Loop Inference Script
import requests
import json
# Example payload calling the served vLLM local endpoint
payload = {
"model": "code-as-world-4b",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Reverse-engineer this video into an executable MuJoCo scene.json code:"},
{"type": "image_url", "image_url": {"url": "file:///path/to/extracted_frame.jpg"}}
]
}
],
"max_tokens": 2048
}
response = requests.post("http://localhost:8000/v1/chat/completions", json=payload)
print(response.json()["choices"][0]["message"]["content"])
3. Comparative Analysis & Benchmarks (Including Break-Even Analysis)
On the standard QuantiPhy-validation benchmark, Code-as-World demonstrates state-of-the-art capability in extracting accurate physical properties directly into code.
| Model Architecture | Model Size / Type | QuantiPhy-validation (MRA Score) | Open Weights? | Local Execution Cost |
|---|---|---|---|---|
| Code-as-World-VL-9B | 9B Fine-Tuned | 55.4 MRA | Yes (Apache 2.0) | $0 (Local Hardware) |
| Gemini-3.1 Flash | Proprietary API | 54.8 MRA | No | Token API Fees |
| Previous Open-Weight SOTA | Open Baseline | ~40.4 MRA | Yes | $0 (Local Hardware) |
Break-Even & TCO Analysis
- Cloud API Costs: Processing thousands of video clips through multi-turn proprietary APIs like Gemini or GPT-4 Vision can quickly tally up to hundreds of dollars per dataset generation run.
- Local Execution: Utilizing Code-as-World on a local workstation (e.g., an RTX 4090 or A10G instance) eliminates variable token costs completely, reaching financial break-even within 2-3 large dataset processing runs while maintaining total data privacy.
4. Community Insights & Real-World Sentiment
Because Code-as-World is a cutting-edge open-source release from MirroS-Lab, community evaluation is rapidly ongoing across developer channels:
- Modularity & Flexibility: Developers appreciate that the output is standard MuJoCo code (
scene.json), which can be directly plugged into RL (Reinforcement Learning) environments such as Gymnasium or Isaac Gym. - Agentic Self-Correction: The 5-round self-verification loop significantly lowers syntax and geometry errors compared to single-shot LLM code outputs.
- Hardware Constraints: Users running lower VRAM GPUs (under 12GB) report needing quantization or offloading to run the 9B parameter version efficiently alongside local MuJoCo rendering.
5. Pro Tips & Maximum Productivity Recipes
- Optimal Frame Extraction: Pre-process input videos to sample keyframes at moments of high physical interaction (e.g., impact, release, or sliding) to give the model maximum structural context.
- Batch Inference with vLLM: Use vLLM’s tensor parallelism and continuous batching capabilities when generating physics representations for large datasets.
- Post-Processing Validation: Integrate an automated syntax checker for
scene.jsonbefore sending files to the MuJoCo simulation environment to bypass invalid XML/JSON schema errors during the loop.
6. Final Verdict & Cost-Benefit Recommendation
Code-as-World represents a massive milestone for embodiment AI, robotics, and physical simulation modeling. By outperforming closed-source models like Gemini-3.1 Flash on physical parameter extraction, MirroS-Lab offers developers an enterprise-grade solution that costs zero dollars in API licensing.
- Adopt Immediately If: You build synthetic training environments for robotics, work with physics-informed AI, or need to reverse-engineer mechanical dynamics from real videos.
- Hold Off If: You only need 2D video analytics without any requirement for interactive or editable 3D simulation code.
7. Frequently Asked Questions (FAQ)
Q1: What makes Code-as-World different from traditional Video-to-3D tools?
Instead of outputting non-editable meshes or point clouds, Code-as-World generates parametric, executable MuJoCo simulation code (scene.json), allowing you to tweak physical properties like mass, friction, and gravity directly.
Q2: Can I run Code-as-World locally on consumer GPU hardware?
Yes. The Code-as-World-VL-4B variant runs comfortably on standard GPUs with 12GB-16GB VRAM, while the Code-as-World-VL-9B model benefits from 24GB VRAM cards like the RTX 3090/4090 for fast local vLLM serving.
Q3: How does the agentic feedback loop work during generation?
Code-as-World generates an initial physics script, renders it inside the MuJoCo simulator, compares the rendered frames against the original video input, and iteratively refines the physical parameters across up to 5 validation rounds.
