NVIDIA Adds Cosmos Simulation Recipes for Self-Driving Cars, Robot Arms, and Surgery

NVIDIA has expanded Cosmos with simulation recipes and released example checkpoints for autonomous driving, two-arm robots, and surgical robotics. Developers can use them to generate synthetic observations and test AI policies in interactive environments.

The Cosmos update combines those task-specific models with broader training support and optimized deployment through Cosmos NIM microservices. Cosmos Framework is also expanding beyond Cosmos 3 to support NVIDIA Alpamayo, GR00T, Qwen, pi0.5, and other compatible Hugging Face models.

The simulation work relies on a feedback loop. An AI policy chooses an action, the environment changes, and a world model generates the observations the policy receives next. That lets developers examine how a driving or robot policy responds to the consequences of its own decisions, rather than only testing it against prerecorded footage.

Driving Simulation With Synchronized Cameras and LiDAR

NVIDIA's driving recipes separate offline scene generation from interactive evaluation. Both use World Scenario Map controls, which describe lanes, objects, road markings, traffic signs, and boundaries. Text prompts specify conditions such as weather, lighting, and time of day, while initial camera images can guide the scene's appearance. Joint camera–LiDAR generation also uses HD-map range controls.

Cosmos3-Nano-Transfer-Auto generates an entire multiview clip together, using a denoising process to produce synchronized camera video and LiDAR observations. Developers can change a daytime drive into a rainy nighttime scene while keeping the road layout and vehicle trajectories, creating variations for training and evaluating perception and driving systems.

The model supports 30-frame-per-second input and output, with up to 400 frames at 480p or 720p. Longer sequences can be generated in successive chunks. NVIDIA's four-GPU reference run produces seven synchronized 720p videos, each containing 297 frames, alongside 99 LiDAR sweeps.

Cosmos3-Nano-Sim-Auto takes a different approach. It is a causal, few-step-distilled variant that generates observations incrementally instead of processing a complete clip at once. Distillation reduces the generation steps needed to produce an output, supporting the interactive workflow.

In the closed-loop setup, a driving policy such as Alpamayo, or a human operator, sends an action to the AlpaSim runtime. AlpaSim updates the simulation state, passes the context to Cosmos, and returns generated camera observations to the policy. The next decision therefore depends on what happened after the previous one.

NVIDIA's framework documentation recommends that developers adapting the system to new sensors or environments fine-tune Transfer-Auto first, then distill the adapted checkpoint into Sim-Auto. That sequence preserves the few-step generation behavior needed for streaming inference.

Robot Demonstrations and Surgical Policy Testing

The robotics recipes address two different needs: changing how a demonstration looks and generating what a robot sees after taking an action.

Cosmos3-Nano-Sim-Transfer converts depth video and a text prompt into RGB video with different lighting, textures, and backgrounds while preserving geometry and motion. For robot demonstrations, it changes the third-person view while retaining the actions, states, and wrist-camera recordings. This allows the same demonstrated behavior to appear in varied visual conditions without changing its underlying action sequence.

Cosmos3-Nano-Sim-Bimanual, by contrast, generates the next visual observation directly from the scene and robot actions. It supports interactive two-arm simulation for YAM and AgiBot robot embodiments, with control supplied by an AI policy or human-operated joysticks. In Sim-Transfer, a physics simulator maintains the state that the model renders; Sim-Bimanual predicts the observation from the action-conditioned scene.

The surgical models apply learned visual dynamics to an environment where conventional simulation involves tissue deformation and complex physical interactions. Cosmos3-H-Embodied-Nano, post-trained on Open-H-Embodiment, can generate surgical video from robot actions, predict actions from video, or generate both together.

Its inputs include an initial RGB frame, a task prompt, the last 16 executed actions, and subsequent robot actions. Developers can use those inputs to generate synthetic data and evaluate policies offline without constructing a physics model of each surgical scene.

Cosmos3-H-Dreams is a causal, few-step model distilled from that system. It generates frames from live robot kinematics, the robot's motion and configuration, for real-time, closed-loop simulation. A human operator or learned policy can control the interaction, which lets developers explore maneuvers and evaluate policies before testing on physical hardware.

A Broader Training Framework

The open, modular Cosmos Framework brings training, post-training, optimization, and inference into one development stack. Developers can adapt models with LoRA, which trains a smaller set of additional parameters, or use full supervised fine-tuning to update all model parameters. Data adapters support JSONL, WebDataset, LeRobot, and custom formats.

Its post-training runtime covers offline preference optimization with DPO, online reinforcement learning with GRPO and PPO, on-policy distillation from a teacher model, and DiffusionNFT for diffusion world models. Rollout generation, reward computation, and training can share GPUs or run on separate pools, and supported simulators can supply experience for embodied-policy training.

Bundled coding-agent skills cover setup, inference, post-training, and troubleshooting. NVIDIA's example workflow gives an agent a bounded task: pin software revisions, validate a dataset, run a smoke test, stop at a specified GPU-hour budget, and compare the trained checkpoint against the base model on held-out data. The skills package repeatable development procedures alongside the model recipes.

The new simulation checkpoints support PyTorch and vLLM-Omni and have been tested on Hopper and Blackwell GPUs. Other compatible checkpoints can run through backends including Hugging Face Diffusers, SGLang, vLLM, and NVIDIA's TensorRT-based inference tools.

Serving Cosmos With NIM

Cosmos NIM packages supported models in optimized containers for API-based deployment. Its inference optimizations include quantization, speculative decoding, caching, token pruning, and routing that accounts for the key-value cache used during generation. NVIDIA reports measured mean speedups of roughly two to three times from speculative decoding on H200 GPUs.

Memory requirements vary substantially by task and precision. NVIDIA lists the following requirements for generation:

NIM also serves fine-tuned Cosmos checkpoints and supports streaming vision-language inference for live-video understanding. NVIDIA reports subsecond alerts with 10 times lower latency for that capability.

The deployment workflow uses an NGC API key and a container pulled from the NVIDIA container registry. Generator deployments offer separate latency and throughput profiles: latency mode prioritizes finishing an individual request sooner, while throughput mode uses independent replicas to process concurrent requests. The latency profile supports one, two, four, or eight GPUs; throughput requires two, four, or eight compatible GPUs.

NVIDIA has scheduled a Cosmos Framework livestream for November 5 to walk developers through using the framework.