Skip to content
Nvidia — Robotics — News

NVIDIA Shows Two-Prompt Workflow for Building Physical AI Datasets With Cosmos

Coding agents coordinate data generation, quality checks, and labeling across five workflows for robotics, autonomous vehicles, and visual inspection.

Table of Contents

NVIDIA has demonstrated how a coding agent can turn two prompts into a labeled physical AI training dataset using Cosmos Data Factory. The workflow links the tools for finding source material, generating visual variations, checking results, and annotating the output, which cuts the manual coordination between those steps.

The two-prompt walkthrough covers five reference blueprints spanning robot footage, autonomous driving, person search, rare events, and industrial defects. In the robotics example, an agent creates new lighting and tabletop conditions from an existing manipulation video, then checks and labels the generated clips without requiring another recording of the task.

Cosmos Data Factory brings those operations into a pipeline that developers can run on their own GPU infrastructure or in the cloud. Its blueprints combine six reusable agent skills, so teams can use a complete workflow or assemble individual components around their data.

NVIDIA Cosmos Data Factory hero
NVIDIA Cosmos Data Factory hero | NVIDIA Technical Blog

What the Two Prompts Set in Motion

The first prompt handles environment preparation. Developers ask their coding agent to clone the CDF Orchestration repository and run its preflight and setup skills. Preflight inspects the environment without changing it and identifies missing prerequisites; setup proceeds after those checks pass.

The second prompt describes the dataset job: which inputs to use, what variations or events to produce, and where to save the results. CDF Orchestration coordinates the workflow, deploys the necessary services, and manages execution on the cluster.

The Cosmos Data Factory repository lists a Kubernetes cluster, object storage, a container registry, an NVIDIA NGC API key, and a Hugging Face read token among the prerequisites. GPU capacity depends on the blueprint and whether generation and language models run locally or through external endpoints. The two prompts are the interface to that infrastructure, which teams still have to provide.

Underneath, the reference blueprints run as Apache Airflow directed acyclic graphs: dependency-ordered tasks that pass results from one processing stage to the next. Kubernetes supports all five blueprints, while Event Video Generation and Video Data Augmentation also support an OSMO backend.

The six core skills cover orchestration, augmentation, automatic labeling, anomaly generation, curation and retrieval, and simulation. The simulation skill uses Isaac Sim and Omniverse Replicator to render labeled images from 3D scenes. Teams can also run labeling or retrieval on data they already have.

AI agent video labeling
AI agent video labeling | NVIDIA Technical Blog

Automatic labeling supplies detections, object tracks, captions, and scene information. Because that stage connects to generation, each generated clip arrives already annotated and ready for downstream training.

Reworking Robot Footage Without Repeating the Task

The robot-manipulation demonstration starts with footage of a robot handling a cube. The requested variations introduce yellow lighting, dim lighting, and a red tabletop, which widen the visual conditions beyond the original recording.

A vision-language model first describes the source video and combines that description with the requested changes into a structured prompt. Cosmos3-Nano then generates the altered footage while retaining the scene structure.

Cosmos Evaluator checks the results for motion artifacts absent from the source, verifies that the requested conditions appear, and scores physical plausibility. Failed variations are retried. The pipeline then labels both the original and generated videos with detections, tracks, and captions.

Video Data Augmentation reference blueprint
Video Data Augmentation reference blueprint | NVIDIA Technical Blog

A robotics team can use this to expand appearance coverage around an action it has already recorded. One manipulation sequence then supplies more examples of the same task under different illumination or surface appearance, so the team does not have to stage each visual condition again around the physical robot.

From Driving Scenarios to Factory Defects

The other four blueprints address different gaps in training data, using retrieval, controlled edits, or generation from a starting image or 3D asset.

Autonomous Vehicle Dataset Generation uses Cosmos Curator to extract metadata and Cosmos Dataset Search to retrieve driving clips similar to an example. Cosmos3-Super-Image2Video can transform the retrieved scenes into rainy, snowy, or foggy variants while preserving layout and motion. Cosmos Evaluator checks the generated footage for physical plausibility.

Image Attribute Augmentation changes clothing, accessories, and colors in cropped images of people. A language model converts the selected attributes into modification prompts, an evaluator verifies the changes, and automatic labeling produces clothing attributes and search queries for person search and re-identification.

Event Video Generation begins with a single scene image and creates clips depicting events such as fires, falls, or shoplifting. The pipeline preserves the starting viewpoint and layout, checks whether the requested event occurs, evaluates motion, and retries failed outputs before adding detections, tracks, captions, and semantic annotations.

Event Video Generation reference blueprint
Event Video Generation reference blueprint | NVIDIA Technical Blog

Defect Image Generation produces inspection examples for printed circuit boards, glass, and metal. It can start with clean inspection images or 3D USD assets. For 3D inputs, Isaac Sim renders images and masks, and Qwen-Image-Edit creates realistic clean references. Cosmos 3 Super uses the references, defect masks, and requested defect types to generate damaged examples while preserving the surrounding part. Outputs include clean references, synthetic defects, and manifests recording source material and generation settings.

These workflows build on Cosmos tools introduced in October. NVIDIA reported that ANYbotics had generated more than 1,500 synthetic clips using the blueprints and validated outputs with Encord to help train industrial-inspection robots to recognize spills and other changes between facility visits.

Compute Costs and Getting Started

NVIDIA measured throughput on a cloud server with eight RTX PRO 6000 Blackwell GPUs, priced at roughly $40 per server-hour, or $960 per day. It reports that defect-dataset assembly fell from two months to one day, a 60-fold speedup, and that autonomous vehicle scenario generation was about 45 times cheaper than its comparison with road collection by a fleet crew.

For video augmentation, NVIDIA reports approximately 305 annotated clips per day at $3.15 in GPU compute per clip, compared with an average of $22 it had paid for third-party data. These are compute-only estimates; source preparation, setup, agent use, storage, and human review can add to the total cost.

Cosmos Data Factory is available through GitHub, Microsoft Azure, Nebius, and CoreWeave. Developers can start with a reference blueprint or combine core skills into a custom pipeline. NVIDIA has also scheduled a Cosmos Labs livestream with Encord and Voxel51 for November 19 at 11 a.m. Pacific, covering dataset workflows, robotic inspection use cases, and questions from developers.

Comments

Latest