NVIDIA has introduced a significant advancement in its Cosmos Predict 2.5, a large-scale world model designed to create physically plausible videos based on text, images, or video clips. This model is particularly useful for applications in robotics, where specific adaptations are necessary for tasks like robot manipulation or viewpoint adjustments.
The challenge lies in the need for targeted fine-tuning to adapt the model to these specific domains. Collecting real-robot trajectories for training is often slow and costly, making the generation of synthetic trajectories through a fine-tuned video world model an attractive alternative. However, the full fine-tuning of a model with 2 billion parameters can be resource-intensive and may lead to catastrophic forgetting of general knowledge.
To address these issues, NVIDIA employs LoRA (Low-Rank Adaptation) and DoRA (Dynamic Low-Rank Adaptation), which integrate small, trainable adapter modules into the frozen base model. This approach reduces memory requirements and allows for the fine-tuning process to be conducted on a single GPU, while also enabling the flexible swapping of adapters for different domains during inference.
Training Methodology
The fine-tuning process utilizes the diffusers and accelerate libraries, supporting both single and multi-GPU training. The training dataset consists of 92 robot manipulation videos paired with text prompts describing pick-and-place tasks, while the test dataset includes 50 prompt-image pairs. The model is trained to generate videos based on these inputs.
During training, the model’s architecture includes a VAE (Variational Autoencoder) for encoding videos, a text encoder for processing prompts, and a diffusion model for generating outputs in latent space. The LoRA adapters are strategically placed within the model’s attention projections and feedforward layers, allowing for efficient updates without altering the frozen weights of the base model.
Evaluation and Results
The training process is designed to optimize the model’s ability to predict the velocity that transports a noisy sample toward the original clean data. The loss function employed is based on mean-squared errors, focusing on non-conditioned frames to ensure effective learning.
Once training is complete, the fine-tuned model can generate videos from the evaluation dataset, demonstrating its capability to produce synthetic robot trajectories for downstream learning tasks. The implementation details, including the training command and evaluation metrics, are provided to facilitate reproducibility and further experimentation.
In conclusion, NVIDIA’s fine-tuning of Cosmos Predict 2.5 with LoRA and DoRA presents a scalable solution for generating synthetic trajectories in robotics, marking a notable step forward in the intersection of AI and robotics.
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
