In a world where machines excel at observing motion, the challenge now lies in predicting it. Enter MolmoMotion, a groundbreaking model unveiled by AllenAI that forecasts 3D motion based on video frames, 3D points on objects, and descriptive language instructions.
What is MolmoMotion?
MolmoMotion is designed to predict the future trajectories of objects in three-dimensional space. By analyzing a video frame alongside marked 3D points and action descriptions—such as “Move and rotate the wooden bowl with fruit on the table”—the model anticipates where these points will move over the next few seconds. This capability significantly outperforms existing forecasting methods.
Key Features and Innovations
The model operates by representing motion as a set of object-attached 3D points in a world frame, which allows for a class-agnostic, view-stable representation of motion. This means it can effectively describe various types of movements without being limited to specific object categories. The architecture leverages Molmo 2 as its backbone, linking language instructions to visual elements in the input.
MolmoMotion comes in two variants: the autoregressive model (MolmoMotion-AR) predicts future coordinates sequentially, while the flow-matching model (MolmoMotion-FM) captures uncertainty by transforming noise into motion. This dual approach enhances the model’s ability to handle diverse forecasting scenarios.
Data and Benchmarking
To support the training of MolmoMotion, the team created MolmoMotion-1M, the largest dataset of 3D point trajectories paired with action descriptions, sourced from 1.16 million videos. Additionally, they developed PointMotionBench, a benchmark designed to evaluate the accuracy of 3D motion forecasting across 2.7K video clips.
Performance and Applications
In evaluations, MolmoMotion demonstrated superior accuracy compared to existing forecasting methods, successfully predicting various object movements. Its applications extend to robotics, where it aids in planning object manipulation tasks, and to video generation, where it enhances the realism of generated motion.
Despite its capabilities, MolmoMotion has limitations, such as its reliance on eight query points per object, which may restrict its ability to represent complex deformable motions. Nevertheless, it marks a significant step toward integrating motion forecasting into machine intelligence, with potential applications in robotics and video generation.
MolmoMotion is now available for public use, inviting further exploration and innovation within the community.
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
