Training large artificial intelligence models often incurs significant costs—not just financially, but also in terms of time and computational resources. A new technique developed by researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and collaborators aims to address this challenge by compressing models during the training phase rather than after.
Introducing CompreSSM
The method, named CompreSSM, focuses on a specific category of AI architectures known as state-space models, which are utilized in various applications, including language processing, audio generation, and robotics. By leveraging concepts from control theory, the researchers can discern which components of a model are essential and which are superfluous, allowing for the removal of unnecessary elements early in the training process.
Efficiency Through Early Compression
“It’s essentially a technique to make models grow smaller and faster as they are training,” explains Makram Chahine, a PhD student and lead author of the study. The key insight is that the significance of different model components stabilizes early in the training process. Utilizing a mathematical tool known as Hankel singular values, the team demonstrated that they could rank the importance of model dimensions after just 10 percent of the training time. This enables the removal of less critical components, allowing the remaining training to proceed at a speed comparable to that of a smaller model.
Promising Results and Comparisons
The results from CompreSSM are noteworthy. In image classification tasks, compressed models retained nearly the same accuracy as their full-sized counterparts while training up to 1.5 times faster. For instance, a model reduced to about a quarter of its original dimension achieved 85.7 percent accuracy on the CIFAR-10 benchmark, compared to 81.8 percent for a model trained from scratch at that smaller size. On the widely used Mamba architecture, CompreSSM achieved approximately 4x training speedups, compressing a 128-dimensional model down to around 12 dimensions while maintaining competitive performance.
Theoretical Foundations and Future Directions
What sets CompreSSM apart from traditional methods is its theoretical basis. Conventional techniques often involve training a full model before pruning it, incurring the full computational cost. In contrast, CompreSSM integrates compression into the learning process itself, making informed decisions mid-training. The researchers have also established that the importance of model states changes smoothly during training, providing confidence that early identified negligible dimensions will remain so.
While CompreSSM shows great promise, it is most effective on models with a strong correlation between internal state dimension and overall performance. The researchers are already exploring extensions to more complex architectures, including those used in modern AI systems. This foundational work has been accepted for presentation at the International Conference on Learning Representations 2026.
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
