In the evolving realm of artificial intelligence, the introduction of NeoMME marks a significant leap forward. This family of encoders, available in 260M and 800M parameter sizes, is designed to process both text and images using a unified approach.
What is NeoMME?
NeoMME stands out from traditional generative visual language models by eliminating the need for a separate pretrained vision tower or a causal language model. Instead, it employs a single bidirectional Transformer that simultaneously processes text tokens and raw image patches. The model is trained from scratch using a masked discrete-diffusion objective, allowing it to generate dense embeddings efficiently.
Performance and Efficiency
Fine-tuned for visual document retrieval, NeoMME utilizes the page-image approach from ColPali, enabling it to return embeddings in a single forward pass. The 260M model achieves an impressive encoding rate of approximately 51 pages per second on an NVIDIA L40S GPU, nearly doubling the throughput of its predecessor, ColModernVBERT. Furthermore, innovations like hierarchical token pooling and asymmetric quantization have reduced the storage requirements for late-interaction indices from about 1.5 MB to just 6 kB per page, while maintaining over 95% of baseline retrieval quality.
Architectural Innovations
NeoMME’s architecture is designed for efficiency and flexibility. Both model sizes share a common framework that includes native multimodal inputs, allowing text and images to be processed through the same computational pathway. The model supports dynamic image resolution, retaining aspect ratios to optimize the number of tokens used for high-resolution documents. With a context length of 16,384 tokens, NeoMME can handle substantial input sizes, accommodating up to two standard 4K UHD images.
Retrieval Capabilities
The NeoMME-Retriever enhances the model’s functionality by incorporating two jointly trained heads for retrieval tasks. The dense head averages hidden state vectors for compact embeddings, while the late-interaction head projects outputs into normalized vectors for finer granularity. This dual-head design allows for flexible retrieval strategies, accommodating various use cases.
In competitive evaluations, NeoMME-Retriever-260M achieved a score of 0.523 on the nDCG@10 metric, outperforming other models with fewer parameters. The 800M variant reached 0.556, demonstrating its capability within the model-size Pareto frontier.
NeoMME is now available in the Hugging Face Transformers library, with all model checkpoints released under the Apache 2.0 license, inviting further exploration and application in the field of multimodal AI.
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
