The realm of artificial intelligence is expanding, and with it, the capabilities of multimodal models. The Sentence Transformers library has unveiled new features that allow for the training and fine-tuning of models capable of processing text, images, audio, and video.
Multimodal Capabilities
In a recent blog post, the author introduced the ability to utilize embedding and reranker models for various applications, including retrieval augmented generation and semantic search. This article delves deeper into how to train or fine-tune these multimodal models on specific datasets, using the Qwen/Qwen3-VL-Embedding-2B model as a case study for Visual Document Retrieval (VDR).
Fine-Tuning for Enhanced Performance
The process of fine-tuning the Qwen/Qwen3-VL-Embedding-2B model has shown significant improvements in performance. By adapting the model to a specific domain, the author achieved an NDCG@10 score of 0.947, surpassing the base model’s score of 0.888 and outperforming all tested VDR models, including those up to four times larger.
Training Components and Dataset Utilization
Training multimodal models involves several key components: the model itself, the dataset, the loss function, and optional training arguments. The SentenceTransformerTrainer facilitates this process, automatically handling image preprocessing alongside text data. The dataset used for this example, tomaarsen/llamaindex-vdr-en-train-preprocessed, consists of approximately 500,000 multilingual query-image samples, with a focus on English samples for training.
The training format must align with the chosen loss function, and the dataset can include various modalities such as text, images, audio, and video. The CachedMultipleNegativesRankingLoss is employed to enhance retrieval tasks, leveraging both hard negatives and in-batch negatives to strengthen the training signal.
Conclusion
The advancements in the Sentence Transformers library highlight the potential of fine-tuning multimodal models for specific tasks. As demonstrated, the ability to adapt models like Qwen/Qwen3-VL-Embedding-2B for Visual Document Retrieval not only improves performance but also showcases the evolving landscape of AI capabilities.
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
