VentusAislant/MSLoRA_CR

Tasks

• visual-question-answering

Citation text from the card

@inproceedings{10.1145/3746027.3754917,
author = {Zhang, Haojie and Liang, Yixiong and Kuang, Hulin and Cen, Lihui and Qu, Zhe and Cen, Yigang and Zeng, Min and Kan, Shichao},
title = {Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning},
year = {2025},
isbn = {9798400720352},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3746027.3754917},
doi = {10.1145/3746027.3754917},
abstract = {Multimodal Biomedical Image Incremental Learning (MBIIL) is essential for handling diverse tasks and modalities in the biomedical domain, as training separate models for each modality or task significantly increases inference costs. Existing incremental learning methods focus on task expansion within a single modality, whereas MBIIL seeks to train a unified model incrementally across modalities. The MBIIL faces two challenges: I) How to preserve previously learned knowledge during incremental updates? II) How to effectively leverage knowledge acquired from existing modalities to support new modalities?  To address these challenges, we propose MSLoRA-CR, a method that fine-tunes Modality-Specific LoRA modules while incorporating Contrastive Regularization to enhance intra-modality knowledge sharing and promote inter-modality knowledge differentiation. Our approach builds upon a large vision-language model (LVLM), keeping the pretrained model frozen while incrementally adapting new LoRA modules 

All datasets marked Permissive or share-alike | JSON