A personalized mixture-of-experts federated fine-tuning framework for vision-language models

DING Meilin; JIN Xiaojia; LI Rongsheng; ZHAO Dongming; GAO Fei; ZHAO Yunfeng; QIU Chao; WANG Xiaofei

doi:10.11959/j.issn.1000-0801.2025197

您当前的位置：

首页 >

文章列表页 >

A personalized mixture-of-experts federated fine-tuning framework for vision-language models

Topic: AI-Empowered Communication Networks | 更新时间：2026-01-20

- A personalized mixture-of-experts federated fine-tuning framework for vision-language models
- Telecommunications Science Vol. 41, Issue 11, Pages: 57-66(2025)
- 作者机构：
  
  1.天津大学智能与计算学部，天津 300350
  2.中国移动通信集团天津有限公司人工智能产业研究院，天津 300020
- 作者简介：
- 基金信息：
  
  2024 Tianjin Manufacturing High-quality Development Special Project “Research on Key Technologies of General Large Models of Autonomous Intelligent Computing Power and Industrial Application Demonstration”(24ZGZNGX00020)
- DOI：10.11959/j.issn.1000-0801.2025197
  CLC： TP393
- Received：10 June 2025，
  
  Revised：2025-07-11，
  
  Accepted：05 August 2025，
  
  Published：20 November 2025
- 稿件说明：
移动端阅览
丁美琳,靳晓嘉,李荣盛等.针对视觉-语言模型的个性化混合专家联邦微调框架[J].电信科学,2025,41(11):57-66.

DING Meilin,JIN Xiaojia,LI Rongsheng,et al.A personalized mixture-of-experts federated fine-tuning framework for vision-language models[J].Telecommunications Science,2025,41(11):57-66.
丁美琳,靳晓嘉,李荣盛等.针对视觉-语言模型的个性化混合专家联邦微调框架[J].电信科学,2025,41(11):57-66. DOI： 10.11959/j.issn.1000-0801.2025197.

DING Meilin,JIN Xiaojia,LI Rongsheng,et al.A personalized mixture-of-experts federated fine-tuning framework for vision-language models[J].Telecommunications Science,2025,41(11):57-66. DOI： 10.11959/j.issn.1000-0801.2025197.

摘要

在视觉-语言模型（vision-language model，VLM）广泛应用于图文检索、图像标注与视觉问答等任务的背景下，如何在保护跨行业数据隐私与应对计算资源受限的前提下，高效训练大规模模型成为通信运营商的关键挑战。联邦学习（federated learning，FL）通过在不共享原始数据的前提下进行分布式协同训练，缓解了数据孤岛和隐私问题。然而，VLM参数量庞大，训练过程中通信与计算开销高，加之FL环境中数据异质性强，导致全局模型泛化能力受限。为此，提出一种针对VLM的个性化混合专家联邦微调（federated personalized low-rank mixture of experts，FedLRM）框架，结合参数高效微调方法——低秩自适应（low-rank adaptation，LoRA）技术与门控机制，在本地构建融合全局与本地特征的混合专家（mixture of experts，MoE）架构，提升在细粒度数据层级的个性化表现。实验表明，在异构数据场景下，FedLRM相较于对比方法准确度最多提升1.88%，验证了其为个性化VLM的联邦优化提供了有效方案。

Abstract

With the widespread application of vision-language model (VLM) in tasks such as image-text retrieval

image captioning

and visual question answering

efficiently training large-scale models under the constraints of cross-industry data privacy and limited computational resources was recognized as a critical challenge for telecom operators. Federated learning (FL) was employed to address data silos and privacy concerns by enabling distributed collaborative training without sharing raw data. However

the massive number of parameters in VLM was found to lead to high communication and computation costs during training. Additionally

the strong data heterogeneity in FL environments was observed to limit the generalization capability of global models. To address these challenges

federated personalized low-rank mixture of experts (FedLRM)

a personalized mixture-of-experts federated fine-tuning framework for VLM was proposed. It combined the parameter-efficient tuning method low-rank adaptation (LoRA) with a gating mechanism to build a local mixture of experts (MoE) architecture that fuses global and local features

enhancing fine-grained personalization. Experiments results show that in heterogeneous data scenarios

FedLRM improves accuracy by up to 1.88% compared to baseline methods

verifying that it provids an effective solution for the federated optimization of personalized vision-language models.

关键词

Keywords

references

ZHANG J Y , HUANG J X , JIN S , et al . Vision-language models for vision tasks: a survey [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024 , 46 ( 8 ): 5625 - 5644 .

QI P , CHIARO D , GUZZO A , et al . Model aggregation techniques in federated learning: a comprehensive survey [J ] . Future Generation Computer Systems , 2024 , 150 : 272 - 293 .

WEN J , ZHANG Z X , LAN Y , et al . A survey on federated learning: challenges and applications [J ] . International Journal of Machine Learning and Cybernetics , 2023 , 14 ( 2 ): 513 - 535 .

HE H Y , CAI J F , ZHANG J , et al . Sensitivity-aware visual parameter-efficient fine-tuning [C ] // Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2023 : 11791 - 11801 .

WANG S W , YU L X , LI J . Lora-ga: low-rank adaptation with gradient approximation [J ] . arXiv preprint , 2024 ,arXiv:2407. 05000

WEI K , LI J , MA C , et al . Personalized federated learning with differential privacy and convergence guarantee [J ] . IEEE Transactions on Information Forensics and Security , 2023 , 18 : 4488 - 4503 .

HAO J F , CHEN P , CHEN J , et al . Multi-task federated learning-based system anomaly detection and multi-classification for microservices architecture [J ] . Future Generation Computer Systems , 2024 , 159 : 77 - 90 .

YE C Y , ZHENG H , HU Z G , et al . PFedSA: personalized federated multi-task learning via similarity awareness [C ] // Proceedings of the 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . Piscataway : IEEE Press , 2023 : 480 - 488 .

LI T , SAHU A K , ZAHEER M , et al . Federated optimization in heterogeneous networks [J ] . arXiv preprint , 2018 , arXiv:1812. 06127

CHEN T L , CHEN X X , DU X Z , et al . AdaMV-MoE: adaptive multi-task vision mixture-of-experts [C ] // Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2023 : 17300 - 17311 .

FENG Y , GENG Y L , ZHU Y F , et al . PM-MOE: mixture of experts on private model parameters for personalized federated learning [C ] // Proceedings of the ACM on Web Conference 2025 . New York : ACM Press , 2025 : 134 - 146 .

MEI H , CAI D , ZHOU A , et al . FedMoE: personalized federated learning via heterogeneous mixture of experts [J ] . arXiv preprint , 2024 , arXiv: 2408.11304 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [J ] . arXiv preprint , 2021 , arXiv:2103. 00020

BANSAL M , KUMAR M , SACHDEVA M , et al . Transfer learning for image classification using VGG19: Caltech-101 image data set [J ] . Journal of Ambient Intelligence and Humanized Computing , 2023 , 14 ( 4 ): 3609 - 3620 .

KRIZHEVSKY A . Learning multiple layers of features from tiny images [D ] . Toronto : University of Toronto , 2009 .

MCMAHAN H B , MOORE E , RAMAGE D , et al . Communication-efficient learning of deep networks from decentralized data [C ] // Proceedings of the International Conference on Artificial Intelligence and Statistics , 2016 .

DENG Y , KAMANI M M , MAHDAVI M . Adaptive personalized federated learning [J ] . arxiv preprint , 2020 , arXiv: 2003. 1346 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

FedGMH: research on label interference elimination via global multi-head

FedDACS: personalized knowledge transfer based on distribution-aware controlled sampling optimization

Related Author

SUN Zhenfeng

NI Zhengwei

QI Xiaochao

NI Zhengwei

Related Institution

College of Information and Electronic Engineering, Zhejiang Gongshang University

School of Information and Electronic Engineering, Zhejiang Gongshang University

⁰