针对视觉-语言模型的个性化混合专家联邦微调框架

丁美琳; 靳晓嘉; 李荣盛; 赵东明; 高菲; 赵云凤; 仇超; 王晓飞

doi:10.11959/j.issn.1000-0801.2025197

您当前的位置：

首页 >

文章列表页 >

针对视觉-语言模型的个性化混合专家联邦微调框架

专题：AI赋能通信网络 | 更新时间：2026-01-20

- 针对视觉-语言模型的个性化混合专家联邦微调框架
- A personalized mixture-of-experts federated fine-tuning framework for vision-language models
- 电信科学 2025年41卷第11期页码：57-66
- 作者机构：
  
  1.天津大学智能与计算学部，天津 300350
  2.中国移动通信集团天津有限公司人工智能产业研究院，天津 300020
- 作者简介：
  
  [ "丁美琳（2000- ），女，天津大学智能与计算学部在读，主要研究方向为边缘计算。" ]
  [ "靳晓嘉（1972- ），男，博士，中国移动通信集团天津有限公司副总经理、正高级工程师，主要研究方向为AI 大模型构建、大数据分析及网络规划等。" ]
  [ "李荣盛（1977- ），男，中国移动通信集团天津有限公司人工智能产业研究院高级工程师，主要研究方向为自智网络、云计算、人工智能。" ]
  [ "赵东明（1984- ），男，博士，中国移动通信集团天津有限公司人工智能产业研究院正高级工程师、技术总监/高级专家，主要研究方向为自然语言处理和大语言模型。" ]
  [ "高菲（1997- ），女，天津大学智能与计算学部硕士生，主要研究方向为边缘计算和联邦学习。" ]
  [ "赵云凤（1997- ），女，天津大学智能与计算学部博士生，主要研究方向为边缘计算、边缘智能和分布式机器学习。" ]
  [ "仇超（1988- ），女，天津大学智能与计算学部副教授，主要研究方向为边缘计算、边缘智能和区块链。" ]
  [ "王晓飞（1982- ），男，天津大学智能与计算学部教授，主要研究方向为边缘计算、边缘智能和边缘系统。" ]
- 基金信息：
  
  2024年天津市制造业高质量发展专项资金项目“自主智能算力的通用大模型关键技术研究及产业化应用示范”(24ZGZNGX00020)
- DOI：10.11959/j.issn.1000-0801.2025197
  中图分类号： TP393
- 收稿：2025-06-10，
  
  修回：2025-07-11，
  
  录用：2025-08-05，
  
  纸质出版：2025-11-20
- 稿件说明：
移动端阅览
丁美琳,靳晓嘉,李荣盛等.针对视觉-语言模型的个性化混合专家联邦微调框架[J].电信科学,2025,41(11):57-66.

DING Meilin,JIN Xiaojia,LI Rongsheng,et al.A personalized mixture-of-experts federated fine-tuning framework for vision-language models[J].Telecommunications Science,2025,41(11):57-66.
丁美琳,靳晓嘉,李荣盛等.针对视觉-语言模型的个性化混合专家联邦微调框架[J].电信科学,2025,41(11):57-66. DOI： 10.11959/j.issn.1000-0801.2025197.

DING Meilin,JIN Xiaojia,LI Rongsheng,et al.A personalized mixture-of-experts federated fine-tuning framework for vision-language models[J].Telecommunications Science,2025,41(11):57-66. DOI： 10.11959/j.issn.1000-0801.2025197.

摘要

在视觉-语言模型（vision-language model，VLM）广泛应用于图文检索、图像标注与视觉问答等任务的背景下，如何在保护跨行业数据隐私与应对计算资源受限的前提下，高效训练大规模模型成为通信运营商的关键挑战。联邦学习（federated learning，FL）通过在不共享原始数据的前提下进行分布式协同训练，缓解了数据孤岛和隐私问题。然而，VLM参数量庞大，训练过程中通信与计算开销高，加之FL环境中数据异质性强，导致全局模型泛化能力受限。为此，提出一种针对VLM的个性化混合专家联邦微调（federated personalized low-rank mixture of experts，FedLRM）框架，结合参数高效微调方法——低秩自适应（low-rank adaptation，LoRA）技术与门控机制，在本地构建融合全局与本地特征的混合专家（mixture of experts，MoE）架构，提升在细粒度数据层级的个性化表现。实验表明，在异构数据场景下，FedLRM相较于对比方法准确度最多提升1.88%，验证了其为个性化VLM的联邦优化提供了有效方案。

Abstract

With the widespread application of vision-language model (VLM) in tasks such as image-text retrieval

image captioning

and visual question answering

efficiently training large-scale models under the constraints of cross-industry data privacy and limited computational resources was recognized as a critical challenge for telecom operators. Federated learning (FL) was employed to address data silos and privacy concerns by enabling distributed collaborative training without sharing raw data. However

the massive number of parameters in VLM was found to lead to high communication and computation costs during training. Additionally

the strong data heterogeneity in FL environments was observed to limit the generalization capability of global models. To address these challenges

federated personalized low-rank mixture of experts (FedLRM)

a personalized mixture-of-experts federated fine-tuning framework for VLM was proposed. It combined the parameter-efficient tuning method low-rank adaptation (LoRA) with a gating mechanism to build a local mixture of experts (MoE) architecture that fuses global and local features

enhancing fine-grained personalization. Experiments results show that in heterogeneous data scenarios

FedLRM improves accuracy by up to 1.88% compared to baseline methods

verifying that it provids an effective solution for the federated optimization of personalized vision-language models.

关键词

Keywords

references

ZHANG J Y , HUANG J X , JIN S , et al . Vision-language models for vision tasks: a survey [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024 , 46 ( 8 ): 5625 - 5644 .

QI P , CHIARO D , GUZZO A , et al . Model aggregation techniques in federated learning: a comprehensive survey [J ] . Future Generation Computer Systems , 2024 , 150 : 272 - 293 .

WEN J , ZHANG Z X , LAN Y , et al . A survey on federated learning: challenges and applications [J ] . International Journal of Machine Learning and Cybernetics , 2023 , 14 ( 2 ): 513 - 535 .

HE H Y , CAI J F , ZHANG J , et al . Sensitivity-aware visual parameter-efficient fine-tuning [C ] // Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2023 : 11791 - 11801 .

WANG S W , YU L X , LI J . Lora-ga: low-rank adaptation with gradient approximation [J ] . arXiv preprint , 2024 ,arXiv:2407. 05000

WEI K , LI J , MA C , et al . Personalized federated learning with differential privacy and convergence guarantee [J ] . IEEE Transactions on Information Forensics and Security , 2023 , 18 : 4488 - 4503 .

HAO J F , CHEN P , CHEN J , et al . Multi-task federated learning-based system anomaly detection and multi-classification for microservices architecture [J ] . Future Generation Computer Systems , 2024 , 159 : 77 - 90 .

YE C Y , ZHENG H , HU Z G , et al . PFedSA: personalized federated multi-task learning via similarity awareness [C ] // Proceedings of the 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) . Piscataway : IEEE Press , 2023 : 480 - 488 .

LI T , SAHU A K , ZAHEER M , et al . Federated optimization in heterogeneous networks [J ] . arXiv preprint , 2018 , arXiv:1812. 06127

CHEN T L , CHEN X X , DU X Z , et al . AdaMV-MoE: adaptive multi-task vision mixture-of-experts [C ] // Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2023 : 17300 - 17311 .

FENG Y , GENG Y L , ZHU Y F , et al . PM-MOE: mixture of experts on private model parameters for personalized federated learning [C ] // Proceedings of the ACM on Web Conference 2025 . New York : ACM Press , 2025 : 134 - 146 .

MEI H , CAI D , ZHOU A , et al . FedMoE: personalized federated learning via heterogeneous mixture of experts [J ] . arXiv preprint , 2024 , arXiv: 2408.11304 .

RADFORD A , KIM J W , HALLACY C , et al . Learning transferable visual models from natural language supervision [J ] . arXiv preprint , 2021 , arXiv:2103. 00020

BANSAL M , KUMAR M , SACHDEVA M , et al . Transfer learning for image classification using VGG19: Caltech-101 image data set [J ] . Journal of Ambient Intelligence and Humanized Computing , 2023 , 14 ( 4 ): 3609 - 3620 .

KRIZHEVSKY A . Learning multiple layers of features from tiny images [D ] . Toronto : University of Toronto , 2009 .

MCMAHAN H B , MOORE E , RAMAGE D , et al . Communication-efficient learning of deep networks from decentralized data [C ] // Proceedings of the International Conference on Artificial Intelligence and Statistics , 2016 .

DENG Y , KAMANI M M , MAHDAVI M . Adaptive personalized federated learning [J ] . arxiv preprint , 2020 , arXiv: 2003. 1346 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

FedGMH：基于全局多头部的标签干扰消除方法研究

FedDACS: 基于分布感知可控采样优化的个性化群体知识迁移