基于联合损失函数和组合对比学习的语义嵌入方法

高晓欣; 陆谣; 孔祥茂; 刘玉玺; 邓伟; 杨淞皓

doi:10.11959/j.issn.1000-0801.2025142

您当前的位置：

首页 >

文章列表页 >

基于联合损失函数和组合对比学习的语义嵌入方法

研究与开发 | 更新时间：2025-08-07

- 基于联合损失函数和组合对比学习的语义嵌入方法
- Semantic embedding via joint loss function and composite contrastive learning
- 电信科学 2025年41卷第7期页码：96-107
- 作者机构：
  
  1.北京中电普华信息技术有限公司，北京 102209
  2.华北电力大学控制与计算机工程学院，北京 102206
- 作者简介：
  
  [ "高晓欣（1983- ），女，北京中电普华信息技术有限公司工程师，主要研究方向为科研体系、标准数字化。" ]
  [ "陆谣（1997- ），女，北京中电普华信息技术有限公司助理工程师，主要研究方向为标准数字化。" ]
  [ "孔祥茂（1993- ），男，北京中电普华信息技术有限公司工程师，主要研究方向为标准数字化、电力数据分析。" ]
  [ "刘玉玺（1984- ），男，博士，北京中电普华信息技术有限公司正高级工程师，主要研究方向为数据模型、人工智能、数据分析、标准数字化。" ]
  [ "邓伟（1976- ），男，北京中电普华信息技术有限公司正高级工程师，主要研究方向为工业互联网、信息通信运营、标准数字化。" ]
  [ "杨淞皓（2001- ），男，华北电力大学控制与计算机工程学院硕士生，主要研究方向为电力标准条款差异性分析。" ]
- 基金信息：
  
  国网总部科技项目(5700-202318834A-4-2-KJ)
- DOI：10.11959/j.issn.1000-0801.2025142
  中图分类号： TP183
- 收稿日期：2024-11-18，
  
  修回日期：2025-05-07，
  
  纸质出版日期：2025-07-20
- 稿件说明：
移动端阅览
高晓欣,陆谣,孔祥茂等.基于联合损失函数和组合对比学习的语义嵌入方法[J].电信科学,2025,41(07):96-107.

GAO Xiaoxin,LU Yao,KONG Xiangmao,et al.Semantic embedding via joint loss function and composite contrastive learning[J].Telecommunications Science,2025,41(07):96-107.
高晓欣,陆谣,孔祥茂等.基于联合损失函数和组合对比学习的语义嵌入方法[J].电信科学,2025,41(07):96-107. DOI： 10.11959/j.issn.1000-0801.2025142.

GAO Xiaoxin,LU Yao,KONG Xiangmao,et al.Semantic embedding via joint loss function and composite contrastive learning[J].Telecommunications Science,2025,41(07):96-107. DOI： 10.11959/j.issn.1000-0801.2025142.

摘要

对比学习在语义嵌入中表现出色，能够通过捕捉数据样本间的关系提升模型的表示能力。然而，其效果主要受正样本构建和目标函数选择的影响。正样本需要精心设计，以确保模型能有效识别有意义的相似性并减少噪声干扰。为此，提出一种新方法，通过拆分、编码、聚合和投射文本来构建正样本。文本被分解为片段，编码用于提取语义内容，聚合用于突出关系，最终投射到适合学习的语义空间。此外，设计了两种监督损失函数，与标准对比损失相辅相成，以增强语义空间的区分性，从而提升模型辨别能力。实验结果表明，该方法在2个公开数据集和1个私有数据集上表现出色，显著提升了语义嵌入质量，解决了对比学习的核心挑战，并为其在自然语言处理领域的进一步应用奠定了基础。

Abstract

Contrastive learning has shown excellent performance in semantic embeddings by capturing relationships between data samples to enhance model representation. However

its effectiveness largely depends on constructing positive samples and selecting appropriate objective functions. Positive samples must be carefully designed to ensure the model can identify meaningful similarities while reducing noise. To address this

a novel method that constructed positive samples by splitting

encoding

aggregating

and projecting text was proposed. The text was broken into segments

encoded to extract semantic content

aggregated to highlight relationships

and projected into a semantic space optimized for learning. Additionally

two supervised loss functions were designed

complementing the standard contrastive loss

to enhance the discriminability of the semantic space and thereby improve the model’s discrimination ability. The experimental results show that this method performes well on two public datasets and one private dataset

significantly improving the quality of semantic embedding

solving the core challenges of contrastive learning

and laying the foundation for further applications in the field of natural language processing.

关键词

Keywords

references

欧阳梦妮 , 樊小超 , 帕力旦·吐尔逊 . 基于目标对齐和语义过滤的多模态情感分析 [J ] . 计算机技术与发展 , 2024 , 34 ( 6 ): 171 - 177 .

OUYANG M N , FAN X C , PALIDAN T S . Multimodal sentiment analysis based on target alignment and semantic filtering [J ] . Computer Technology and Development , 2024 , 34 ( 6 ): 171 - 177 .

吴加辉 , 加云岗 , 王志晓 , 等 . 基于深度学习的微博疫情舆情文本情感分析 [J ] . 计算机技术与发展 , 2024 , 34 ( 7 ): 175 - 183 .

WU J H , JIA Y G , WANG Z X , et al . Sentiment analysis of weibo epidemic public opinion text based on deep learning [J ] . Computer Technology and Development , 2024 , 34 ( 7 ): 175 - 183 .

陈鹏 , 邰彬 , 石英 , 等 . 基于知识图谱的电力设备缺陷问答系统研究 [J ] . 广西师范大学学报(自然科学版) , 2024 , 42 ( 16 ): 149 - 163 .

CHEN P , TAI B , SHI Y , et al . Research on defect Q&A system of power equipment based on knowledge graph [J ] . Journal of Guangxi Normal University (Natural Science Edition) , 2024 , 42 ( 16 ): 149 - 163 .

贾鹏 . 基于大语言模型的农业知识问答系统的研究与设计 [D ] . 秦皇岛 : 河北科技师范学院 , 2024 .

JIA P . Research and design of agricultural knowledge question answering system based on macrolanguage model [D ] . Qinhuangdao : Hebei Normal University of Science & Technology , 2024 .

李东亚 , 白涛 , 香慧敏 , 等 . 基于ERNIE及改进DPCNN的棉花病虫害问句意图识别 [J ] . 山东农业科学 , 2024 , 25 ( 16 ): 143 - 151 .

LI D Y , BAI T , XIANG H M , et al . Intention recognition of cotton pest questions based on ERNIE and improved DPCNN [J ] . Shandong Agricultural Sciences , 2024 , 25 ( 16 ): 143 - 151 .

谌文佳 , 杨琳 , 李金林 . 嵌入意图识别的医疗健康问答文本语义分类模型 [J ] . 数据分析与知识发现 , 2025 , 9 ( 2 ): 26 - 38 .

CHEN W J , YANG L , LI J L . Semantic classification model for healthcare Q & A texts with embedded intent recognition [J ] . Data Analysis and Knowledge Discovery , 2025 , 9 ( 2 ): 26 - 38 .

CHEN T , KORNBLITH S , NOROUZI M , et al . A simple framework for contrastive learning of visual representations [J ] . arXiv preprint , 2020 : 2002 .05709.

KHOSLA P , TETERWAK P , WANG C , et al . Supervised contrastive learning [C ] // Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020) . [ S.l. : s.n. ] , 2020 : 18661 - 18673 .

GAO T Y , YAO X C , CHEN D Q . SimCSE: simple contrastive learning of sentence embeddings [J ] . arXiv preprint , 2021 : 2104 .08821.

WU X , GAO C C , ZANG L J , et al . ESimCSE: enhanced sample building method for contrastive learning of unsupervised sentence embedding [J ] . arXiv preprint , 2021 : 2109 .04380.

BAHDANAU D , CHO K , BENGIO Y . Neural machine translation by jointly learning to align and translate [J ] . arXiv preprint , 2014 : 1409 .0473.

YANG Z C , YANG D Y , DYER C , et al . Hierarchical attention networks for document classification [C ] // Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (USAACL) . [ S.l. : s.n. ] , 2016 : 1480 - 1489 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [J ] . arXiv preprint , 2017 : 1706 .03762.

DEVLIN J , CHANG M W , LEE K , et al . BERT: pre-training of deep bidirectional transformers for language understanding [J ] . arXiv preprint , 2018 : 1810 .04805.

TANG H , JI D H , LI C L , et al . Dependency graph enhanced dual-transformer structure for aspect-based sentiment classification [C ] // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (USAACL) . [ S.l. : s.n. ] , 2020 : 6578 - 6588 .

HADSELL R , CHOPRA S , LECUN Y . Dimensionality reduction by learning an invariant mapping [C ] // Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) . Piscataway : IEEE Press , 2006 : 1735 - 1742 .

CHOPRA S , HADSELL R , LECUN Y . Learning a similarity metric discriminatively, with application to face verification [C ] // Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) . Piscataway : IEEE Press , 2005 : 539 - 546 .

HE K M , FAN H Q , WU Y X , et al . Momentum contrast for unsupervised visual representation learning [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 9729 - 9738 .

GUNEL B , DU J , CONNEAU A , et al . Supervised contrastive learning for pre-trained language model fine-tuning [J ] . arXiv preprint arXiv , 2020 : 2011 .01403.

KHOSLA P , TETERWAK P , WANG C , et al . Supervised contrastive learning [J ] . Advances in Neural Information Processing Systems , 2020 , 33 : 18661 - 18673 .

LI B H , ZHOU H , HE J X , et al . On the sentence embeddings from pre-trained language models [C ] // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . [ S.l. : s.n. ] , 2020 : 9119 - 9130 .

WU Z F , WANG S N , GU J T , et al . CLEAR: contrastive learning for sentence representation [J ] . arXiv preprint , 2020 : 2012 .15466.

FANG H C , WANG S C , ZHOU M , et al . CERT: contrastive self-supervised learning for language understanding [J ] . arXiv preprint , 2020 : 2005 .12766.

CARLSSON F , GOGOULOU E , YLIPÄÄ E , et al . Semantic re-tuning with contrastive tension [C ] // Proceedings of the 2021 International Conference on Leaening Representations(ICLR) . [ S.l. : s.n. ] , 2021 : 1 - 21 .

MENG Y , XIONG C Y , BAJAJ P , et al . COCO-LM: correcting and contrasting text sequences for language model pretraining [J ] . arXiv preprint , 2021 : 2102 .08473.

WANG Y S , WU A , NEUBIG G . English contrastive learning can learn universal cross-lingual sentence embeddings [J ] . arXiv preprint , 2022 : 2211 .06127.

XIONG L E , XIONG C Y , LI Y , et al . Approximate nearest neighbor negative contrastive learning for dense text retrieval [J ] . arXiv preprint , 2020 : 2007 .00808.

LEE K , CHANG M W , TOUTANOVA K . Latent retrieval for weakly supervised open domain question answering [C ] // Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (USAACL) . [ S.l. : s.n. ] , 2019 : 6086 - 6096 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

双分支引导对比学习的无监督行人重识别

基于GNN与注意力机制的文本分类模型

数据增强的多模式时间感知序列推荐

基于对比学习的社交媒体地理位置预测方法

多模型融合的客服工单文本分类方法的研究与实现