Semantic embedding via joint loss function and composite contrastive learning

GAO Xiaoxin; LU Yao; KONG Xiangmao; LIU Yuxi; DENG Wei; YANG Songhao

doi:10.11959/j.issn.1000-0801.2025142

您当前的位置：

首页 >

文章列表页 >

Semantic embedding via joint loss function and composite contrastive learning

Research and Development | 更新时间：2025-08-07

- Semantic embedding via joint loss function and composite contrastive learning
- Telecommunications Science Vol. 41, Issue 7, Pages: 96-107(2025)
- 作者机构：
  
  1.北京中电普华信息技术有限公司，北京 102209
  2.华北电力大学控制与计算机工程学院，北京 102206
- 作者简介：
- 基金信息：
  
  The Science and Technology Project of State Grid Corporation of China(5700-202318834A-4-2-KJ)
- DOI：10.11959/j.issn.1000-0801.2025142
  CLC： TP183
- Received：18 November 2024，
  
  Revised：2025-05-07，
  
  Published：20 July 2025
- 稿件说明：
移动端阅览
高晓欣,陆谣,孔祥茂等.基于联合损失函数和组合对比学习的语义嵌入方法[J].电信科学,2025,41(07):96-107.

GAO Xiaoxin,LU Yao,KONG Xiangmao,et al.Semantic embedding via joint loss function and composite contrastive learning[J].Telecommunications Science,2025,41(07):96-107.
高晓欣,陆谣,孔祥茂等.基于联合损失函数和组合对比学习的语义嵌入方法[J].电信科学,2025,41(07):96-107. DOI： 10.11959/j.issn.1000-0801.2025142.

GAO Xiaoxin,LU Yao,KONG Xiangmao,et al.Semantic embedding via joint loss function and composite contrastive learning[J].Telecommunications Science,2025,41(07):96-107. DOI： 10.11959/j.issn.1000-0801.2025142.

摘要

对比学习在语义嵌入中表现出色，能够通过捕捉数据样本间的关系提升模型的表示能力。然而，其效果主要受正样本构建和目标函数选择的影响。正样本需要精心设计，以确保模型能有效识别有意义的相似性并减少噪声干扰。为此，提出一种新方法，通过拆分、编码、聚合和投射文本来构建正样本。文本被分解为片段，编码用于提取语义内容，聚合用于突出关系，最终投射到适合学习的语义空间。此外，设计了两种监督损失函数，与标准对比损失相辅相成，以增强语义空间的区分性，从而提升模型辨别能力。实验结果表明，该方法在2个公开数据集和1个私有数据集上表现出色，显著提升了语义嵌入质量，解决了对比学习的核心挑战，并为其在自然语言处理领域的进一步应用奠定了基础。

Abstract

Contrastive learning has shown excellent performance in semantic embeddings by capturing relationships between data samples to enhance model representation. However

its effectiveness largely depends on constructing positive samples and selecting appropriate objective functions. Positive samples must be carefully designed to ensure the model can identify meaningful similarities while reducing noise. To address this

a novel method that constructed positive samples by splitting

encoding

aggregating

and projecting text was proposed. The text was broken into segments

encoded to extract semantic content

aggregated to highlight relationships

and projected into a semantic space optimized for learning. Additionally

two supervised loss functions were designed

complementing the standard contrastive loss

to enhance the discriminability of the semantic space and thereby improve the model’s discrimination ability. The experimental results show that this method performes well on two public datasets and one private dataset

significantly improving the quality of semantic embedding

solving the core challenges of contrastive learning

and laying the foundation for further applications in the field of natural language processing.

关键词

Keywords

references

欧阳梦妮 , 樊小超 , 帕力旦·吐尔逊 . 基于目标对齐和语义过滤的多模态情感分析 [J ] . 计算机技术与发展 , 2024 , 34 ( 6 ): 171 - 177 .

OUYANG M N , FAN X C , PALIDAN T S . Multimodal sentiment analysis based on target alignment and semantic filtering [J ] . Computer Technology and Development , 2024 , 34 ( 6 ): 171 - 177 .

吴加辉 , 加云岗 , 王志晓 , 等 . 基于深度学习的微博疫情舆情文本情感分析 [J ] . 计算机技术与发展 , 2024 , 34 ( 7 ): 175 - 183 .

WU J H , JIA Y G , WANG Z X , et al . Sentiment analysis of weibo epidemic public opinion text based on deep learning [J ] . Computer Technology and Development , 2024 , 34 ( 7 ): 175 - 183 .

陈鹏 , 邰彬 , 石英 , 等 . 基于知识图谱的电力设备缺陷问答系统研究 [J ] . 广西师范大学学报(自然科学版) , 2024 , 42 ( 16 ): 149 - 163 .

CHEN P , TAI B , SHI Y , et al . Research on defect Q&A system of power equipment based on knowledge graph [J ] . Journal of Guangxi Normal University (Natural Science Edition) , 2024 , 42 ( 16 ): 149 - 163 .

贾鹏 . 基于大语言模型的农业知识问答系统的研究与设计 [D ] . 秦皇岛 : 河北科技师范学院 , 2024 .

JIA P . Research and design of agricultural knowledge question answering system based on macrolanguage model [D ] . Qinhuangdao : Hebei Normal University of Science & Technology , 2024 .

李东亚 , 白涛 , 香慧敏 , 等 . 基于ERNIE及改进DPCNN的棉花病虫害问句意图识别 [J ] . 山东农业科学 , 2024 , 25 ( 16 ): 143 - 151 .

LI D Y , BAI T , XIANG H M , et al . Intention recognition of cotton pest questions based on ERNIE and improved DPCNN [J ] . Shandong Agricultural Sciences , 2024 , 25 ( 16 ): 143 - 151 .

谌文佳 , 杨琳 , 李金林 . 嵌入意图识别的医疗健康问答文本语义分类模型 [J ] . 数据分析与知识发现 , 2025 , 9 ( 2 ): 26 - 38 .

CHEN W J , YANG L , LI J L . Semantic classification model for healthcare Q & A texts with embedded intent recognition [J ] . Data Analysis and Knowledge Discovery , 2025 , 9 ( 2 ): 26 - 38 .

CHEN T , KORNBLITH S , NOROUZI M , et al . A simple framework for contrastive learning of visual representations [J ] . arXiv preprint , 2020 : 2002 .05709.

KHOSLA P , TETERWAK P , WANG C , et al . Supervised contrastive learning [C ] // Proceedings of the 34th Conference on Neural Information Processing Systems (NeurIPS 2020) . [ S.l. : s.n. ] , 2020 : 18661 - 18673 .

GAO T Y , YAO X C , CHEN D Q . SimCSE: simple contrastive learning of sentence embeddings [J ] . arXiv preprint , 2021 : 2104 .08821.

WU X , GAO C C , ZANG L J , et al . ESimCSE: enhanced sample building method for contrastive learning of unsupervised sentence embedding [J ] . arXiv preprint , 2021 : 2109 .04380.

BAHDANAU D , CHO K , BENGIO Y . Neural machine translation by jointly learning to align and translate [J ] . arXiv preprint , 2014 : 1409 .0473.

YANG Z C , YANG D Y , DYER C , et al . Hierarchical attention networks for document classification [C ] // Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (USAACL) . [ S.l. : s.n. ] , 2016 : 1480 - 1489 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [J ] . arXiv preprint , 2017 : 1706 .03762.

DEVLIN J , CHANG M W , LEE K , et al . BERT: pre-training of deep bidirectional transformers for language understanding [J ] . arXiv preprint , 2018 : 1810 .04805.

TANG H , JI D H , LI C L , et al . Dependency graph enhanced dual-transformer structure for aspect-based sentiment classification [C ] // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (USAACL) . [ S.l. : s.n. ] , 2020 : 6578 - 6588 .

HADSELL R , CHOPRA S , LECUN Y . Dimensionality reduction by learning an invariant mapping [C ] // Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) . Piscataway : IEEE Press , 2006 : 1735 - 1742 .

CHOPRA S , HADSELL R , LECUN Y . Learning a similarity metric discriminatively, with application to face verification [C ] // Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) . Piscataway : IEEE Press , 2005 : 539 - 546 .

HE K M , FAN H Q , WU Y X , et al . Momentum contrast for unsupervised visual representation learning [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2020 : 9729 - 9738 .

GUNEL B , DU J , CONNEAU A , et al . Supervised contrastive learning for pre-trained language model fine-tuning [J ] . arXiv preprint arXiv , 2020 : 2011 .01403.

KHOSLA P , TETERWAK P , WANG C , et al . Supervised contrastive learning [J ] . Advances in Neural Information Processing Systems , 2020 , 33 : 18661 - 18673 .

LI B H , ZHOU H , HE J X , et al . On the sentence embeddings from pre-trained language models [C ] // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . [ S.l. : s.n. ] , 2020 : 9119 - 9130 .

WU Z F , WANG S N , GU J T , et al . CLEAR: contrastive learning for sentence representation [J ] . arXiv preprint , 2020 : 2012 .15466.

FANG H C , WANG S C , ZHOU M , et al . CERT: contrastive self-supervised learning for language understanding [J ] . arXiv preprint , 2020 : 2005 .12766.

CARLSSON F , GOGOULOU E , YLIPÄÄ E , et al . Semantic re-tuning with contrastive tension [C ] // Proceedings of the 2021 International Conference on Leaening Representations(ICLR) . [ S.l. : s.n. ] , 2021 : 1 - 21 .

MENG Y , XIONG C Y , BAJAJ P , et al . COCO-LM: correcting and contrasting text sequences for language model pretraining [J ] . arXiv preprint , 2021 : 2102 .08473.

WANG Y S , WU A , NEUBIG G . English contrastive learning can learn universal cross-lingual sentence embeddings [J ] . arXiv preprint , 2022 : 2211 .06127.

XIONG L E , XIONG C Y , LI Y , et al . Approximate nearest neighbor negative contrastive learning for dense text retrieval [J ] . arXiv preprint , 2020 : 2007 .00808.

LEE K , CHANG M W , TOUTANOVA K . Latent retrieval for weakly supervised open domain question answering [C ] // Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (USAACL) . [ S.l. : s.n. ] , 2019 : 6086 - 6096 .

Views

567

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Cross-view contrastive learning and multi-channel hypergraph convolution for session-based recommendation

Node classification of complex network based on enhanced graph neural network and contrastive learning

Dual branch guided contrastive learning for unsupervised pedestrian re-identification

Text classification model based on GNN and attention mechanism

Multi-pattern time-aware sequential recommendation with data augmentation

Related Author

FANG Chi

SUI Xinyi

WANG Ruiqin

REN Yubin

TAN Yanli

WANG Yu

XU Peiling

LIANG Fengmei

Related Institution

School of Information Engineering, Huzhou University

Department of Electronic Engineering, Taiyuan Institute of Technology

School of Information and Communication Engineering, North University of China

Department of Electronic Information, Shanxi Institute of Economics and Business

College of Electronic Information Engineering, Taiyuan University of Technology

⁰