Self-supervised video temporal resampling method for machine vision

Liu Jianran; Ji Wen; Fu Zhe

doi:10.11959/j.issn.1000-0801.2026048

您当前的位置：

首页 >

文章列表页 >

Self-supervised video temporal resampling method for machine vision

Research and Development | 更新时间：2026-03-05

- Self-supervised video temporal resampling method for machine vision
- Telecommunications Science Vol. 42, Issue 2, Pages: 13-28(2026)
- 作者机构：
  
  1.中国科学院计算技术研究所处理器芯片全国重点实验室，北京 100190
  2.中国科学院大学，北京101408
  3.中科工业人工智能研究院，江苏南京 211135
  4.龙眼国科（北京）智能信息技术有限公司，北京 100010
  5.交控科技股份有限公司，北京 100071
- 作者简介：
- 基金信息：
  
  The National Key Research and Development Program of China(2024YFE0212000);China Postdoctoral Science Foundation(GZC20251091);The Beijing Natural Science Foundation(L221004);Jiangsu Provincial Key Laboratory of Industrial Artificial Intelligence
- DOI：10.11959/j.issn.1000-0801.2026048
  CLC： TP393;TN911.73
- Received：29 June 2025，
  
  Revised：2025-08-08，
  
  Accepted：25 August 2025，
  
  Published：20 February 2026
- 稿件说明：
移动端阅览
刘建然,纪雯,付哲.面向机器视觉的自监督视频时域重采样方法[J].电信科学,2026,42(02):13-28.

Liu Jianran,Ji Wen,Fu Zhe.Self-supervised video temporal resampling method for machine vision[J].Telecommunications Science,2026,42(02):13-28.
刘建然,纪雯,付哲.面向机器视觉的自监督视频时域重采样方法[J].电信科学,2026,42(02):13-28. DOI： 10.11959/j.issn.1000-0801.2026048.

Liu Jianran,Ji Wen,Fu Zhe.Self-supervised video temporal resampling method for machine vision[J].Telecommunications Science,2026,42(02):13-28. DOI： 10.11959/j.issn.1000-0801.2026048.

摘要

针对视频时域重采样中帧间内容非线性变化导致的语义冗余问题，提出一种基于自监督特征嵌入和聚类的视频时域自适应重采样方法。所提方法通过预训练的ResNet-18提取视频帧特征并微调，利用自监督度量学习构建帧间相似性度量，采用余弦相似度衡量相邻帧相似性，并设计损失函数使同一视频序列内的帧在嵌入空间中流形分布光滑，同时抑制不同视频帧间相似度。之后，对嵌入后的帧特征进行基于流形等分点的时序数据聚类，并确保视频首尾完整。重采样后的视频序列经H.266/VVC编码，解码端结合插帧网络重建原始帧。实验表明，该方法在BDmAP和Pareto mAP指标上平均提高约2.3%和19.4%，且计算开销满足实时处理需求，有效平衡压缩效率、视觉任务精度与零样本兼容性，为机器视觉场景下的视频传输提供新思路。

Abstract

To address the semantic redundancy caused by nonlinear inter-frame content changes during video temporal resampling

an adaptive resampling method based on self-supervised feature embedding and clustering was proposed. In this method

features from video frames were extracted using a pre-trained ResNet-18

which was subsequently fine-tuned. Self-supervised metric learning was employed to construct an inter-frame similarity measure

and cosine similarity was used to gauge the resemblance between adjacent frames. A loss function was designed to ensure a smooth manifold distribution for frames from the same video sequence in the embedding space

while similarity between frames from different videos was simultaneously suppressed. Subsequently

the embedded frame features were subjected to temporal data clustering based on manifold bisection points

and the integrity of the video’s first and last frames was ensured. The sampled video sequence was then encoded using H.266/VVC

and the original frames were reconstructed at the decoding end by a frame interpolation network. Through experiments

it was demonstrated that average improvements of approximately 2.3% and 19.4% were achieved on the BDmAP and Pareto mAP metrics

respectively. Furthermore

the computational overhead was found to meet the demands of real-time processing. The proposed approach effectively balances compression efficiency

visual task accuracy

and zero-shot compatibility. It offers a novel solution for video transmission in machine vision scenarios.

关键词

Keywords

references

Guan Y C , Liao H C , Li Z N , et al . World models for autonomous driving: an initial survey [J ] . IEEE Transactions on Intelligent Vehicles , 2024 , 99 : 1 - 17 .

Ji W , Xu J C , Qiao H X , et al . Visual IoT: enabling Internet of things visualization in smart cities [J ] . IEEE Network , 2019 , 33 ( 2 ): 102 - 110 .

Dui H Y , Zhang S R , Liu M , et al . IoT-enabled real-time traffic monitoring and control management for intelligent transportation systems [J ] . IEEE Internet of Things Journal , 2024 , 11 ( 9 ): 15842 - 15854 .

Kim A , Woo S T , Park M , et al . Deep learning-guided video compression for machine vision tasks [J ] . EURASIP Journal on Image and Video Processing , 2024 ( 1 ): 32 .

熊皓萱 , 徐媛媛 , 朱琨 . 面向机器视觉的VVC帧内编码算法 [J ] . 信号处理 , 2025 , 41 ( 2 ): 350 - 358 .

Xiong H X , Xu Y Y , Zhu K . VVC intra-coding scheme for machines [J ] . Journal of Signal Processing , 2025 , 41 ( 2 ): 350 - 358 .

Ge X T , Luo J X , Zhang X J , et al . Task-aware encoder control for deep video compression [C ] // Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2024 : 26036 - 26045 .

Schiappa M C , Rawat Y S , Shah M . Self-supervised learning for videos: a survey [J ] . ACM Computing Surveys , 2023 , 55 ( 13 s): 1 - 37 .

Hua H , Tang Y L , Xu C L , et al . V2Xum-LLM: cross-modal video summarization with temporal prompt instruction tuning [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . Palo Alto : AAAI Press , 2025 , 39 ( 4 ): 3599 - 3607 .

Zhao Y , Ye M , Ji L P , et al . Temporal adaptive learned surveillance video compression [J ] . IEEE Transactions on Broadcasting , 2025 , 71 ( 1 ): 142 - 153 .

Tian J R , Lin Z X , Dai Y , et al . Keyframes selection from multiscene videos for stress detection [J ] . Information Processing & Management , 2025 , 62 ( 5 ): 104215 .

Zeng J H , Liang G , Ma Y X , et al . Pornographic video detection based on semantic and image enhancement [J ] . The Computer Journal , 2024 , 67 ( 10 ): 3009 - 3019 .

Lee J , Hwang K I . YOLO with adaptive frame control for real-time object detection applications [J ] . Multimedia Tools and Applications , 2022 , 81 ( 25 ): 36375 - 36396 .

Duan Y Q , Lu J W , Feng J J , et al . Deep localized metric learning [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2018 , 28 ( 10 ): 2644 - 2656 .

Roth K , Milbich T , Sinha S , et al . Revisiting training strategies and generalization performance in deep metric learning [EB ] . 2020 .

Zhang D Y , Li Y M , Zhang Z F . Deep metric learning with spherical embedding [EB ] . 2020 .

Fu Z R , Li Y , Mao Z D , et al . Deep metric learning with self-supervised ranking [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . Palo Alto : AAAI Press , 2021 , 35 ( 2 ): 1370 - 1378 .

Jia X H , Han K , Zhu Y K , et al . Joint representation learning and novel category discovery on single- and multi-modal data [C ] // Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2022 : 590 - 599 .

Wu Z R , Xiong Y J , Yu S X , et al . Unsupervised feature learning via non-parametric instance discrimination [C ] // Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2018 : 3733 - 3742 .

Tagiew R , Klasek P , Tilly R , et al . OSDaR23: open sensor data for rail 2023 [C ] // Proceedings of the 2023 8th International Conference on Robotics and Automation Engineering (ICRAE) . Piscataway : IEEE Press , 2024 : 270 - 276 .

Kay W , Carreira J , Simonyan K , et al . The Kinetics human action video dataset [EB ] . 2017 .

Wieckowski A , Brandenburg J , Hinz T , et al . Vvenc: an open and optimized VVC encoder implementation [C ] // Proceedings of the 2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) . Piscataway : IEEE Press , 2021 : 1 - 2 .

Wieckowski A , Hege G , Bartnik C , et al . Towards a live software decoder implementation for the upcoming versatile video coding (VVC) codec [C ] // Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP) . Piscataway : IEEE Press , 2020 : 3124 - 3128 .

Zhang G Z , Zhu Y H , Wang H N , et al . Extracting motion and appearance via inter-frame attention for efficient video frame interpolation [C ] // Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Piscataway : IEEE Press , 2023 : 5682 - 5692 .

Xue T F , Chen B A , Wu J J , et al . Video enhancement with task-oriented flow [J ] . International Journal of Computer Vision , 2019 , 127 ( 8 ): 1106 - 1125 .

Merz G , Liu Y C , Burke C J , et al . Detection, instance segmentation, and classification for astronomical surveys with deep learning (deepdisc): detectron2 implementation and demonstration with Hyper Suprime-Cam data [J ] . Monthly Notices of the Royal Astronomical Society , 2023 , 526 ( 1 ): 1122 - 1137 .

Girshick R . Fast R-CNN [C ] // Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV) . Piscataway : IEEE Press , 2016 : 1440 - 1448 .

Lin T Y , Maire M , Belongie S , et al . Microsoft COCO: common objects in context [C ] // Proceedings of the 2014 European Conference on Computer Vision (ECCV) . Cham : Springer , 2014 : 740 - 755 .

Perera A , Adzic V , Kalva H , et al . Comparative analysis of VCM and AhG8 for machine vision applications [EB ] . 2025 .

Pont-Tuset J , Perazzi F , CAELLES S , et al . The 2017 Davis challenge on video object segmentation [EB ] . 2017 .

Zhou C . Yolact++ better real-time instance segmentation [M ] . Davis : University of California, Davis , 2020 .

Views

4740

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Machine vision technology,standardization,and application in safety industry

Related Author

LIU Jianran

JI Wen

FU Zhe

Yuan ZHANG

Huifen WANG

Hanming WANG

Related Institution

State Key Lab of Processors, Institute of Computing Technology, CAS

Longan Guoke (Beijing) Intelligent Information Technology Co., Ltd.

⁰