Deep learning-based prediction of multi-level just noticeable distortion

Haifeng XU; Hongkui WANG; Haibing YIN; Chuqiao CHEN

doi:10.11959/j.issn.1000-0801.2024015

您当前的位置：

首页 >

文章列表页 >

Deep learning-based prediction of multi-level just noticeable distortion

Research and Development | 更新时间：2024-06-05

- Deep learning-based prediction of multi-level just noticeable distortion
- Telecommunications Science Vol. 40, Issue 1, Pages: 35-47(2024)
- 作者机构：
  
  1. 杭州电子科技大学通信工程学院，浙江杭州 310018
  2. 杭州电子科技大学网安学院，浙江杭州 310018
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.1000-0801.2024015
  CLC： TN919
- Published Online：2024-01，
  
  Published：20 January 2024
- 稿件说明：
移动端阅览
Haifeng XU, Hongkui WANG, Haibing YIN, et al. Deep learning-based prediction of multi-level just noticeable distortion[J]. Telecommunications science, 2024, 40(1): 35-47.
DOI：

Haifeng XU, Hongkui WANG, Haibing YIN, et al. Deep learning-based prediction of multi-level just noticeable distortion[J]. Telecommunications science, 2024, 40(1): 35-47. DOI： 10.11959/j.issn.1000-0801.2024015.

摘要

视觉恰可察觉失真（just noticeable distortion，JND）直接反映人眼视觉系统对视觉信号噪声的敏感程度，广泛应用于图像和视频处理领域。针对视频 JND 阈值的多层级预测问题，将其转化为用户满意率（satisfied user ratio，SUR）曲线的预测问题，并提出一种基于特征融合的SUR曲线预测模型。该模型主要分为关键帧选择模块、特征提取和融合模块以及SUR分数回归模块。在关键帧选择模块，根据视觉感知机制，提出空时域感知复杂度并以此作为视频关键帧判决指标。在特征提取和融合模块，基于密集残差块（dense residual block，RDB）提出多尺度密集残差网络实现图像特征提取和多尺度融合。实验结果表明，所提出的SUR曲线预测模型在JND阈值预测精度方面整体优于现有模型，且在运行效率上平均降低8.1%的时间成本。同时，该模型还可以用于预测其他层级JND阈值，可直接应用于视频多层级感知编码优化。

Abstract

Visual just noticeable distortion (JND) directly reflects the sensitivity of the human visual system to visual signal noise

and is widely used in image and video processing.Aiming at the multilevel prediction problem of video JND threshold

it was transformed into the prediction problem of satisfied user ratio (SUR) curve

and a feature fusion-based SUR curve prediction model was proposed.The model was mainly divided into key frame extraction module

feature extraction and fusion module

and SUR score regression module.In the key frame extraction module

according to the visual perception mechanism

the spatial-temporal domain perception complexity was proposed and used as the video key frame judgment index.In the feature extraction and fusion module

a multi-scale dense residual network was proposed based on dense residual block (RDB) to realize image feature extraction and multi-scale fusion.The experimental results show that the proposed SUR curve prediction model is overall better than the existing models in terms of JND prediction accuracy and reduces the time cost by 8.1% on average in terms of operational efficiency.Meanwhile

the model can also be used to predict other layers of JND thresholds

which can be directly applied to video multilevel perceptual coding optimization.

关键词

Keywords

references

LIN W S , GHINEA G . Progress and opportunities in modelling just-noticeable difference (JND) for multimedia [J ] . IEEE Transactions on Multimedia , 2021 ( 24 ): 3706 - 3721 .

WU J , LI L , DONG W , et al . Enhanced just noticeable difference model for images with pattern complexity [J ] . IEEE Transactions on Image Processing , 2017 , 26 ( 6 ): 2682 - 2693 .

WANG H , YU L , LIANG J , et al . Hierarchical predictive coding-based JND estimation for image compression [J ] . IEEE Transactions on Image Processing , 2020 ( 30 ): 487 - 500 .

BAE S H , KIM M . A novel generalized DCT-based JND profile based on an elaborate CM-JND model for variable block-sized transforms in monochrome images [J ] . IEEE Transactions on Image Processing , 2014 , 23 ( 8 ): 3227 - 3240 .

骆琼华 , 王鸿奎 , 殷海兵 , 等 . 基于熵掩蔽的 DCT 域恰可察觉失真模型 [J ] . 电信科学 , 2023 , 39 ( 2 ): 59 - 70 .

LUO Q H , WANG H K , YIN H B , et al . Just noticeable distortion model based on entropy masking in DCT domain [J ] . Telecommunications Science , 2023 , 39 ( 2 ): 59 - 70 .

邢亚芬 , 殷海兵 , 王鸿奎 , 等 . 基于视频时域感知特性的恰可察觉失真模型 [J ] . 电信科学 , 2022 , 38 ( 2 ): 92 - 102 .

XING Y F , YIN H B , WANG H K , et al . Video temporal perception characteristics based just noticeable difference model [J ] . Telecommunications Science , 2022 , 38 ( 2 ): 92 - 102 .

JIN L N , LIN J Y , HU S D , et al . Statistical study on perceived JPEG image quality via MCL-JCI dataset construction and analysis [J ] . Electronic Imaging , 2016 , 28 ( 13 ): 1 - 9 .

WANG H Q , GAN W H , HU S D , et al . MCL-JCV:a JND-based H.264/AVC video quality assessment dataset [C ] // Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP) . Piscataway:IEEE Press , 2016 : 1509 - 1513 .

WANG H Q , KATSAVOUNIDIS I , ZHOU J T , et al . VideoSet:a large-scale compressed video quality dataset based on JND measurement [J ] . Journal of Visual Communication and Image Representation , 2017 ( 46 ): 292 - 302 .

HUANG Q , WANG H Q , LIM S C , et al . Measure and prediction of HEVC perceptually lossy/lossless boundary QP values [C ] // Proceedings of the 2017 Data Compression Conference (DCC) . Piscataway:IEEE Press , 2017 : 42 - 51 .

ZHANG X , YANG C , WANG H , et al . Satisfied-user-ratio modeling for compressed video [J ] . IEEE Transactions on Image Processing , 2020 ( 29 ): 3777 - 3789 .

ZHANG Y , LIU H H , YANG Y , et al . Deep learning based just noticeable difference and perceptual quality prediction models for compressed video [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2022 , 32 ( 3 ): 1197 - 1212 .

LIN J Y , JIN L N , HU S D , et al . Experimental design and analysis of JND test on coded image/video [C ] // SPIE Optical Engineering + Applications.Applications of Digital Image Processing XXXVIII .[S.l.:s.n. ] , 2015 : 324 - 334 .

FAN C L , ZHANG Y , HAMZAOUI R , et al . Learning-based satisfied user ratio prediction for symmetrically and asymmetrically compressed stereoscopic images [J ] . IEEE MultiMedia , 2021 , 28 ( 3 ): 8 - 20 .

LIU X H , CHEN Z H , WANG X , et al . JND-pano:database for just noticeable difference of JPEG compressed panoramic images [C ] // Pacific Rim Conference on Multimedia . Berlin:Springer , 2018 : 458 - 468 .

SHEN X L , NI Z K , YANG W H , et al . A JND dataset based on VVC compressed images [C ] // Proceedings of the 2020 IEEE International Conference on Multimedia ＆ Expo Workshops (ICMEW) . Piscataway:IEEE Press , 2020 : 1 - 6 .

LIN H H , CHEN G G , JENADELEH M , et al . Large-scale crowdsourced subjective assessment of picturewise just noticeable difference [J ] . IEEE Transactions on Circuits and Systems for Video Technology , 2022 , 32 ( 9 ): 5859 - 5873 .

TIAN T , WANG H L , ZUO L X , et al . Just noticeable difference level prediction for perceptual image compression [J ] . IEEE Transactions on Broadcasting , 2020 , 66 ( 3 ): 690 - 700 .

LIU H H , ZHANG Y , ZHANG H , et al . Deep learning based picture-wise just noticeable distortion prediction model for image compression [J ] . IEEE Transactions on Image Processing:a Publication of the IEEE Signal Processing Society , 2019 ( 29 ): 641 - 656 .

ITU-T P . 910.Subjective video quality assessment methods for multimedia applications [S ] . Geneva:ITU-T Publications , 2022 .

ZHANG Y , TIAN Y , KONG Y , et al . Residual dense network for image super-resolution [C ] // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2018 : 2472 - 2481 .

PARK T , LIU M Y , WANG T C , et al . Semantic image synthesis with spatially-adaptive normalization [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway:IEEE Press , 2019 : 2337 - 2346 .

BAO L , YANG Z , WANG S , et al . Real image denoising based on multi-scale residual dense block and cascaded U-Net with block-connection [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops . Piscataway:IEEE Press , 2020 : 448 - 449 .

WANG Q , WU B , ZHU P , et al . ECA-Net:Efficient channel attention for deep convolutional neural networks [C ] // Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . Piscataway:IEEE Press , 2020 : 11534 - 11542 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

GMTBLC: a deep learning-based bi-modal network traffic classification method

Research on antenna dip angle recognition method based on improved YOLOv5

Mixed signal modulation recognition method based on temporal depth residual shrinkage network

Research on fault prediction of computer network nodes driven by log information

Self-attention mechanism-based CSI eigenvector feedback for massive MIMO

Related Author

WEI Debin

JIANG Qinlong

WEN Jinglong

WANG Xinrui

ZHANG Ning

PAN Feng

GENG Lujing

CHEN Zuhao

Related Institution

School of Information Engineering, Dalian University

China Mobile Group Design Institute Co., Ltd.

China Mobile Group Co., Ltd.

The 63rd Research Institute, National University of Defense Technology

School of Electronic and Information Engineering, Nanjing University of Information Science and Technology

⁰