Lightweight face image restoration algorithm based on multi-scale feature fusion

ZHAO Xiao; ZHAO Ziyi; YANG Chen

doi:10.11959/j.issn.1000-0801.2024183

您当前的位置：

首页 >

文章列表页 >

Lightweight face image restoration algorithm based on multi-scale feature fusion

Research and Development | 更新时间：2024-09-23

- Lightweight face image restoration algorithm based on multi-scale feature fusion
- Telecommunications Science Vol. 40, Issue 8, Pages: 42-51(2024)
- 作者机构：
  
  陕西科技大学电子信息与人工智能学院，陕西西安 710021
- 作者简介：
- 基金信息：
  
  The National Natural Science Foundation of China(61971272;61601271)
- DOI：10.11959/j.issn.1000-0801.2024183
  CLC： TP391
- Received：17 April 2024，
  
  Revised：2024-06-20，
  
  Published：20 August 2024
- 稿件说明：
移动端阅览
赵晓,赵子怡,杨晨.基于多尺度特征融合的轻量化人脸图像修复算法[J].电信科学,2024,40(08):42-51.

ZHAO Xiao,ZHAO Ziyi,YANG Chen.Lightweight face image restoration algorithm based on multi-scale feature fusion[J].Telecommunications Science,2024,40(08):42-51.
赵晓,赵子怡,杨晨.基于多尺度特征融合的轻量化人脸图像修复算法[J].电信科学,2024,40(08):42-51. DOI： 10.11959/j.issn.1000-0801.2024183.

ZHAO Xiao,ZHAO Ziyi,YANG Chen.Lightweight face image restoration algorithm based on multi-scale feature fusion[J].Telecommunications Science,2024,40(08):42-51. DOI： 10.11959/j.issn.1000-0801.2024183.

摘要

针对当前遮挡的人脸图像修复中修复图像质量差和模型参数量大的问题，提出了一种基于多尺度特征融合的改进U-Net的轻量化人脸图像修复模型——LM-UNET。首先，使用深度可分离卷积替换原有卷积，增强模型对不同通道和上下文信息的特征表达能力，实现模型轻量化；其次，在跳跃连接中设计了多尺度特征注意力融合模块，充分融合不同尺度特征的信息，内嵌残差块减少特征间语义差距，提高模型修复准确率；最后，引入了位置注意力模块，增强人脸图像的显著信息，提升模型对人脸位置像素信息的有效提取能力。在基于CK+数据集生成的遮挡人脸数据集MFD上对该算法进行训练、验证和测试，修复后的图像的峰值信噪比（PSNR）达到30.49 dB，结构相似性（SSIM）达到96.85%，与其他模型的对比实验结果表明，该模型对存在遮挡的人脸修复图像质量和视觉效果更好。

Abstract

Aiming at the problems of poor quality of restored images and large number of model parameters in the current occluded face image restoration

a lightweight face image restoration model based on multi-scale feature fusion with improved U-Net

LM-UNET

was proposed. Firstly

the original convolution was replaced by a depthwise separable convolution to enhance the feature expression ability of the model for different channels and contextual information. Secondly

a multi-scale feature attention fusion module was designed in the jump connection to fully fuse the information of different scale features

and the embedded residual block reduced the semantic gap between features to improve the repair accuracy of the model. Finally

a positional attention module was introduced to enhance the salient information of the face image

and improve the model’s effective extraction ability of facial positional pixel information of the model. The algorithm was trained

validated and tested on the occluded face dataset MFD generated based on the CK+ dataset

and the PSNR of the repaired image reached 30.49 dB and SSIM reached 96.85%. The experimental results of comparing the model with the other models show that the model has better image quality and visual effect for restoration of the face in the presence of occlusion.

关键词

Keywords

references

BALLESTER C , BERTALMIO M , CASELLES V , et al . Filling-in by joint interpolation of vector fields and gray levels [J ] . IEEE Transactions on Image Processing , 2001 , 10 ( 8 ): 1200 - 1211 .

CRIMINISI A , PÉREZ P , TOYAMA K . Region filling and object removal by exemplar-based image inpainting [J ] . IEEE Transactions on Image Processing , 2004 , 13 ( 9 ): 1200 - 1212 .

曹燕 , 金炜 , 符冉迪 . 基于加权优先级和分类匹配的图像修复方法 [J ] . 电信科学 , 2017 , 33 ( 4 ): 94 - 100 .

CAO Y , JIN W , FU R D . Image restoration method based on weighted prioritization and classification matching [J ] . Telecommunication Science , 2017 , 33 ( 4 ): 94 - 100 .

李悦 , 钱亚冠 , 关晓惠 , 等 . 面向人脸识别的口罩区域修复算法 [J ] . 电信科学 , 2021 , 37 ( 8 ): 66 - 76 .

LI Y , QIAN Y G , GUAN X H , et al . A mask region repair algorithm for face recognition [J ] . Telecommunication Science , 2021 , 37 ( 8 ): 66 - 76 .

陈北京 , 王鹏 , 喻乐延 , 等 . 注意力融合双流特征的局部GAN生成人脸检测算法 [J ] . 东南大学学报（自然科学版） , 2023 , 53 ( 3 ): 543 - 551 .

CHEN B J , WANG P , YU L Y , et al . Localized GAN-generated face detection algorithm for attention-fused dual-stream features [J ] . Journal of Southeast University (Natural Science Edition) , 2023 , 53 ( 3 ): 543 - 551 .

卢奇 . 局部信息缺失的RGB-D人脸修复及识别 [D ] . 成都 : 西南交通大学 , 2022 .

LU Q . RGB-D face restoration and recognition with localized information loss [D ] . Chengdu : Southwest Jiaotong University , 2022 .

ZHENG C , CHAMT J , CAI J . TFill: image completion via a transformer-based architecture [J ] . 2021 . arXiv: 2104.00845 , 2021.

RONNEBERGER O , FISCHER P , BROX T . U-Net: convolutional networks for biomedical image segmentation [C ] // Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention . Cham : Springer , 2015 : 234 - 241 .

LIU H Y , JIANG B , XIAO Y , et al . Coherent semantic attention for image inpainting [C ] // Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE , 2019 : 4170 - 4179 .

ZENG Y H , FU J L , CHAO H Y , et al . Learning pyramid context encoder network for high-quality image inpainting [C ] // Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE , 2019 : 1486 - 1494 .

罗仕胜 , 陈明举 , 陈柳 , 等 . 基于面部特征点的人脸图像修复网络 [J ] . 中国科技论文 , 2021 , 16 ( 7 ): 729 - 734, 742 .

LUO S S , CHEN M J , CHEN L , et al . Face image restoration network based on facial feature points [J ] . Chinese Science and Technology Paper , 2021 , 16 ( 7 ): 729 - 734, 742 .

QIN J , BAI H , ZHAO Y . Multi-scale attention network for image inpainting [J ] . Computer Vision and Image Understanding , 2021 , 204 : 103155 .

WANG N , ZHANG Y , ZHANG L . Dynamic selection network for image inpainting [J ] . IEEE Transactions on Image Processing , 2021 , 30 : 1784 - 1798 .

LIAO L , XIAO J , WANG Z , et al . Image inpainting guided by coherence priors of semantics and textures [C ] // Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition . Washington : IEEE Computer Society , 2021 : 6539 - 6548 .

GUO Y , LI Y , WANG L , et al . Depthwise convolution is all you need for learning multiple visual domains [C ] // Proceedings of the AAAI Conference on Artificial Intelligence . Palo Alto : AAAI Press , 2019 : 8368 - 8375 .

YU J , LIN Z , YANG J , et al . Free-form image inpainting with gated convolution [C ] // Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision . Piscataway : IEEE Press , 2019 : 4470 - 4479 .

ZHU M , HE D , LI X , et al . Image inpainting by end-to-end cascaded refinement with mask awareness [J ] . IEEE Transactions on Image Processing , 2021 , 30 : 4855 - 4866 .

LI J , WANG N , ZHANG L , et al . Recurrent feature reasoning for image inpainting [C ] // Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Piscataway : IEEE Press , 2020 : 7757 - 7765 .

ZENG Y H , FU J L , CHAO H Y , et al . Aggregated contextual transformations for high-resolution image inpainting [J ] . IEEE Transactions on Visualization and Computer Graphics , 2023 , 29 ( 7 ): 3266 - 3280 .

WANG F P , LI W L , LIU Y , et al . Face inpainting algorithm combining edge information with gated convolution [J ] . Journal of Frontiers of Computer Science and Technology , 2021 , 15 ( 1 ): 150 - 162 .

Views

547

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A speech enhancement method based on depthwise separable convolution and MambaUNet

Related Author

MENG Xiangcai

ZHUANG Yunpeng

ZHONG Qiang

OU Shifeng

Related Institution

Intelligent Control Department of Yantai Vocational College

Yantai Hongwu Electromechanical Technology Co., Ltd.

College of Physics and Electronic Information, Yantai University

⁰