基于自监督对抗学习的多尺度知识蒸馏方法
CSTR:
作者:
作者单位:

1.安徽理工大学;2.西安电子科技大学

作者简介:

通讯作者:

中图分类号:

TP391

基金项目:

安徽理工大学医学专项培育项目(No. YZ2023H2C005); 安徽理工大学环境友好材料与职业健康研究院研发专项基金资助项目(ALW2021YF04); 国家自然科学基金项目(62102003); 安徽理工大学研究生创新基金(No.2023cx2139)


Multi-scale Knowledge Distillation Method Based On Self-supervised Adversarial Learning
Author:
Affiliation:

1.Anhui University Science and Technology;2.Xidian University

Fund Project:

Medical Special Cultivation Project of Anhui University of Science and Technology(No.YZ2023H2C005); Funded by Research Foundation of the Institute of Environment-friendly Materials and Occupational Health (Wuhu), Anhui University of Science and Technology(ALW2021YF04); National Natural Science Foundation of China(62102003); Anhui University of Science and Technology Graduate Innovation Fund(No.2023cx2139)

  • 摘要
  • |
  • 图/表
  • |
  • 访问统计
  • |
  • 参考文献
  • |
  • 相似文献
  • |
  • 引证文献
  • |
  • 资源附件
  • |
  • 文章评论
    摘要:

    针对离线知识蒸馏中因教师与学生之间规模差距过大,知识难以有效传递,导致学生性能不佳的问题,提出了一种基于自监督对抗学习的多尺度知识蒸馏方法 (SAMKD)。利用自监督和对抗学习来进一步开发中间多尺度特征与网络末端输出特征 logits 的潜力。首先,引入多角度几何变换图像监督网络学习模型;其次,设计多分支辅助网络来提取主干网络的多尺度特征,获得图像的 logits 输出;最后,利用生成对抗网络的二元博弈思想进行多阶段对抗训练,通过该对抗训练能够将多层次的知识通过蒸馏方法充分传递。通过实验分析,在三个具有挑战性的公开数据集 CIFAR-10、CIFAR-100 和 Tiny-ImageNet 上进行充分实验,所提方法与其他先进知识蒸馏方法的比较中展现了强大的竞争力。

    Abstract:

    This paper proposes a multi-scale knowledge distillation method based on self-supervised adversarial learning, named SAMKD, to address the challenge of ineffective knowledge transfer caused by the substantial scale difference between teachers and students in offline knowledge distillation. SAMKD leverages self-supervised and adversarial learning to further exploit the potential of intermediate multi-scale features and logits at the network’s output. Firstly, a supervised network learning model for multi-angle geometrically transformed images is introduced. Secondly, a multi-branch auxiliary network is designed to extract multi-scale features from the backbone network and obtain image logits. Finally, the binary game concept from generative adversarial networks is utilized for multi-stage adversarial training, enabling comprehensive knowledge transfer through the distillation process. Experimental analyses conducted on three challenging public datasets, CIFAR-10, CIFAR-100, and Tiny-ImageNet, demonstrate the strong competitiveness of the proposed method against other state-of-the-art knowledge distillation methods.

    参考文献
    相似文献
    引证文献
引用本文
分享
文章指标
  • 点击次数:
  • 下载次数:
  • HTML阅读次数:
  • 引用次数:
历史
  • 收稿日期:2024-03-20
  • 最后修改日期:2024-07-10
  • 录用日期:2024-07-12
  • 在线发布日期: 2024-07-25
  • 出版日期:
文章二维码