“Knowledge Distillation” Explanation and Implementation

Abstract

A clear and comprehensive explanation of Knowledge Distillation is presented in this video. In addition, the approach presented in the paper “Distilling the Knowledge in a Neural Network” is implemented from scratch using PyTorch.

read the “Distilling the Knowledge in a Neural Network” paper: https://arxiv.org/abs/1503.02531