Article Source

• EfficientML.ai Lecture 9 - Knowledge Distillation (MIT 6.5940, Fall 2023)

Knowledge Distillation

• EfficientML.ai Lecture 9 - Knowledge Distillation (MIT 6.5940, Fall 2023)
• Instructor: Prof. Song Han
• Slides: https://efficientml.ai

Abstract

Large generative models (e.g., large language models, diffusion models) have shown remarkable performance, but they require a massive amount of computational resources. To make them more accessible, it is crucial to improve their efficiency.This course will introduce efficient AI computing techniques that enable powerful deep learning applications on resource-constrained devices. Topics include model compression, pruning, quantization, neural architecture search, distributed training, data/model parallelism, gradient compression, and on-device fine-tuning. It also introduces application-specific acceleration techniques for large language models, diffusion models, video recognition, and point cloud. This course will also cover topics about quantum machine learning. Students will get hands-on experience deploying large language models (e.g., LLaMA 2) on a laptop.