The Growth of DNNs and the Need for Efficiency
On ImageNet, AlexNet brought the Top-5 error down to 16.4% in 2012, then GoogLeNet 6.7%, ResNet 3.6%, and SENet 2.3%, passing the human error rate (5.1%) from 2015. Model size and compute grew along the way.
Model size went from Transformer (0.05B, 2017) → GPT-2 (1.5B) → GPT-3 (175B) → MT-NLG (530B, 2022), roughly 10,000x in 5 years. Over the same period GPU memory grew about 5x, from TPUv2 16GB to A100 80GB. Because of this gap, models stop fitting on a single GPU and distributed techniques like tensor parallelism become necessary. ChatGPT showing “at capacity” or limiting users to 50 messages per 3 hours was also a serving cost problem.
Redundancy
DNNs contain a lot of redundancy. Many weights contribute little to the output or overlap with other weights in what they do. This redundancy is what makes pruning, quantization, and distillation work: unnecessary weights can be removed, important information survives lower precision, and the core knowledge can be moved into a smaller model. Large models also tolerate quantization better because they have more redundancy, so the remaining weights absorb the precision loss of the others.
Model Compression
Model Compression is the umbrella category, with pruning (removing unimportant weights, neurons, heads, or tokens), quantization (FP32 → FP16 → INT8 → INT4), knowledge distillation, low-rank factorization, and NAS underneath. These techniques are combined rather than used alone. In the Lite Transformer example, applying quantization and pruning on top of the architecture change went from 176MB to 9.7MB (18.2x) while BLEU only dropped from 39.9 to 39.5.
Integer vs Floating Point
Quantization maps the continuous weight distribution of FP16 onto discrete values like INT8 (-128 to 127) or INT4 (-8 to 7) with a scale factor. FP16 takes 2 bytes per weight, INT8 takes 1, and INT4 takes 0.5, so memory shrinks, and integer arithmetic is faster and uses less power than floating point in hardware. The cost is a possible drop in accuracy.
In practice the formats are mixed by use case: FP32 or BF16 for training, FP16 or W8A8 for server inference, W4A16 or W4A8 for edge inference, and FP16 for attention because of softmax precision. Starting with Blackwell (B100/B200), FP4 (NVFP4) has hardware support. It is 4-bit but keeps floating point semantics, so its dynamic range is wider than INT4, and NVIDIA claims less than 1% accuracy drop versus FP8.
Quantization Techniques
RTN (Round-To-Nearest) rounds each weight to the nearest integer. It holds up on large models (70B+) or at INT8, but quality breaks down at INT4 on a 7B model. The lecture shows an example where the model is given the Mona Lisa and fails to answer “Leonardo da Vinci”.
AWQ (Activation-aware Weight Quantization) is from Song Han’s lab. It analyzes activation distributions and protects the important weight channels while quantizing. It has become the de facto standard for weight-only (W4A16) quantization. At the same INT4, the quality gap versus RTN is large.
SmoothQuant deals with the activation outlier problem. Activations are hard to quantize because of outliers while weights are evenly distributed and easy, so the transformation Y = XW = (X/s)(sW) migrates quantization difficulty from activations to weights. Both sides then become easier to quantize, which makes W8A8 workable.
TinyChatEngine is a lightweight LLM inference engine written from scratch in C/C++ by Han’s lab. It serves AWQ-quantized models on edge devices. There is a demo running LLaMA-2-7B W4A16 at 30 tokens/s on a Jetson Orin.
Sparsity and Pruning
Pruning sets unimportant weights to zero, which makes the matrix sparse. Multiplication by zero always gives zero, so those operations can be skipped. A100’s 2:4 structured sparsity uses a pattern where 2 out of every 4 values must be zero, which the Tensor Cores recognize in hardware and skip, for a theoretical 2x speedup.
The jump from Q8000 (Turing) 261 TOPS to A100 1248 TOPS in the lecture graph should not be read as a sparsity effect alone. A100’s INT8 dense performance is 624 TOPS, so 261 → 624 comes from the generation change (3rd-gen Tensor Cores), and 624 → 1248 corresponds to the theoretical 2x from 2:4 sparsity.
SpAtten
SpAtten prunes unimportant tokens and heads at runtime in attention. In the sentiment classification example on “As a visual treat, the film is almost perfect”, layer 1 computes all 11 tokens and 12 heads, layer 2 drops to 5 tokens and 10 heads (38%), layer 3 to 2 tokens and 8 heads (12%), and at the end it classifies Positive with only “film perfect” left.
Token importance is computed by accumulating attention probability vertically. Function words like “I”, “a”, “the” have low cumulative importance and get cut first. The distinguishing point is that this is token/head-level dynamic pruning whose outcome changes per input, not weight pruning. Where FlashAttention reduces attention’s memory access, SpAtten reduces the amount of computation itself.
On-Device / Edge AI
When data cannot leave the device for privacy, offline, or personalization reasons, both inference and training have to happen on the edge device. Training is the harder one. Inference is a single forward pass, but training stores all activations from the forward pass, computes gradients in the backward pass, and keeps optimizer state on top.
MCUNet V3
The MCUNet V3 paper reports cutting training memory from 303MB (PyTorch) down to 141KB, a 2300x reduction. Against the 652MB TensorFlow (cloud) baseline it is about 4600x. Three techniques applied in order:
- Quantization-Aware Scaling: correct gradients by the inverse square of the weight scale. 5.7MB → 2.9MB (2.0x)
- Sparse Layer Update: run backward only on important layers. 2.9MB → 335KB (8.8x)
- Sparse Tensor Update: update only the important parts within a tensor. 335KB → 141KB (2.4x)
VLM and VILA
ViT is a vision encoder that splits an image into patches and feeds them to a Transformer. A VLM attaches an LLM to that and understands both images and text; LLaVA and GPT-4V belong here. LLaVA is a CLIP ViT + projection layer + LLaMA structure. VILA is a VLM where NVIDIA and Song Han’s lab improved on LLaVA, and it can be quantized with AWQ and deployed on a Jetson. VLM (image → text, understanding) and Stable Diffusion (text → image, generation) go in opposite directions; what they share is the use of vision-language alignment models like CLIP.
Hardware Trend
Looking at 42 years of microprocessor data, transistor counts keep rising but clock frequency and single-thread performance have stalled since around 2005 (the end of Dennard scaling). The direction since then has been dedicated compute units like Tensor Cores rather than more general-purpose cores.
The “317x in 8 years” figure in the lecture (K20X 3.94 → A100 1248) needs careful reading. The K20X number is FP32 TFLOPS while the A100 number is INT8 with sparsity included, so it is not a fixed-precision performance gain. It is a trendline that folds together precision reduction, Tensor Cores, and sparsity.
Memory Wall
Compute (TOPS) grows 2 to 3x per generation while HBM bandwidth grows around 2x. As the gap accumulates, the bottleneck becomes memory access rather than compute. This is why FlashAttention increases computation through recomputation and still wins by reducing HBM reads/writes.
Cloud vs Edge
The lecture slides have a table comparing B100 (1,750 TOPS, 700W) against Qualcomm and Apple mobile chips in TOPS/W, with the edge chips ahead in efficiency. Edge chips are designed around performance per watt because of battery and thermal constraints. The numbers themselves are vendor marketing figures though, and do not compare directly: the precision behind each TOPS figure differs by vendor and is mostly undisclosed, and the power side mixes GPU module TDP with whole-SoC estimates. What to take away is roughly this: with a power budget around 10W, computation has to be cut in software through quantization and pruning.
Course Overview
The following lectures cover pruning/sparsity, quantization, NAS, knowledge distillation, on-device training, efficient attention (FlashAttention and others), and distributed training/inference one by one. Software (6.5940) is best read alongside Hardware (6.5930).