Tag: papers
All the articles with the tag "papers".
-
Notes on: On-Policy Distillation of Language Models (GKD)
The actual GKD objective — a λ-weighted mix of fixed-dataset and student-sampled sequences, a tunable JSD(β) divergence family, and where each choice wins across T5-family experiments.
-
Notes on: TinyBERT
Jiao et al.'s four-loss layer-wise distillation and two-stage framework, with the actual GLUE table — TinyBERT6 essentially matches BERT-base, TinyBERT4 keeps 96.8% of it at 7.5x smaller.
-
Notes on: DistilBERT
Sanh et al.'s actual architecture and loss recipe — halved layers initialized every-other-layer from BERT, a triple loss, and the specific GLUE/SQuAD numbers behind the "97% at 40% smaller" headline.
-
Notes on: Distilling the Knowledge in a Neural Network
Hinton, Vinyals & Dean's 2015 paper — the mechanics of soft-target distillation, the "omit a digit" MNIST experiment, and how it scales an ensemble of specialists back into one model.