KFD: Selective Token Filtering and Adaptive Weighting for Efficient Knowledge Distillation

Abstract

Knowledge distillation (KD) transfers knowledge from large language models (LLMs) to smaller or similarly sized models in order to obtain efficient yet capable systems. However, performing distillation over all tokens is computationally expensive and may weaken the transfer signal. To address this limitation, Knowledge-Filtered Distillation (KFD) is introduced as a selective distillation approach in which tokens are filtered according to the divergence KL(M2 divided by M0) between a teacher model (M2) and a base model (M0), while the student model (M1) is also derived from the same base model. Only tokens whose divergence exceeds a predefined threshold are distilled. For the selected tokens, the teacher distribution is normalized over the Top-5 predictions, whereas tokens outside this case receive a label-ranking bonus. The proposed conditional Top-5/bonus target design is shown theoretically to yield a lower label-focused target error than using only Top-5 normalization or only the bonus across all tokens. In addition, the KL and cross-entropy (CE) losses are balanced through a dynamically computed batch-level coefficient alpha. Experiments on multiple Turkish text datasets show that KFD consistently outperforms CE-only training, achieving higher accuracy with less data and shorter training time. KFD also outperforms entropy-based token selection methods and highlights the role of student initialization in effective knowledge transfer, thereby providing an efficient and scalable distillation framework for teacher-student models of equal size.