# Generalized Knowledge Distillation (GKD) Training

This directory contains configuration recipes for training models using Generalized Knowledge Distillation (GKD), an on-policy distillation method implemented in TRL.

## References

- **Paper**: [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://arxiv.org/abs/2306.13649)
- **TRL Docs**: [GKD Trainer Documentation](https://huggingface.co/docs/trl/main/en/gkd_trainer)
- **Demo Space**: [HuggingFace GKD Demo](https://huggingface.co/spaces/HuggingFaceH4/on-policy-distillation)
