awesome-knowledge-distillation-of-llms
github.com/tebmer/awesome-knowledge-distillation-of-llms ↗This repository collects papers for "A Survey on Knowledge Distillation of Large Language Models". We break down KD into Knowledge Elicitation and Distillation Algorithms, and explore the Skill & Vertical Distillation of LLMs.
1.3k
GitHub Stars
221
Curated Resources
5
Categories
5 hours ago
Last Refreshed
💡 NewsKD AlgorithmsSkill DistillationVerticalization DistillationEncoder-based KD
Use this list with your AI agent
Add the Context Awesome MCP server to Claude, Cursor, or any MCP client, then ask:
"Show me agent resources from awesome-knowledge-distillation-of-llms"
Installation instructions →What's inside
Skill Distillation
- Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language ModelsAgent
CoRL
- AgentTuning: Enabling Generalized Agent Abilities for LLMsAgent
arXiv
- Aligning Large and Small Language Models via Chain-of-Thought ReasoningAlignment
EACL
- Aligning Large Language Models through Synthetic FeedbacksAlignment
EMNLP
- Alpaca: Aligning Language Model with Human PreferencesContext Following
-
- Annollm: Making large language models to be better crowdsourced annotatorsNLP Task Specialization
arXiv
KD Algorithms
- Aligning Large Language Models through Synthetic FeedbackDistillation Algorithms
EMNLP
- An Empirical Study of Instruction-tuning Large Language Models in ChineseKnowledge Elicitation
EMNLP
- APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and InferenceKnowledge Elicitation
arXiv
- Baby Llama: Knowledge Distillation from an Ensemble of Teachers Trained on a Small Dataset with No Performance PenaltyDistillation Algorithms
CoNLL
- Beyond human data: Scaling self-training for problem-solving with language modelsKnowledge Elicitation
arXiv
- Beyond Imitation: Leveraging Fine-grained Quality Signals for AlignmentKnowledge Elicitation
arXiv
Verticalization Distillation
- AlpaCare: Instruction-tuned large language models for medical applicationMedical & Healthcare
arXiv
- AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse DatasetsScience
arXiv
- Biomedgpt: Open Multimodal Generative Pre-trained Transformer for BiomedicineScience
arXiv
- ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain KnowledgeMedical & Healthcare
arXiv
- ChatLaw: Open-Source Legal Large Language Model with Integrated External Knowledge BasesLaw
arXiv
- DARWIN Series: Domain Specific Large Language Models for Natural ScienceScience
arXiv
Showing a sample of 221 resources. View the full list on GitHub →