awesome-llm-judges
github.com/haizelabs/awesome-llm-judges ↗⚖️ Awesome LLM Judges ⚖️
203
GitHub Stars
38
Curated Resources
6
Categories
4 hours ago
Last Refreshed
🌱 Starter🎭 Multi-Judge🎯 Finetuned Models🛡️ Safety👨⚖️ Judging the Judges: Meta-Evaluation✨ Contributing
Use this list with your AI agent
Add the Context Awesome MCP server to Claude, Cursor, or any MCP client, then ask:
"Show me 🛑 content moderation resources from awesome-llm-judges"
Installation instructions →What's inside
🛡️ Safety
- A STRONGREJECT for Empty Jailbreaks (Sections C.4 & C.5)🛑 Content Moderation
- Debate Helps Supervise Unreliable Experts🔍 Scalable Oversight
- Great Models Think Alike and this Undermines AI Oversight🔍 Scalable Oversight
- Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations🛑 Content Moderation
- LLM Critics Help Catch LLM Bugs🔍 Scalable Oversight
- On Scalable Oversight with Weak LLMs Judging Strong LLMs🔍 Scalable Oversight
🌱 Starter
🎭 Multi-Judge
- Can ChatGPT Defend its Belief in Truth? Evaluating LLM Reasoning via Debate🤔 Debate
- ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate🤔 Debate
- Debating with More Persuasive LLMs Leads to More Truthful Answers🤔 Debate
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- ScaleEval: Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate🤔 Debate
🎯 Finetuned Models
- Critique-out-Loud Reward Models🏆 Generative Reward Models
- Generative Verifiers: Reward Modeling as Next-Token Prediction🏆 Generative Reward Models
- HALU-J: Critique-Based Hallucination Judge🌀 Hallucination
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges
- Lynx: An Open Source Hallucination Evaluation Model🌀 Hallucination
- MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents🌀 Hallucination
👨⚖️ Judging the Judges: Meta-Evaluation
- Evaluating Large Language Models at Evaluating Instruction Following
- From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge
- JudgeBench: A Benchmark for Evaluating LLM-based Judges
- Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges⚖️ Biases
- Large Language Models are Inconsistent and Biased Evaluators⚖️ Biases
- Large Language Models are not Fair Evaluators⚖️ Biases
✨ Contributing
Showing a sample of 38 resources. View the full list on GitHub →