awesome-ai-eval
github.com/Vvkmnn/awesome-ai-eval ↗Assessing the reliability, accuracy, and safety of large language models, RAG pipelines, and AI agents.
Use this list with your AI agent
Add the Context Awesome MCP server to Claude, Cursor, or any MCP client, then ask:
"Show me safety resources from awesome-ai-eval"
Installation instructions →What's inside
Benchmarks
- AdvBenchSafety
Harmful behaviors and strings for measuring jailbreak resistance.
- AgentBenchAgents
LLMs acting as agents across operating system, database, web, and game environments.
- AGIEvalGeneral
Human-centric exams spanning admission tests, law, and math.
- Anthropic Model-Written EvalsSafety
Model-written datasets for testing persona, sycophancy, and AI risk behaviors.
- ARC-AGI-2General
Abstract reasoning puzzles that are easy for people and hard for models.
- AstaBenchAgents
Scientific research tasks covering literature review, replication, and data analysis.
Tools
- AgentDojoAdversarial
Environment for testing prompt-injection attacks and defenses on tool-using agents.
- agenttraceFrameworks
Terminal UI for reviewing coding-agent sessions by cost, failures, latency, and diffs.
- Aleph Alpha Eval FrameworkFrameworks
Evaluation framework with built-in benchmarks for reasoning, coding, and safety.
- autoevalsFrameworks
Scorers for LLM outputs, from string heuristics to model-graded factuality, in Python and TypeScript.
- BEIRRetrieval
Benchmark suite covering dense, sparse, and hybrid retrieval tasks.
- BloomFrameworks
Agentic framework that generates behavioral evaluations of frontier models from a described behavior.
Resources
- AI Evals for Engineers & PMsGuides
Cohort course by Hamel Husain and Shreya Shankar.
- AlignEvalGuides
Eugene Yan on building and aligning LLM judges.
- Applied LLMsGuides
Lessons from a year of building with LLMs, with evaluation as a core practice.
- awesome-evalsLists
Resources for building and evaluating AI agents.
- Awesome-LLM-EvalLists
Evaluation tools, papers, datasets, and leaderboards, in Chinese and English.
- Awesome-RAG-EvaluationLists
RAG evaluation metrics, datasets, and leaderboards.
Platforms
- Amazon Bedrock EvaluationsCloud
Managed scoring of foundation models and RAG pipelines.
- Arize PhoenixSelf-Hosted
OpenTelemetry-native tracing and evaluation for RAG, LLMs, and agents.
- Athina AIManaged
Evaluation and monitoring platform with preset evals and VPC deployment.
- BraintrustManaged
Evaluation and observability platform with experiments, datasets, and CI regression tests.
- Confident AIManaged
DeepEval-backed platform for scheduled eval suites and production monitoring.
- Datadog LLM ObservabilityManaged
LLM traces, quality metrics, and safety signals inside Datadog.
Leaderboards
- ARC Prize Leaderboard
ARC-AGI scores for frontier models and open submissions.
- Arena
Crowdsourced head-to-head model voting, formerly LMArena.
- CompassRank
OpenCompass rankings across multi-domain suites.
- HELM Leaderboards
Stanford results across HELM scenarios and metrics.
- Open LLM Leaderboard
Hugging Face rankings of open models on IFEval, MMLU-Pro, GPQA, and more.
- Open Medical-LLM Leaderboard
Hugging Face rankings on medical question answering.
Showing a sample of 150 resources. View the full list on GitHub →