awesome-ai-eval

Adversarial prompt benchmark for jailbreak and misuse resistance measurement.

Evaluates LLMs acting as agents across simulated domains like games and coding.

Human-centric standardized exams spanning entrance tests, legal, and math scenarios.

Next-generation reasoning benchmark where pure LLMs score 0% but humans can solve every task.

AI2 benchmark for scientific research AI agents covering literature review, experiment replication, and data analysis.

Bias-sensitive QA sets measuring stereotype reliance and ambiguous cases.

End-to-end LLM developer platform for prompt engineering, evaluation, and deployment.

Managed service for scoring foundation models and RAG pipelines.

Safety layer that evaluates prompts and responses for policy compliance.

OpenTelemetry-native observability and evaluation toolkit for RAG, LLMs, and agents.

Evaluation flows and risk reports wired into Prompt Flow projects.

Conversation analytics platform for evaluating chatbot quality, sentiment, and user satisfaction.

Statistical evaluation framework that runs AI agents N times, computes confidence intervals, and detects regressions in CI/CD.

Production-ready evaluation framework with 90+ pre-loaded benchmarks for reasoning, coding, and safety.

Automated instruction-following evaluator with length-controlled LLM judge scoring.

Anthropic's evaluation suite for safety, capabilities, and alignment testing of language models.

Automated multi-turn red teaming framework that simulates attacker-target interactions for jailbreak testing.

SOC-2 compliant LLM evaluation and monitoring platform with 50+ preset evaluations and VPC deployment.

Cohort course from Hamel & Shreya with lifetime reader, Discord, AI Eval Assistant, and live office hours.

Eugene Yan's guide on building LLM judges by following methodical alignment processes.

Practical lessons from a year of building with LLMs, emphasizing evaluation as a core practice.

Next.js chatbot with Phoenix tracing, dataset replays, and evaluation jobs.

Ecosystem list centered on ChainForge experiments and extensions.

Cross-lingual (Chinese) compendium of eval tooling, papers, datasets, and leaderboards.

AGI reasoning leaderboard tracking ARC-AGI-2 performance across frontier models and open submissions.

OpenCompass leaderboard comparing frontier and research models across multi-domain suites.

Aggregated leaderboard comparing multi-agent safety and reliability suites.

Crowdsourced LLM comparison platform (formerly LMSYS Chatbot Arena) with 6M+ user votes for Elo ratings.

Community suites and scores covering accuracy, safety, and instruction following.

Hugging Face benchmark board with IFEval, MMLU-Pro, GPQA, and more.