Suggest a tool

OpenAI Evals

Repository: openai/evals

Metrics as of , from the GitHub or GitLab API of each repository. Refreshed monthly.

What OpenAI Evals is

OpenAI Evals is a framework for evaluating large language models and systems built on them. It includes a registry of existing evals covering different dimensions of OpenAI models. Users can write custom evals, including private ones built from their own data. Basic and model-graded evals follow templates and need no evaluation code. Their data is written in JSON and their parameters in YAML. The Completion Function Protocol covers prompt chains and tool-using agents. It requires Python 3.9 or later. Running evals requires an OpenAI API key. Registry data is stored with Git LFS. Results can optionally be logged to a Snowflake database. The README notes that evals can also be configured in the OpenAI Dashboard.

Written from the project's README, read , and its documentation (see sources).

Category
AI and LLM evaluation
License
MIT
Language
Python
Changelog
Releases on GitHub

Repository metrics

Repository metrics
Stars19,4921
Forks3,0921
Open issues and PRs3401
Contributors4421
Commits in the last 90 days01
Last commit2026-04-141
Last releasev0.1.1, 2023-04-061
Fetched

1 Fetched from the GitHub or GitLab API on . Hover a value for its own date.

Status

slowLast commit 91 to 365 days before the fetch date.

Computed from the last commit date and the archive flag on the fetch date. See the status rules.

Alternatives

Listed AI and LLM evaluation tools, same primary language first, then by GitHub stars. Each line gives one fact from the tool's documentation where it differs from OpenAI Evals's, with its source. See also OpenAI Evals alternatives.

  • DeepEval: Model providers: OpenAI, Azure OpenAI, Ollama, Anthropic, Gemini, LiteLLM; custom models through DeepEvalBaseLLM. source: Docs: Introduction to LLM Metrics

  • Ragas: Model providers: OpenAI, Anthropic, Google directly; Azure OpenAI, AWS Bedrock, Google Vertex AI and others through LiteLLM. source: Docs: Customise models

  • Language Model Evaluation Harness: Model providers: Hugging Face transformers, vLLM, SGLang, GGUF via llama.cpp, NeMo, Megatron-LM; OpenAI, Anthropic, LiteLLM, local API servers. source: README

  • garak: Model providers: Hugging Face, Replicate, OpenAI, AWS Bedrock, LiteLLM, Cohere, Groq, NIM, ggml/GGUF, REST endpoints. source: README

  • Giskard: Model providers: Provider SDK extras such as openai and anthropic; default judge model openai/gpt-4o-mini. source: README

  • TruLens: Model providers: OpenAI, Azure OpenAI, LiteLLM, Google Gemini, AWS Bedrock, Snowflake Cortex, HuggingFace, LangChain models, OrcaRouter. source: README

  • HELM: Model providers: Models from various providers through one interface, such as OpenAI, Anthropic Claude, Google Gemini. source: README

  • Inspect: Model providers: OpenAI, Anthropic, Google, Grok, Mistral, DeepSeek; AWS Bedrock, Azure AI; Groq, Together AI; local models. source: Docs: Model Providers

Comparisons

How to install

pip install evals
From README, read .

Questions

Is OpenAI Evals open source?

Yes. OpenAI Evals is released under MIT, an OSI-approved license, as reported by the GitHub API on 2026-09-22.

Is OpenAI Evals maintained?

On 2026-09-22, the last commit to the default branch was on 2026-04-14, so the listed status is slow. Rule: Last commit 91 to 365 days before the fetch date.

How many GitHub stars does OpenAI Evals have?

19,492 stars on 2026-09-22, from the GitHub API. The number is refreshed at each monthly update.

What language is OpenAI Evals written in?

The repository's primary language, as reported by the GitHub API, is Python.

How do I install OpenAI Evals?

The README gives this command: pip install evals

Sources

  1. GitHub REST API: repository, read
  2. GitHub REST API: commits, read
  3. GitHub REST API: tags, read
  4. GitHub REST API: contributors, read
  5. LICENSE file, read
  6. README, read