Suggest a tool

Promptfoo

Repository: promptfoo/promptfooHomepage

Metrics as of , from the GitHub or GitLab API of each repository. Refreshed monthly.

What Promptfoo is

Promptfoo is a command-line tool and library for evaluating and red-teaming applications built on large language models. It runs automated evaluations of prompts and models. Models can be compared side by side. Supported providers include OpenAI, Anthropic, Azure, Bedrock and Ollama. Red teaming and vulnerability scanning produce security reports for LLM apps. Code scanning reviews pull requests for LLM-related security and compliance issues. Checks can run in CI/CD pipelines. Results appear in a web viewer or on the command line. The README states that evals run locally. It installs through npm, Homebrew or pip. The project is now part of OpenAI and remains MIT licensed.

Written from the project's README, read .

Category
AI and LLM evaluation, Security testing
License
MIT
Language
TypeScript
Changelog
Releases on GitHub

Repository metrics

Repository metrics
Stars25,3711
Forks2,3531
Open issues and PRs6281
Contributors3491
Commits in the last 90 days7751
Last commit2026-09-221
Last release0.123.1, 2026-09-181
Fetched

1 Fetched from the GitHub or GitLab API on . Hover a value for its own date.

Status

activeLast commit within 90 days of the fetch date.

Computed from the last commit date and the archive flag on the fetch date. See the status rules.

Alternatives

Listed AI and LLM evaluation tools, same primary language first, then by GitHub stars. Each line gives one fact from the tool's documentation where it differs from Promptfoo's, with its source. See also Promptfoo alternatives.

  • OpenAI Evals: Model providers: Models on the OpenAI API; completion functions in evals/registry/completion_fns or any CompletionFn implementation. source: Docs: How to run evals

  • DeepEval: Model providers: OpenAI, Azure OpenAI, Ollama, Anthropic, Gemini, LiteLLM; custom models through DeepEvalBaseLLM. source: Docs: Introduction to LLM Metrics

  • Ragas: Model providers: OpenAI, Anthropic, Google directly; Azure OpenAI, AWS Bedrock, Google Vertex AI and others through LiteLLM. source: Docs: Customise models

  • Language Model Evaluation Harness: Model providers: Hugging Face transformers, vLLM, SGLang, GGUF via llama.cpp, NeMo, Megatron-LM; OpenAI, Anthropic, LiteLLM, local API servers. source: README

  • garak: Model providers: Hugging Face, Replicate, OpenAI, AWS Bedrock, LiteLLM, Cohere, Groq, NIM, ggml/GGUF, REST endpoints. source: README

  • Giskard: Model providers: Provider SDK extras such as openai and anthropic; default judge model openai/gpt-4o-mini. source: README

  • TruLens: Model providers: OpenAI, Azure OpenAI, LiteLLM, Google Gemini, AWS Bedrock, Snowflake Cortex, HuggingFace, LangChain models, OrcaRouter. source: README

  • HELM: Model providers: Models from various providers through one interface, such as OpenAI, Anthropic Claude, Google Gemini. source: README

Comparisons

How to install

npm install -g promptfoo
From README, read .

Questions

Is Promptfoo open source?

Yes. Promptfoo is released under MIT, an OSI-approved license, as reported by the GitHub API on 2026-09-22.

Is Promptfoo maintained?

On 2026-09-22, the last commit to the default branch was on 2026-09-22, so the listed status is active. Rule: Last commit within 90 days of the fetch date.

How many GitHub stars does Promptfoo have?

25,371 stars on 2026-09-22, from the GitHub API. The number is refreshed at each monthly update.

What language is Promptfoo written in?

The repository's primary language, as reported by the GitHub API, is TypeScript.

How do I install Promptfoo?

The README gives this command: npm install -g promptfoo

Sources

  1. GitHub REST API: repository, read
  2. GitHub REST API: commits, read
  3. GitHub REST API: latest release, read
  4. GitHub REST API: contributors, read
  5. README, read