Suggest a tool

Promptfoo vs DeepEval: open-source AI and LLM evaluation tools compared

Metrics as of , from the GitHub or GitLab API of each repository. Refreshed monthly.

Summary

Promptfoo: CLI and library for LLM evaluations, side-by-side model comparison and red teaming of LLM-based applications.

DeepEval: Python framework for unit testing LLM applications, agents and RAG pipelines with LLM-as-a-judge and local metrics.

Both are listed under AI and LLM evaluation. The table gives repository metrics from the GitHub API with their fetch date, then the documented facts used for every comparison in this category, each linked to the README or docs page it comes from. A cell reads "not documented" when the fact was not found in the project's documentation; that does not mean the feature is absent.

Side by side

Promptfoo and DeepEval: metrics and documented facts, in the fixed row order used for AI and LLM evaluation comparisons.
FactPromptfooDeepEval
Stars25,371118,3931
Forks2,35311,9661
Contributors34913371
Last release2026-09-1812026-09-221
Last commit2026-09-2212026-09-221
Commits in 90 days77515551
LicenseMIT1Apache-2.01
Primary languageTypeScript1Python1
Statusactive1active1
Model providersOpenAI, Anthropic, Azure, Bedrock, Ollama and others source: READMEOpenAI, Azure OpenAI, Ollama, Anthropic, Gemini, LiteLLM; custom models through DeepEvalBaseLLM source: Docs: Introduction to LLM Metrics
Evaluation metricsAssertions for equality, JSON structure, similarity and custom functions source: Docs: Assertions & metricsG-Eval, DAG; agentic, RAG, multi-turn, MCP and multimodal metrics; hallucination, bias, toxicity, JSON correctness source: README
Test definition formatYAML config file (promptfooconfig.yaml) with prompts, providers and test cases source: Docs: Getting StartedCode (Python test files with assert_test, run by deepeval test run; Pytest-style) source: README
CI integrationGitHub Actions guide and GitHub Marketplace action; CI/CD guide for other platforms source: Docs: CI/CD Integration for LLM Evaluation and Securitydeepeval test run in any shell-step CI; GitHub Actions example; GitLab CI, CircleCI, Jenkins named source: Docs: Unit Testing in CI/CD
Report formatsHTML, JSON, CSV, JUnit XML (promptfoo eval --output) source: Docs: Output FormatsTestRun JSON files (results_folder); dashboard export as html or md (file_type); optional SQLite store source: Docs: Flags and Configs
Install methodnpm (npm install -g promptfoo), Homebrew (brew install promptfoo), pip (pip install promptfoo) source: READMEpip (pip install -U deepeval) source: README

1 Fetched from the GitHub or GitLab API on . Hover a value for its own date.

When each fits

Written from each project's documented scope, not from preference. Neither tool is ranked.

Promptfoo

Fits projects that test prompts and models of LLM-based apps; its README describes a CLI and library that evaluates and red-teams LLM-based apps. source: README

DeepEval

Fits projects that unit test LLM apps such as AI agents, RAG pipelines and chatbots; its README describes a framework similar to Pytest but specialized for LLM apps. source: README

More on these tools

Sources

  1. GitHub REST API: repository, read
  2. GitHub REST API: repository, read
  3. GitHub REST API: contributors, read
  4. GitHub REST API: contributors, read
  5. GitHub REST API: latest release, read
  6. GitHub REST API: latest release, read
  7. GitHub REST API: commits, read
  8. GitHub REST API: commits, read
  9. README, read
  10. Docs: Introduction to LLM Metrics, read
  11. Docs: Assertions & metrics, read
  12. README, read
  13. Docs: Getting Started, read
  14. Docs: CI/CD Integration for LLM Evaluation and Security, read
  15. Docs: Unit Testing in CI/CD, read
  16. Docs: Output Formats, read
  17. Docs: Flags and Configs, read

Documented facts collected 2026-09-22. See the methodology for how metrics and facts are gathered.