Suggest a tool

Promptfoo vs Language Model Evaluation Harness: open-source AI and LLM evaluation tools compared

Metrics as of , from the GitHub or GitLab API of each repository. Refreshed monthly.

Summary

Promptfoo: CLI and library for LLM evaluations, side-by-side model comparison and red teaming of LLM-based applications.

Language Model Evaluation Harness: Framework for evaluating generative language models on academic benchmarks across transformers, vLLM, SGLang and API backends.

Both are listed under AI and LLM evaluation. The table gives repository metrics from the GitHub API with their fetch date, then the documented facts used for every comparison in this category, each linked to the README or docs page it comes from. A cell reads "not documented" when the fact was not found in the project's documentation (1 of 12 fact cells here); that does not mean the feature is absent.

Side by side

Promptfoo and Language Model Evaluation Harness: metrics and documented facts, in the fixed row order used for AI and LLM evaluation comparisons.
FactPromptfooLanguage Model Evaluation Harness
Stars25,371114,0551
Forks2,35313,5851
Contributors34914601
Last release2026-09-1812026-08-311
Last commit2026-09-2212026-09-141
Commits in 90 days7751911
LicenseMIT1MIT1
Primary languageTypeScript1Python1
Statusactive1active1
Model providersOpenAI, Anthropic, Azure, Bedrock, Ollama and others source: READMEHugging Face transformers, vLLM, SGLang, GGUF via llama.cpp, NeMo, Megatron-LM; OpenAI, Anthropic, LiteLLM, local API servers source: README
Evaluation metricsAssertions for equality, JSON structure, similarity and custom functions source: Docs: Assertions & metricsacc, acc_norm, perplexity, matthews_corrcoef, f1, bleu, chrf, ter; exact_match and others through metric_list source: Docs: Task Configuration
Test definition formatYAML config file (promptfooconfig.yaml) with prompts, providers and test cases source: Docs: Getting StartedYAML task configs with Jinja2 prompts; CLI (lm_eval) and Python API (simple_evaluate) source: README
CI integrationGitHub Actions guide and GitHub Marketplace action; CI/CD guide for other platforms source: Docs: CI/CD Integration for LLM Evaluation and Securitynot documentednot found in the project's documentation as of
Report formatsHTML, JSON, CSV, JUnit XML (promptfoo eval --output) source: Docs: Output FormatsResults JSON via --output_path; sample logs via --log_samples; Hugging Face Hub, W&B and Zeno uploads source: README
Install methodnpm (npm install -g promptfoo), Homebrew (brew install promptfoo), pip (pip install promptfoo) source: READMEpip from a Git clone (pip install -e .); model backends as extras such as lm_eval[hf], lm_eval[vllm], lm_eval[api] source: README

1 Fetched from the GitHub or GitLab API on . Hover a value for its own date.

When each fits

Written from each project's documented scope, not from preference. Neither tool is ranked.

Promptfoo

Fits projects that test prompts and models of LLM-based apps; its README describes a CLI and library that evaluates and red-teams LLM-based apps. source: README

Language Model Evaluation Harness

Fits projects that test generative language models on academic benchmark tasks; its README describes a unified framework for a large number of evaluation tasks. source: README

More on these tools

Sources

  1. GitHub REST API: repository, read
  2. GitHub REST API: repository, read
  3. GitHub REST API: contributors, read
  4. GitHub REST API: contributors, read
  5. GitHub REST API: latest release, read
  6. GitHub REST API: latest release, read
  7. GitHub REST API: commits, read
  8. GitHub REST API: commits, read
  9. README, read
  10. README, read
  11. Docs: Assertions & metrics, read
  12. Docs: Task Configuration, read
  13. Docs: Getting Started, read
  14. Docs: CI/CD Integration for LLM Evaluation and Security, read
  15. Docs: Output Formats, read

Documented facts collected 2026-09-22. See the methodology for how metrics and facts are gathered.