Methodology · published
How comparison tables are sourced
Metrics as of , from the GitHub or GitLab API of each repository. Refreshed monthly.
Each comparison table on FreeQATools has two kinds of rows. Metric rows come from the GitHub or GitLab API and carry the fetch date. Fact rows come from each project's README or docs, and every cell links its source or reads "not documented".
A fixed row set per category
Each category has one fixed set of fact rows. Every comparison in a category shows the same rows in the same order. The rows cover documented scope, such as supported languages, browsers, protocols, test authoring format, CI integration and report formats. Some categories add their own rows, such as fault types for chaos tools.
Facts are collected once per tool, not once per pair. The same fact therefore reads the same on every comparison, alternatives entry and tool page that shows it. A tool in several categories has facts for the rows of each of its categories.
One fact, one source, one quote
Every documented cell links the README or docs page it came from. The fact record also stores the date that page was fetched. It also stores a verbatim quote from that page. A separate script, check-facts, looks for each quote in a stored copy of the fetched page. The script runs as its own step before the site build.
The check-facts script also confirms that the fetch date stored for each fact matches the stored copy of the page. Values are short facts in the project's terms, such as a list of browsers or report formats. Values that describe quality or preference are not used (comparisons).
Not documented, and what does not count
When a project's documentation does not state a fact, the cell reads "not documented". The note beside it says the fact was not found in the documentation as of the date the tool's facts were collected. "Not documented" does not mean the tool lacks the feature. Values that state an absence are not recorded as facts.
Features of a paid edition or a hosted service are not counted as facts of the open-source tool. When the docs describe an open-source route to the same feature, the cell cites that route instead. Notes about a project's own CI, or other projects running the tool in CI, do not count as CI integration.
When each fits
Each comparison page has a "When each fits" section with one sentence per tool. The sentence is derived from documented scope and follows one pattern: "Fits projects that {scope}; its docs describe {fact}.", with "its README describes" when the source is the README. Each sentence links its own source. The check-facts script verifies its quote like the quotes behind the table cells.
Both tools get sentences of the same structure, so neither is presented as the preferred option. The comparison pages are listed on the compare page. Tool status is covered in how tool status is derived.