Testing frameworks now cover very different jobs, from AI evaluation and security testing to agent-assisted quality engineering and conventional application tests. This July 2026 comparison uses ToolVitals data for project health, shipping activity, licensing, GitHub popularity, and the overall ToolVitals score.
The ranking includes only open or source-visible tools. Tools are ordered by ToolVitals score first, with GitHub stars used only as a secondary popularity signal; the supplied scores and ordering are kept unchanged.
Rankings
| Rank | Tool | Openness | License | Health | Shipping | GitHub Stars | Score | Status |
|---|---|---|---|---|---|---|---|---|
| 1 | Promptfoo | OSI-approved OSS | MIT | 100 | 100 | 23681 | 100 | 🟢 Excellent |
| 2 | Agentic QE Fleet | OSI-approved OSS | MIT | 80 | 95 | 423 | 89 | 🟢 Excellent |
| 3 | kotest | OSI-approved OSS | Apache-2.0 | 85 | 83 | 4787 | 87 | 🟢 Excellent |
| 4 | Lost Pixel | OSI-approved OSS | MIT | 41 | 0 | 1686 | 38 | 🔴 Needs Attention |
Top 3 Highlights
Promptfoo ranks first with a ToolVitals score of 100, perfect health and shipping scores, and 23,681 GitHub stars. Its current first-party site focuses on evaluation and security testing for AI applications, including agents and RAG systems, with red teaming that covers prompt injection, data leakage, business-rule violations, and insecure tool use. It also documents integration with CI/CD systems and developer workflows. Promptfoo is OSI-approved OSS under the MIT license.
Agentic QE Fleet ranks second with a ToolVitals score of 89, a health score of 80, and a shipping score of 95. Its first-party site positions the project as an npm package that adds specialized quality-engineering agents and skills to existing coding agents through MCP, covering work such as test generation, coverage-gap analysis, and flaky-test detection. Its 423 GitHub stars indicate a much smaller community footprint than the other top-three projects, but the stronger ToolVitals score determines its position. Agentic QE Fleet is OSI-approved OSS under the MIT license.
kotest ranks third with a ToolVitals score of 87, a health score of 85, a shipping score of 83, and 4,787 GitHub stars. Its official site describes a Kotlin testing stack with multiple test styles, coroutine support, parameterized and data-driven tests, more than 350 assertions, property testing, and Kotlin Multiplatform support. This makes it the top-three option centered on general Kotlin application testing rather than AI-specific evaluation or agent-driven QE. kotest is OSI-approved OSS under the Apache-2.0 license.
Want to see the full health, shipping, openness, and evidence details for every tool in Testing Frameworks? Browse all Testing Frameworks Tools →