deepeval

Deepeval is Pytest for LLM output. You write test cases, attach metrics like hallucination, answer relevancy, or whether the agent called the right tool, and run them in CI so a prompt change can't silently make your product worse. The framework is Apache 2.0, pip-installable, and runs entirely in your own environment.

The metrics are LLM-as-a-judge, which means 'free' evals still burn tokens at your model provider on every run. Point it at Ollama and a local model to bring that to zero. Confident AI, the company behind it, sells the cloud layer: dashboards, regression history, and monitoring, free for 2 seats and 5 test runs a week, then $200/mo.

Solo devs get real value from the local library alone. The paid platform starts making sense at 3+ people who need shared dashboards and run history. Promptfoo and Ragas are the closest OSS comparisons; LangSmith and Braintrust are the hosted incumbents it undercuts.

The catch: the free cloud tier's 5-runs-a-week ceiling is a hard wall for anything running on every commit. That limit is the funnel into the $200/mo tier, so budget for it or stay local.