deepeval
The LLM Evaluation Framework
The Lens
By Erik Loyd, SaaS CEO and former COO/CFO of an AWS Premier Partner.
Updated Oct 2026
DeepEval is Pytest for LLM apps. Write test cases, attach metrics like hallucination, answer relevancy or whether an agent called the right tool, and run them in CI so a prompt change can't quietly break your product. The library is Apache 2.0, one pip install away, and free.
Most metrics are LLM-as-a-judge, so every "free" run still bills your model provider. A local Ollama model (deepeval set-ollama) takes that to zero, trading some judge quality. Run history no longer needs the cloud: results save locally to JSON or SQLite.
Confident AI, the company behind it, sells the hosted layer. Free covers 2 seats, 1 project and 5 test runs a week. Starter is $200/month per organization with unlimited seats; Team is $2,000/month for SSO, SOC2 and RBAC. Solo: free. Small teams: free until you want shared dashboards. Larger teams: Starter or Team. Promptfoo and Ragas are the open source rivals; LangSmith and Braintrust the hosted ones.
The catch: telemetry is on by default. DeepEval sends event names, metric names and your IP to PostHog until you set DEEPEVAL_TELEMETRY_OPT_OUT=1. The README also pushes deepeval login, which logs every test case to Confident AI's cloud. Skip both before testing on customer data.
Free vs Self-Hosted vs Paid
open coreFree: The deepeval library: Apache 2.0, pip install -U deepeval (Python 3.9+), every metric runs in your own environment. Judge calls bill at whatever LLM provider you configure; a local Ollama model makes them free. Confident AI's Free tier ($0, forever) adds cloud reports and tracing with hard limits: 2 user seats, 1 project, 5 test runs per week (additional runs are locked) and 1 GB-month of trace spans (extra spans are dropped).
Self-hosted: Nothing to host for the library. Test runs save locally as JSON files (default) or a single SQLite database, so run history works without an account. The Confident AI platform is a hosted service; the only self-managed option is the Enterprise tier's dedicated on-prem deployment.
Paid: Starter $200/month per organization: unlimited seats, 5 projects, online evals on live traffic, annotation queues, alerting, 5 GB-months of traces then $1 per GB-month. Team $2,000/month per organization: unlimited projects, 75 GB-months of traces, SSO, SOC2, custom RBAC, metric and dataset versioning. Enterprise is custom pricing (on-prem, HIPAA, custom data residency, 24x7 support). These are monthly prices; Confident AI says annual plans are discounted but publishes no annual figure. Startups with under $5M raised can apply for a grant.
The library is free and complete for local evals, with run history saved to JSON or SQLite. Shared cloud reports are free for 2 seats and 5 test runs a week, then $200/month per organization for Starter or $2,000/month for Team's SSO and SOC2.
What to do by team size
- Solo
- free; the local library with Ollama or your own API key
- Small team
- free locally; Starter at $200/month per organization once you need shared dashboards
- Medium team
- Starter at $200/month with unlimited seats for shared reports and live-traffic evals
- Large team
- Team at $2,000/month for SSO, SOC2 and RBAC, or Enterprise for on-prem and HIPAA
Get tools like this every Wednesday
One featured tool, three on the radar. No fluff.
Similar Tools
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
Open source LLM engineering platform
AI Observability & Evaluation
A desktop app to prototype agent ideas, inspect every harness step, replay failures, and evaluate performance, all in one place. Local-first, cloud-ready for managed agents.
Latitude is the open-source ai monitoring platform.
A low score is not a verdict on quality. Young and niche tools start low by design. How we calculate scores
Trust Signals
License: Apache License 2.0
Use freely. Patent grant included.
Commercial use: ✓ Yes
About
- Owner
- Confident AI (Organization)
- Stars
- 19,457
- Forks
- 2,195