CA

DeepEval

Developer tool

by Confident AI

Open-source (Apache 2.0) framework that unit-tests LLM applications, agents, RAG pipelines and chatbots with pytest-native evals, 50+ research-backed LLM-as-a-judge metrics, synthetic golden generation and simulated conversations.

Find alternatives to DeepEval

What it does

• Evaluation and observability

Sources

Pricing

Official documentation · 21 Oct 2026

Open licence, no charge: "Apache 2.0 licensed"; the Confident AI platform that syncs with it is a separate commercial product ("Is DeepEval the same as Confident AI? No.").

Description

Official documentation · 21 Oct 2026

"DeepEval is an open-source framework for evaluating AI agents, multi-turn chatbots, voice agents, RAG pipelines, and other LLM applications." and "Pytest-native evals that run in CI/CD or as Python scripts."

What it does

Official documentation · 21 Oct 2026

"DeepEval enables teams to build reliable evaluation pipelines to test any AI system."

Sold within

Official documentation · 21 Oct 2026

Obtained as open-source software ("Star us on GitHub", "Apache 2.0 licensed"); it needs no parent product or account.

Name

Official documentation · 21 Oct 2026

Page title and heading "DeepEval - The LLM Evaluation Framework".

URL

Official documentation · 21 Oct 2026

The framework's own site, linked as "DeepEval" from the Confident AI menu; footer "Open-source LLM evaluation framework. Apache 2.0 licensed." and "Confident AI Inc."

Deployment

Official documentation · 21 Oct 2026

"Iterate locally, on your own environment, on your own criteria."

Interfaces

Official documentation · 21 Oct 2026

Python code and CLI: "deepeval test run" and "from deepeval.metrics import TaskCompletenessMetric".

Modalities

Official documentation · 21 Oct 2026

"Multi-modal by default" - "Text, images, and audio — all first-class. Same test case, same runner, same metrics across every modality."

Intended for

Official documentation · 21 Oct 2026

"DeepEval is for AI engineers, ML teams, and developers building LLM products who want a reliable way to measure quality, compare changes, and ship with confidence."

Also from Confident AI

• LLM Observability Platform

  LLM Observability traces LLM applications, evaluates production traces, and alerts teams when quality metrics deteriorate.

• LLM Evaluation Platform

  Evaluates LLM applications and agents with 50+ research-backed metrics, mostly LLM-as-a-judge, over golden datasets and simulated multi-turn conversations run against any endpoint, with experiments that compare prompts or models and CI checks that block regressions.

• AI Red Teaming Platform

  Simulates adversarial attacks against an LLM app or agent reachable through an API endpoint, mapped to OWASP LLM Top 10, OWASP Agentic AI Top 10, NIST AI RMF and MITRE ATLAS, scores each finding by CVSS and returns the failed prompt, output and remediation guidance.

• DeepTeam Developer tool

  Open-source (Apache 2.0) LLM red-teaming framework that uses LLMs to simulate single- and multi-turn adversarial attacks across 120+ vulnerabilities and maps results to OWASP Top 10 for LLMs, MITRE ATLAS, NIST AI RMF and the EU AI Act from Python.

Open Confident AI in the directory

Something wrong here? Send a correction — quote this product id: confident-ai-deepeval.