LangSmith Evaluation
LangSmith's evaluation service for agents: run evaluations on curated datasets to compare agent versions and catch regressions, score live traffic with online evaluators, collect human feedback in annotation queues and optimise prompts in a playground.
Find alternatives to LangSmith Evaluation
What it does
- Evaluation and observability
Sources
- Pricing
-
Developer "$0 / seat per month then pay as you go" and Plus "$39 / seat per month"; "Trace volume (Observability & Evaluation)" is included monthly on both, then "Pay as you go thereafter".
- Description
-
Official documentation · 18 Oct 2026
"Run evaluations on curated datasets during development to compare agent versions, benchmark performance, and catch regressions before users do. Monitor performance in production with online evals that score user interactions with your agent in real-time"
- What it does
-
Official documentation · 18 Oct 2026
Evaluator types: "human evaluation through annotation queues, heuristic checks (like validating outputs or checking if code compiles), LLM-as-judge evaluators that score against criteria you define, and pairwise comparison".
- Sold within
-
The pricing row "Trace volume (Observability & Evaluation)" names Evaluation and includes a monthly allowance on both the free Developer plan and Plus.
- Name
-
Official documentation · 18 Oct 2026
The LangSmith Platform menu lists "Evaluation: Score and improve agent performance".
- URL
-
Official documentation · 18 Oct 2026
The component's own page, titled "LangSmith: AI Agent & LLM Model Evaluation Platform".
- Deployment
-
Part of the hosted LangSmith platform; the pricing page lists "Self-hosted and hybrid deployment options" for Enterprise.
- Interfaces
-
Official documentation · 18 Oct 2026
"Experiment with prompts in the Playground" and "all from the UI".
- Modalities
-
Official documentation · 18 Oct 2026
"Conversation evals" and "Multi-modal evals".
Also from LangChain
-
LangChain Developer tool
Open-source framework for composing language-model calls, tools and retrieval into applications.
-
LangGraph Developer tool
Open-source library for building agents as explicit graphs of steps and state.
-
LangSmith Platform
Traces, evaluates and monitors language-model applications in development and production.
-
LangSmith Context Hub Agent platform
Versions and deploys prompts and agent context. Part of the LangSmith platform (categorized under 'No-Code Agents', 'Observability', 'Evaluation' or 'Deployment infrastructure' depending on the module).
-
LangSmith Deployment Agent platform
Deploys and operates production agents. Part of the LangSmith platform (categorized under 'No-Code Agents', 'Observability', 'Evaluation' or 'Deployment infrastructure' depending on the module).
-
LangSmith Engine Agent platform
Analyzes agent traces and recommends fixes. Part of the LangSmith platform (categorized under 'No-Code Agents', 'Observability', 'Evaluation' or 'Deployment infrastructure' depending on the module).
-
LangSmith Fleet Agent platform
No-code agent building, deployment into Slack/Teams/Gmail, and human-in-the-loop approval controls. Part of the LangSmith platform (categorized under 'No-Code Agents', 'Observability', 'Evaluation' or 'Deployment infrastructure' depending on the module).
-
LangSmith LLM Gateway Agent platform
Governs model calls and spending across providers. Part of the LangSmith platform (categorized under 'No-Code Agents', 'Observability', 'Evaluation' or 'Deployment infrastructure' depending on the module).
-
LangSmith Sandboxes Agent platform
Runs agent-generated code in isolated virtual machines. Part of the LangSmith platform (categorized under 'No-Code Agents', 'Observability', 'Evaluation' or 'Deployment infrastructure' depending on the module).
-
LangSmith Chat Application
LangSmith Chat analyzes application traces, threads, prompts and evaluations.
Open LangChain in the directory
Something wrong here? Send a correction — quote this product id: langsmith-evaluation.