H

AI Evals

Developer tool

by Harness

Scores AI-agent responses for faithfulness, relevance, and task completion.

Find alternatives to AI Evals

What it does

• Evaluation and observability

Features 8

• Native pipeline stepRuns eval suites as a Harness pipeline step alongside build, test and deploy, gating the release.
• 50+ scoring metricsScores faithfulness, relevance, tone and task completion with LLM-as-judge, deterministic and semantic checks.
• Trajectory evaluationEvaluates multi-step agent behavior at the step, tool-call and final-outcome level.
• Safety and policy scoringBuilds prompt-injection detection and policy-compliance scoring into every suite.
• Run comparisonDiffs any two runs with per-item deltas to catch regressions during PR or CI.
• Online evaluationScores sampled live traffic to detect drift and promotes production traces into dataset cases.
• AI asset registryVersions prompts, agents, MCP tools and skills in a central registry with rollback and promotion.
• Open-source SDKApache 2.0 evaluation SDK that runs in any CI system, notebook or test suite.

Sources

Pricing

Official documentation · 16 Oct 2026

"AI Evals is in beta"

Description

Official documentation · 16 Oct 2026

"AI Evals scores every response"

What it does

Official documentation · 16 Oct 2026

"faithfulness, relevance, and task completion"

Sold within

Official documentation · 16 Oct 2026

"a native Harness pipeline step"

Name

Official documentation · 16 Oct 2026

"Harness AI Evals turns"

URL

Official documentation · 16 Oct 2026

"Harness AI Evals turns"

Deployment

Official documentation · 16 Oct 2026

"They can log in"

Interfaces

Official documentation · 18 Oct 2026

Page: 'Built on an Apache 2.0 evaluation SDK that runs in any CI system, notebook, or test suite'.

Also from Harness

• Harness Platform Platform

  AI-native software delivery platform: pipelines and canary deployments, AI security testing, runtime API and AI protection, cost management and policy governance, driven by AI agents that recommend while people decide.

• Security Testing Agent AI agent

  Uses agents to triage and remediate software vulnerabilities.

• Runtime Protection Agent AI agent

  Blocks prompt injection and unsafe model output.

• Cost Management Agent AI agent

  Tracks cloud and AI spending and automatically fixes waste.

• AI SRE Application

  Correlates deployment, feature flag, and infrastructure changes during incidents.

• AI SAST Developer tool

  AI SAST scans agent-generated code and provides AI-generated fix suggestions.

• AI SCA Developer tool

  AI SCA applies reachability analysis and EPSS scoring to open-source risk detection.

• AI Posture Application

  AI Posture discovers AI assets and agents and assesses their security posture.

• AI Firewall Application

  AI Firewall blocks prompt injection, evasion, and malicious payloads targeting AI applications.

• AI and Cloud Costs Application

  AI and Cloud Costs translates natural-language governance intent into cloud rules and remediates violations automatically.

• Software Delivery Agent AI agent

  Automates CI/CD and infrastructure-as-code delivery.

• AgentTrace Developer tool

  Monitors AI-agent LLM calls, tool calls, and decisions.

• Harness AI Test Automation Developer tool

  Creates intent-based tests and runs them within CI/CD pipelines with AI-assisted failure analysis.

• AI Code Review Developer tool

  Reviews pull requests by grouping diffs by risk so the riskiest changes come first, enforcing required checks that cannot be bypassed, and citing delivery history and the knowledge graph for context.

• AI DLC Insights Application

  Engineering-intelligence module that tracks DORA metrics, developer productivity and AI coding-tool spend together: which AI tools teams use, token consumption by developer, team, agent and model, AI-authored code and its effect on delivery.

• Worker Agents Agent platform

  Governed AI agents defined in a committed file that run inside Harness pipelines on events such as CI failures and pull requests, with prebuilt agents for autofix, code review, code coverage, feature-flag cleanup, manifest remediation and IaCM remediation.

• API Advanced Protection Application

  Detects and blocks API attacks, bots and Layer 7 DDoS with behavioral AI trained on live traffic, covering credential stuffing, account takeover and business-logic abuse such as BOLA and mass assignment.

Open Harness in the directory

Something wrong here? Send a correction — quote this product id: harness-ai-evals.