lm-evaluation-harness
Framework for evaluating language models on a common set of benchmark tasks.
Find alternatives to lm-evaluation-harness
What it does
- Evaluation and observability
Features 6
- Standard benchmark suiteOver 60 standard academic LLM benchmarks with hundreds of subtasks and variants.
- PluginsRegister model backends, filters, metrics and aggregations from your own package instead of forking.
- vLLM inferenceFast and memory-efficient inference with vLLM.
- Commercial API supportEvaluates models through commercial APIs including OpenAI and TextSynth.
- Adapter evaluationEvaluation of adapters such as LoRA supported in Hugging Face PEFT.
- Custom prompts and metricsEasy support for custom prompts and evaluation metrics.
Sources
- Pricing
-
Official documentation · 23 Aug 2026
MIT licensed on the repository page. EleutherAI's own site has no product pages — its Releases page files each artifact as Library, Dataset or Model — so the repository is the tool's official home.
- Sold within
-
Official documentation · 30 Sep 2026
The repository is published under the "MIT license" and installs with "git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness" or "pip install lm_eval[hf]". No paid offering is named.
- Interfaces
-
Platform listing · 14 Oct 2026
PyPI listings published by eleutherai: 'pip install lm-eval' / 'Command-line arguments and subcommands'.
- Intended for
-
README: 'used in hundreds of papers'; 'ensures reproducibility and comparability between papers'
Also from EleutherAI
-
GPT-NeoX Developer tool
Library for training large autoregressive language models on GPU clusters.
Open EleutherAI in the directory
Something wrong here? Send a correction — quote this product id: eleutherai-lm-evaluation-harness.