GPT-NeoX
Library for training large autoregressive language models on GPU clusters.
What it does
- Model training
Features 6
- Distributed trainingDistributed training with ZeRO and 3D parallelism, launched via Slurm, MPI or the IBM Job Step Manager.
- Architectural innovationsRotary and alibi positional embeddings, parallel feedforward attention layers and flash attention.
- Predefined architecture configurationsReady-made configurations for Pythia, PaLM, Falcon and LLaMA 1 and 2.
- Mixture of ExpertsDropless Mixture of Experts support through the megablocks library.
- Hugging Face exportConverts GPT-NeoX checkpoints to the Hugging Face Transformers format.
- Experiment monitoringBuilt-in support for Weights & Biases, TensorBoard and Comet experiment monitoring.
Sources
- Pricing
-
Official documentation · 23 Aug 2026
Apache-2.0 licensed on the repository page. The name is shared with the GPT-NeoX-20B model trained using it; only the library is a product.
- Sold within
-
Official documentation · 30 Sep 2026
The repository is published under the "Apache-2.0 license" — "Licensed under the Apache License, Version 2.0" — with install steps beginning "To install the remaining basic dependencies, run:". No paid offering is named.
- Intended for
-
README: aims to 'accelerate research into large-scale training'; adopted by 'researchers at Oak Ridge National Lab... Carnegie Mellon University'
Also from EleutherAI
-
lm-evaluation-harness Developer tool
Framework for evaluating language models on a common set of benchmark tasks.
Open EleutherAI in the directory
Something wrong here? Send a correction — quote this product id: eleutherai-gpt-neox.