GPT-Neo is the name of the codebase for transformer-based language models loosely styled around the GPT architecture. An implementation of model & data parallel GPT2 & GPT3 -like models, with the ability to scale up to full GPT3 sizes* (and possibly more!), using the mesh-tensorflow library.

Developed by https://www.eleuther.ai/