Yuan 1.0 by Inspur
A GPT-3 equivalent for the Chinese language
About Yuan 1.0 by Inspur
Chinese server builder Inspur trained a monster text-generating neural network Yuan 1.0 said to pass Turing test, and require many fewer GPUs than GPT-3.
Inspur has turned its hand to AI, and claims it has produced a text-and-code-generating machine-learning model superior to GPT-3 produced by OpenAI. And did so using significantly fewer GPUs.
Inspur's model is called Yuan 1.0 and produces text in Chinese. The Chinese server maker says the model has 245.7 billion parameters (GPT-3 has 175 billion), claims it can pass a Turing test, and reckons it can beat humans at an idiom-reading comprehension task.
What they did: When you’re training models of this side, a lot of the hard stuff is plumbing – literally. You need to figure out how to build well-optimized pipelines for training your model on thousands of GPUs, which involves salami slicing different stages of model training to maximize data efficiency. Similarly, you need to feed these GPUs with data in the right order, further increasing efficiency. The paper includes some nice discussion of how the Inspur researchers tried to do this.
Compute: They used 2128 GPUs to train the 245B model, with a context length of 2048 tokens.
Data, via AI helping AI: To train the model, they build a dataset of 5TB of predominantly Chinese text. (By comparison, Huawei’s GPT3 equivalent PanGu was trained on 1.1TB of text, and ERNIE 3.0 was trained on 4TB of data). They train a BERT-style model to help do automatic filtering of the data. Their data comes from Common Crawl, Sogou News, SogouT, Encyclopedia, and Books.
How good is it? Yuan 1.0 does well on a variety of standard benchmarks. The most interesting result is on the quality of its text generation – here, the authors adopt the same approach as in the original GPT3 paper, where they generate text of different forms and see how well humans can distinguish generated text from ‘real’ text. The results are striking – humans are 49.57% accurate (compared to 52% for GPT3), meaning the Yuan 1.0 outputs are so good they’re indistinguishable from human-written text.