Large-scale pretraining for dialogue
DialoGPT adapts pretraining techniques to response generation using hundreds of Gigabytes of colloquial data. Like GPT-2, DialoGPT is formulated as an autoregressive (AR) language model, and uses a multi-layer transformer as model architecture. Unlike GPT-2, which trains on general text data, DialoGPT draws on 147M multi-turn dialogues extracted from Reddit discussion threads. The implementation is based on the huggingface pytorch-transformer and OpenAI GPT-2.
- Commonsense question answering examples
|User | Bot | |---------|--------| |who is the first president of the United States | George Washington | |what is the boiling point of water? | I think it's about 212 F. | |which one is bigger, sun or moon? | The sun . | |which animal has black and white stripes? | A black and white striped cat . |
- Handling context that does not appear in training examples (note that training examples are sampled from 2005 to 2017)
|User | Bot | |---------|--------| |The trading war between China and US is still happening . | I'm sure the Chinese government and the U . S . will be fine with that as long as they keep their arms on the table and the money flowing . | |who won the world cup in 2018 ? | Germany , I believe . | |Nvidia's Titan RTX is really good . | I have this card and i love it . No complaints . | |Can Porsche beat Tesla with its new Taycan EV ? | I don't think so . The Tesla is more powerful than the porsche .
- Our bot's response to philosophy questions
|User | Bot | |---------|--------| |What is the meaning of a good life ? | I think it's that we're all connected to our past lives and the meaning of life is to live the way you want to and how you want to . | |Which is more important -- being successful or being happy ? | Being successful is just endless . Being happy is the only thing that matters . | |Is peace the only way to stop war ? | Not if the world is not willing to work with you . Peace is the ultimate goal , but peace requires more than that . | | How much freedom should people have ? | The right amount of freedom is the freedom to do as you please , as long as you don't hurt people or property .|
|User | Bot | |---------|--------| |What do you think of the DialoGPT repo ? | It's pretty good , but it's very easy to find a bug . |
Please start a issue if you spot any :)
This project is a joint project between MSR AI and Microsoft Dynamics 365 AI Research team