Large-scale pretraining for dialogue

About DialoGPT

DialoGPT adapts pretraining techniques to response generation using hundreds of Gigabytes of colloquial data. Like GPT-2, DialoGPT is formulated as an autoregressive (AR) language model, and uses a multi-layer transformer as model architecture. Unlike GPT-2, which trains on general text data, DialoGPT draws on 147M multi-turn dialogues extracted from Reddit discussion threads. The implementation is based on the huggingface pytorch-transformer and OpenAI GPT-2.

  • Commonsense question answering examples

|User | Bot | |---------|--------| |who is the first president of the United States | George Washington | |what is the boiling point of water? | I think it's about 212 F. | |which one is bigger, sun or moon? | The sun . | |which animal has black and white stripes? | A black and white striped cat . |

  • Handling context that does not appear in training examples (note that training examples are sampled from 2005 to 2017)

|User | Bot | |---------|--------| |The trading war between China and US is still happening . | I'm sure the Chinese government and the U . S . will be fine with that as long as they keep their arms on the table and the money flowing . | |who won the world cup in 2018 ? | Germany , I believe . | |Nvidia's Titan RTX is really good . | I have this card and i love it . No complaints . | |Can Porsche beat Tesla with its new Taycan EV ? | I don't think so . The Tesla is more powerful than the porsche .

  • Our bot's response to philosophy questions

|User | Bot | |---------|--------| |What is the meaning of a good life ? | I think it's that we're all connected to our past lives and the meaning of life is to live the way you want to and how you want to . | |Which is more important -- being successful or being happy ? | Being successful is just endless . Being happy is the only thing that matters . | |Is peace the only way to stop war ? | Not if the world is not willing to work with you . Peace is the ultimate goal , but peace requires more than that . | | How much freedom should people have ? | The right amount of freedom is the freedom to do as you please , as long as you don't hurt people or property .|

  • Finally

|User | Bot | |---------|--------| |What do you think of the DialoGPT repo ? | It's pretty good , but it's very easy to find a bug . |

Please start a issue if you spot any :)

This project is a joint project between MSR AI and Microsoft Dynamics 365 AI Research team

Source: https://www.microsoft.com/en-us/research/project/large-scale-pretraining-for-response-generation/

Ready to start building?

At Apideck we're building the world's biggest API network. Discover and integrate over 12,000 APIs.

Check out the API Tracker