Large-scale pretraining for dialogue

About DialoGPT

DialoGPT adapts pretraining techniques to response generation using hundreds of Gigabytes of colloquial data. Like GPT-2, DialoGPT is formulated as an autoregressive (AR) language model, and uses a multi-layer transformer as model architecture. Unlike GPT-2, which trains on general text data, DialoGPT draws on 147M multi-turn dialogues extracted from Reddit discussion threads. The implementation is based on the huggingface pytorch-transformer and OpenAI GPT-2.

  • Commonsense question answering examples
who is the first president of the United StatesGeorge Washington
what is the boiling point of water?I think it's about 212 F.
which one is bigger, sun or moon?The sun .
which animal has black and white stripes?A black and white striped cat .
  • Handling context that does not appear in training examples (note that training examples are sampled from 2005 to 2017)
The trading war between China and US is still happening .I'm sure the Chinese government and the U . S . will be fine with that as long as they keep their arms on the table and the money flowing .
who won the world cup in 2018 ?Germany , I believe .
Nvidia's Titan RTX is really good .I have this card and i love it . No complaints .
Can Porsche beat Tesla with its new Taycan EV ?I don't think so . The Tesla is more powerful than the porsche .
  • Our bot's response to philosophy questions
What is the meaning of a good life ?I think it's that we're all connected to our past lives and the meaning of life is to live the way you want to and how you want to .
Which is more important -- being successful or being happy ?Being successful is just endless . Being happy is the only thing that matters .
Is peace the only way to stop war ?Not if the world is not willing to work with you . Peace is the ultimate goal , but peace requires more than that .
How much freedom should people have ?The right amount of freedom is the freedom to do as you please , as long as you don't hurt people or property .
  • Finally
What do you think of the DialoGPT repo ?It's pretty good , but it's very easy to find a bug .

Please start a issue if you spot any :)

This project is a joint project between MSR AI and Microsoft Dynamics 365 AI Research team

Source: https://www.microsoft.com/en-us/research/project/large-scale-pretraining-for-response-generation/

Ready to start building?

At Apideck we're building the world's biggest API network. Discover and integrate over 12,000 APIs.

Check out the API Tracker