ctrl by Salesforce

A Conditional Transformer Language Model for Controllable Generation

About ctrl by Salesforce

Large-scale language models show promising text generation capabilities, but users cannot easily control this generation process. Salesforce released CTRL, a 1.6 billion-parameter conditional transformer language model, trained to condition on control codes that specify domain, subdomain, entities, relationships between entities, dates, and task-specific behavior. Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation.

The code currently supports two functionalities:

  • Generating from a trained model, two models are available for download - one with a sequence length of 256 and another with a sequence length of 512 -- they are trained with word-level vocabularies and through a sliding window approach can generate well beyond their trained sequence lengths.
  • Source attribution - given a prompt, prints the perplexity of the prompt conditional on each domain control code (see Section 5 of the paper).

ctrl by Salesforce screenshots