Browsing by Author "Raffel, Colin"

Now showing 1 - 3 of 3
  • No Thumbnail Available
    Item
    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
    (arXiv, 2019-10-19) Raffel, Colin; Shazeer, Noam; Roberts, Adam; Lee, Katherine; Narang, Sharan; Matena, Michae; Zhou, Yanqi; Li, Wei; Liu, Peter J.
    Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts all text-based language problems into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled data sets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new ``Colossal Clean Crawled Corpus'', we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our data set, pre-trained models, and code.
  • No Thumbnail Available
    Item
    FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
    (arXiv, 2020-01-21) Sohn, Kihyuk; Berthelot, David; Li, Chun‑Liang; Zhang, Zizhao; Carlini, Nicholas; Cubuk, Ekin D; Kurakin, Alex; Zhang, Han; Raffel, Colin
    Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images. For a given image, the pseudo-label is only retained if the model produces a high-confidence prediction. The model is then trained to predict the pseudo-label when fed a strongly-augmented version of the same image. Despite its simplicity, we show that FixMatch achieves state-of-the-art performance across a variety of standard semi-supervised learning benchmarks, including 94.93% accuracy on CIFAR-10 with 250 labels and 88.61% accuracy with 40 -- just 4 labels per class. Since FixMatch bears many similarities to existing SSL methods that achieve worse performance, we carry out an extensive ablation study to tease apart the experimental factors that are most important to FixMatch's success. We make our code available at this https URL.
  • No Thumbnail Available
    Item
    mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
    (arXiv, 2020-10-22) Xue, Linting; Constant, Noah; Roberts, Adam; Kale, Mihir; Al‑Rfou, Rami; Siddhant, Aditya; Barua, Aditya; Raffel, Colin
    The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.