mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
No Thumbnail Available
Date
2020-10-22
Journal Title
Journal ISSN
Volume Title
Publisher
arXiv
Abstract
The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model checkpoints used in this work are publicly available.
Description
영어 위주의 T5를 101개 언어로 확장한 mT5 모델을 제안하며, 다국어 벤치마크에서 우수한 결과를 보여줍니다. 특히 제로샷 번역 시 특정 언어로 전이되는 문제를 완화한 전략도 포함됩니다
©2020 Google Research
Keywords
mT5, Multilingual NLP, Text-to-Text, Transfer Learning, Zero-shot