Towards a Human-like Open-Domain Chatbot
No Thumbnail Available
Date
2020-01-27
Journal Title
Journal ISSN
Volume Title
Publisher
arXiv
Abstract
We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Average (SSA), which captures key elements of a human-like multi-turn conversation. Our experiments show strong correlation between perplexity and SSA. The fact that the best perplexity end-to-end trained Meena scores high on SSA (72% on multi-turn evaluation) suggests that a human-level SSA of 86% is potentially within reach if we can better optimize perplexity. Additionally, the full version of Meena (with a filtering mechanism and tuned decoding) scores 79% SSA, 23% higher in absolute SSA than the existing chatbots we evaluated.
Description
2.6B 파라미터 Meena 모델을 제안하며, SSA(Sensibleness and Specificity Average)라는 새로운 평가 지표를 통해 다중턴 대화 성능을 평가합니다. 최고 성능 모델은 SSA 79%를 기록하며, 인간 수준(86%)에 근접하는 가능성을 제시합니다
©2020 Google Research
Keywords
Meena, Open-Domain Chatbot, SSA, Multi-turn Conversation, Dialogue Systems