First-pass decoding with n-gram approximation of RNNLM: The problem of rare words

Abstract

Recurrent Neural Network Language Models (RNNLMs) can be utilized in first-pass decoding by approximating them to N- gram models. Although these approximated RNNLMs have shown to improve the Word Error Rate (WER), our experiments show that the word-based N-gram approximation seems to be poor at predicting words that occur with low frequency. In our ongoing work, we plan to switch from words to subword units for building approximated RNNLMs to improve the rare word prediction without compromising the general WER. To support this aim, we outline the various challenges and discuss the important factors for building better RNNLM approximations for the first-pass decoding.

Publication
In Machine Learning in Speech and Language Processing Workshop 2018 (MLSLP)
Date
Links