Recurrent Neural Network for Text Classification with Multi-Task Learning

Recurrent Neural Network for Text Classification with Multi-Task Learning

17 May 2016 | Pengfei Liu, Xipeng Qiu*, Xuanjing Huang
This paper proposes three recurrent neural network (RNN) based multi-task learning models for text classification. The models share information between tasks through different mechanisms, aiming to improve performance by leveraging related tasks. The first model uses a single shared layer for all tasks, the second uses different layers for different tasks but allows information exchange between them, and the third introduces a shared layer for all tasks. A gating mechanism is introduced to selectively utilize shared information. The models are trained jointly on multiple related tasks, and experiments on four text classification tasks show that the joint learning improves performance compared to single-task learning. The paper also discusses the use of RNNs for specific-task text classification, highlighting the effectiveness of LSTM networks in handling variable-length text. The models are trained using backpropagation and gradient-based optimization, with parameters fine-tuned for each task. The shared layer in Model-III is pre-trained using an unsupervised language model to enhance performance. Experiments on four text classification tasks (SST-1, SST-2, SUBJ, and IMDB) show that the proposed models outperform state-of-the-art neural models. The shared-layer architecture achieves the best performance, with improvements in accuracy compared to single-task learning. The models are also compared with other neural models such as NBOW, RNTN, DCNN, and Tree-LSTM, demonstrating their effectiveness. The paper also includes a case study analyzing how the shared-layer LSTM model processes text, showing that it can capture complex semantic structures and improve predictions. Error analysis reveals that the model struggles with complicated sentence structures and sentences requiring reasoning. Future work includes investigating other sharing mechanisms for different tasks.This paper proposes three recurrent neural network (RNN) based multi-task learning models for text classification. The models share information between tasks through different mechanisms, aiming to improve performance by leveraging related tasks. The first model uses a single shared layer for all tasks, the second uses different layers for different tasks but allows information exchange between them, and the third introduces a shared layer for all tasks. A gating mechanism is introduced to selectively utilize shared information. The models are trained jointly on multiple related tasks, and experiments on four text classification tasks show that the joint learning improves performance compared to single-task learning. The paper also discusses the use of RNNs for specific-task text classification, highlighting the effectiveness of LSTM networks in handling variable-length text. The models are trained using backpropagation and gradient-based optimization, with parameters fine-tuned for each task. The shared layer in Model-III is pre-trained using an unsupervised language model to enhance performance. Experiments on four text classification tasks (SST-1, SST-2, SUBJ, and IMDB) show that the proposed models outperform state-of-the-art neural models. The shared-layer architecture achieves the best performance, with improvements in accuracy compared to single-task learning. The models are also compared with other neural models such as NBOW, RNTN, DCNN, and Tree-LSTM, demonstrating their effectiveness. The paper also includes a case study analyzing how the shared-layer LSTM model processes text, showing that it can capture complex semantic structures and improve predictions. Error analysis reveals that the model struggles with complicated sentence structures and sentences requiring reasoning. Future work includes investigating other sharing mechanisms for different tasks.
Reach us at info@study.space