The paper presents two approaches to improve sequence learning with recurrent networks using unlabeled data: predicting the next element in a sequence (a language model) and using a sequence autoencoder. These methods can serve as pretraining steps for supervised sequence learning algorithms, enhancing stability and generalization. The authors demonstrate that long short-term memory (LSTM) recurrent networks trained with these pretraining methods achieve better performance on various text classification tasks, such as IMDB, DBpedia, and 20 Newsgroups. They also show that using more unlabeled data from related tasks can significantly improve the generalization of subsequent supervised models, equivalent to adding more labeled data. The semi-supervised approach is compared to other unsupervised sequence learning methods and is found to have advantages in terms of ease of fine-tuning. The paper includes detailed experimental results and discusses the effectiveness of the proposed methods in various benchmarks.The paper presents two approaches to improve sequence learning with recurrent networks using unlabeled data: predicting the next element in a sequence (a language model) and using a sequence autoencoder. These methods can serve as pretraining steps for supervised sequence learning algorithms, enhancing stability and generalization. The authors demonstrate that long short-term memory (LSTM) recurrent networks trained with these pretraining methods achieve better performance on various text classification tasks, such as IMDB, DBpedia, and 20 Newsgroups. They also show that using more unlabeled data from related tasks can significantly improve the generalization of subsequent supervised models, equivalent to adding more labeled data. The semi-supervised approach is compared to other unsupervised sequence learning methods and is found to have advantages in terms of ease of fine-tuning. The paper includes detailed experimental results and discusses the effectiveness of the proposed methods in various benchmarks.