This paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer, a novel neural network component that significantly increases model capacity while maintaining computational efficiency. The MoE layer consists of multiple feed-forward sub-networks (experts) and a trainable gating network that selects a sparse combination of these experts for each input. This approach allows for massive increases in model capacity, with the authors achieving over 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. The MoE layer is applied to tasks such as language modeling and machine translation, where model capacity is critical for absorbing vast amounts of training data. The authors demonstrate that their models achieve significantly better results than state-of-the-art models at lower computational costs on large language modeling and machine translation benchmarks. The paper addresses several challenges in conditional computation, including the shrinking batch problem, network bandwidth limitations, and expert utilization imbalance. The authors propose techniques to increase batch size, optimize network bandwidth, and balance expert utilization through additional loss functions. These solutions enable the efficient training of very large models. The authors evaluate their models on the 1-Billion-Word Language Modeling Benchmark and the 100-Billion-Word Google News Corpus. On the 1-Billion-Word benchmark, their models achieve significantly lower perplexity than previous state-of-the-art models, with the largest model (with 4096 experts) achieving a 24% lower perplexity on the test set. On the 100-Billion-Word corpus, their models also show significant improvements in performance. In machine translation tasks, the authors achieve BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks, respectively, which are significantly higher than previous results. Their multilingual machine translation model also outperforms other models on multiple language pairs. The paper concludes that conditional computation can significantly improve model capacity and performance, and that the proposed MoE layer is a promising approach for building large-scale neural networks. The authors believe that conditional computation will continue to be an important area of research in the future.This paper introduces a Sparsely-Gated Mixture-of-Experts (MoE) layer, a novel neural network component that significantly increases model capacity while maintaining computational efficiency. The MoE layer consists of multiple feed-forward sub-networks (experts) and a trainable gating network that selects a sparse combination of these experts for each input. This approach allows for massive increases in model capacity, with the authors achieving over 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. The MoE layer is applied to tasks such as language modeling and machine translation, where model capacity is critical for absorbing vast amounts of training data. The authors demonstrate that their models achieve significantly better results than state-of-the-art models at lower computational costs on large language modeling and machine translation benchmarks. The paper addresses several challenges in conditional computation, including the shrinking batch problem, network bandwidth limitations, and expert utilization imbalance. The authors propose techniques to increase batch size, optimize network bandwidth, and balance expert utilization through additional loss functions. These solutions enable the efficient training of very large models. The authors evaluate their models on the 1-Billion-Word Language Modeling Benchmark and the 100-Billion-Word Google News Corpus. On the 1-Billion-Word benchmark, their models achieve significantly lower perplexity than previous state-of-the-art models, with the largest model (with 4096 experts) achieving a 24% lower perplexity on the test set. On the 100-Billion-Word corpus, their models also show significant improvements in performance. In machine translation tasks, the authors achieve BLEU scores of 40.56 and 26.03 on the WMT'14 En→Fr and En→De benchmarks, respectively, which are significantly higher than previous results. Their multilingual machine translation model also outperforms other models on multiple language pairs. The paper concludes that conditional computation can significantly improve model capacity and performance, and that the proposed MoE layer is a promising approach for building large-scale neural networks. The authors believe that conditional computation will continue to be an important area of research in the future.