This paper presents evidence that efficient exploration significantly improves the performance of large language models (LLMs) when gathering human feedback. The authors propose an agent that uses double Thompson sampling with an epistemic neural network (ENN) to generate queries, which allows for more efficient exploration and better performance with fewer queries. The study compares different exploration algorithms, including passive exploration, Boltzmann exploration, infomax, and double Thompson sampling. The results show that active exploration, particularly double Thompson sampling, leads to higher performance with fewer queries. The study also highlights the importance of uncertainty estimation and the choice of exploration algorithm in achieving high performance. The experiments demonstrate that efficient exploration can significantly reduce the number of queries needed to reach high performance levels, potentially accelerating the development of superhuman ingenuity in LLMs. The paper also discusses the architecture and training of reward models, including point estimate and ENN models, and how they contribute to the effectiveness of exploration algorithms. The results show that ENN models provide better uncertainty estimates, leading to more accurate and efficient exploration. The study concludes that efficient exploration is a critical component in improving the performance of LLMs through human feedback.This paper presents evidence that efficient exploration significantly improves the performance of large language models (LLMs) when gathering human feedback. The authors propose an agent that uses double Thompson sampling with an epistemic neural network (ENN) to generate queries, which allows for more efficient exploration and better performance with fewer queries. The study compares different exploration algorithms, including passive exploration, Boltzmann exploration, infomax, and double Thompson sampling. The results show that active exploration, particularly double Thompson sampling, leads to higher performance with fewer queries. The study also highlights the importance of uncertainty estimation and the choice of exploration algorithm in achieving high performance. The experiments demonstrate that efficient exploration can significantly reduce the number of queries needed to reach high performance levels, potentially accelerating the development of superhuman ingenuity in LLMs. The paper also discusses the architecture and training of reward models, including point estimate and ENN models, and how they contribute to the effectiveness of exploration algorithms. The results show that ENN models provide better uncertainty estimates, leading to more accurate and efficient exploration. The study concludes that efficient exploration is a critical component in improving the performance of LLMs through human feedback.