This paper presents an empirical study of the naive Bayes classifier, analyzing its performance under various data conditions. The naive Bayes classifier simplifies learning by assuming feature independence given the class. Despite this unrealistic assumption, it often performs well, competing with more sophisticated classifiers in practice. The study investigates the factors affecting naive Bayes performance, using Monte Carlo simulations to systematically analyze classification accuracy across randomly generated problems. It shows that low-entropy feature distributions yield good performance, and that naive Bayes works well for nearly-functional feature dependencies, performing best in two cases: completely independent features and functionally dependent features. Surprisingly, the accuracy of naive Bayes is not directly correlated with the degree of feature dependencies measured as class-conditional mutual information. Instead, a better predictor of accuracy is the information loss due to the independence assumption. The paper also explores the relationship between feature dependencies and classification error. It demonstrates that the information loss, defined as the difference between the mutual information between features and class under the true distribution and under the naive Bayes assumption, is a better predictor of classification error than the strength of feature dependencies. This is shown through experiments on various problem generators, including those with zero and non-zero Bayes risk. The study concludes that while naive Bayes has limitations, its performance is influenced by data characteristics such as distribution entropy and information loss. Further research is needed to better understand the relationship between information-theoretic metrics and the behavior of naive Bayes, as well as to improve approximation techniques for learning efficient Bayesian network classifiers and probabilistic inference.This paper presents an empirical study of the naive Bayes classifier, analyzing its performance under various data conditions. The naive Bayes classifier simplifies learning by assuming feature independence given the class. Despite this unrealistic assumption, it often performs well, competing with more sophisticated classifiers in practice. The study investigates the factors affecting naive Bayes performance, using Monte Carlo simulations to systematically analyze classification accuracy across randomly generated problems. It shows that low-entropy feature distributions yield good performance, and that naive Bayes works well for nearly-functional feature dependencies, performing best in two cases: completely independent features and functionally dependent features. Surprisingly, the accuracy of naive Bayes is not directly correlated with the degree of feature dependencies measured as class-conditional mutual information. Instead, a better predictor of accuracy is the information loss due to the independence assumption. The paper also explores the relationship between feature dependencies and classification error. It demonstrates that the information loss, defined as the difference between the mutual information between features and class under the true distribution and under the naive Bayes assumption, is a better predictor of classification error than the strength of feature dependencies. This is shown through experiments on various problem generators, including those with zero and non-zero Bayes risk. The study concludes that while naive Bayes has limitations, its performance is influenced by data characteristics such as distribution entropy and information loss. Further research is needed to better understand the relationship between information-theoretic metrics and the behavior of naive Bayes, as well as to improve approximation techniques for learning efficient Bayesian network classifiers and probabilistic inference.