This paper introduces a Bayesian Active Learning (BALD) approach for classification and preference learning. The method is based on information-theoretic principles, aiming to minimize uncertainty in the model parameters by selecting the most informative data points. The approach is applied to Gaussian Process Classification (GPC), a powerful non-parametric kernel-based model, and extended to preference learning. The key idea is to maximize the decrease in expected posterior entropy, which is equivalent to the conditional mutual information between the unknown output and the parameters. This allows the algorithm to select data points where the model is most uncertain about the output, while the parameters are confident. This is interpreted as seeking data points where the parameters under the posterior disagree the most. The method is implemented using a Gaussian approximation to the posterior, which allows for efficient computation. The algorithm is shown to perform well compared to other active learning methods, including decision-theoretic approaches, and has lower computational complexity. It is also extended to preference learning by reformulating the problem as a classification task. The paper also discusses related methodologies, including the Informative Vector Machine (IVM), Maximum Entropy Sampling (MES), and mutual information-based approaches. It compares these methods to BALD and highlights the advantages of the proposed approach, particularly in terms of accuracy and computational efficiency. Experiments on various datasets show that BALD outperforms other active learning methods in terms of classification accuracy and efficiency. The method is shown to be effective in both classification and preference learning tasks, and is able to handle complex models with infinite-dimensional parameter spaces. The paper concludes that BALD is a promising approach for active learning in non-parametric models, with minimal approximations and good computational performance.This paper introduces a Bayesian Active Learning (BALD) approach for classification and preference learning. The method is based on information-theoretic principles, aiming to minimize uncertainty in the model parameters by selecting the most informative data points. The approach is applied to Gaussian Process Classification (GPC), a powerful non-parametric kernel-based model, and extended to preference learning. The key idea is to maximize the decrease in expected posterior entropy, which is equivalent to the conditional mutual information between the unknown output and the parameters. This allows the algorithm to select data points where the model is most uncertain about the output, while the parameters are confident. This is interpreted as seeking data points where the parameters under the posterior disagree the most. The method is implemented using a Gaussian approximation to the posterior, which allows for efficient computation. The algorithm is shown to perform well compared to other active learning methods, including decision-theoretic approaches, and has lower computational complexity. It is also extended to preference learning by reformulating the problem as a classification task. The paper also discusses related methodologies, including the Informative Vector Machine (IVM), Maximum Entropy Sampling (MES), and mutual information-based approaches. It compares these methods to BALD and highlights the advantages of the proposed approach, particularly in terms of accuracy and computational efficiency. Experiments on various datasets show that BALD outperforms other active learning methods in terms of classification accuracy and efficiency. The method is shown to be effective in both classification and preference learning tasks, and is able to handle complex models with infinite-dimensional parameter spaces. The paper concludes that BALD is a promising approach for active learning in non-parametric models, with minimal approximations and good computational performance.