This paper presents a practical Bayesian framework for learning in feedforward networks, enabling objective comparisons between solutions, stopping rules for pruning or growing networks, and quantification of error bars. The framework uses Bayesian evidence to automatically penalize overcomplex models, embodying Occam's razor. It also allows for the objective choice of weight decay terms, regularization, and comparison with alternative models like splines or radial basis functions. The Bayesian approach provides a measure of the effective number of well-determined parameters in a model and quantifies error bars on network parameters and outputs.
The framework is based on the Bayesian interpolation method described in the companion paper. It addresses the issue of setting free parameters in neural networks, such as the number of hidden units and the regularizing constant α. The paper discusses the probabilistic interpretation of network learning, where the likelihood and prior are defined, and the posterior probability is derived. The framework allows for the evaluation of the evidence, which is used to compare alternative models and determine the most probable parameters.
The paper also discusses the challenges of evaluating the evidence for neural networks, where the objective function M is not quadratic and has multiple local minima. It introduces a local version of the evidence calculation, which is used to compare different solutions. The paper demonstrates the effectiveness of the Bayesian framework in a small interpolation problem, showing that the evidence can predict generalization ability and identify better solutions. It also discusses the relationship between the evidence and generalization error, showing that the evidence is a good predictor of generalization ability. The paper concludes that the Bayesian framework provides a practical and objective method for learning in feedforward networks.This paper presents a practical Bayesian framework for learning in feedforward networks, enabling objective comparisons between solutions, stopping rules for pruning or growing networks, and quantification of error bars. The framework uses Bayesian evidence to automatically penalize overcomplex models, embodying Occam's razor. It also allows for the objective choice of weight decay terms, regularization, and comparison with alternative models like splines or radial basis functions. The Bayesian approach provides a measure of the effective number of well-determined parameters in a model and quantifies error bars on network parameters and outputs.
The framework is based on the Bayesian interpolation method described in the companion paper. It addresses the issue of setting free parameters in neural networks, such as the number of hidden units and the regularizing constant α. The paper discusses the probabilistic interpretation of network learning, where the likelihood and prior are defined, and the posterior probability is derived. The framework allows for the evaluation of the evidence, which is used to compare alternative models and determine the most probable parameters.
The paper also discusses the challenges of evaluating the evidence for neural networks, where the objective function M is not quadratic and has multiple local minima. It introduces a local version of the evidence calculation, which is used to compare different solutions. The paper demonstrates the effectiveness of the Bayesian framework in a small interpolation problem, showing that the evidence can predict generalization ability and identify better solutions. It also discusses the relationship between the evidence and generalization error, showing that the evidence is a good predictor of generalization ability. The paper concludes that the Bayesian framework provides a practical and objective method for learning in feedforward networks.