2003 February | Kenneth Kreutz-Delgado, Joseph F. Murray, Bhaskar D. Rao, Kjersti Engan, Te-Won Lee, Terrence J. Sejnowski
This paper presents algorithms for learning domain-specific overcomplete dictionaries to obtain maximum likelihood and maximum a posteriori (MAP) estimates based on Bayesian models with concave/Schur-concave (CSC) negative log priors. These priors are suitable for obtaining sparse representations of environmental signals within an appropriately chosen dictionary. The dictionary elements can be interpreted as concepts, features, or words that succinctly describe environmental events. The algorithms iterate between sparse representations found by variants of FOCUSS and an update of the dictionary using these representations. Experiments with synthetic data and natural images show improved performance over other independent component analysis (ICA) methods in terms of signal-to-noise ratios. In overcomplete cases, the true dictionary and sparse sources can be accurately recovered. Natural image tests show that overcomplete dictionaries have higher coding efficiency, with better compression and accuracy than complete dictionaries. The paper discusses the use of FOCUSS for sparse solutions to linear inverse problems, the development of algorithms for learning environmentally adapted dictionaries, and the application of Bayesian models with CSC priors for sparse representation. It also covers related work, including other methods for solving linear inverse problems, and discusses the properties of supergaussian priors and sparse coding. The paper concludes with a discussion of the FOCUSS algorithm, dictionary learning, and the use of CSC priors for sparse coding.This paper presents algorithms for learning domain-specific overcomplete dictionaries to obtain maximum likelihood and maximum a posteriori (MAP) estimates based on Bayesian models with concave/Schur-concave (CSC) negative log priors. These priors are suitable for obtaining sparse representations of environmental signals within an appropriately chosen dictionary. The dictionary elements can be interpreted as concepts, features, or words that succinctly describe environmental events. The algorithms iterate between sparse representations found by variants of FOCUSS and an update of the dictionary using these representations. Experiments with synthetic data and natural images show improved performance over other independent component analysis (ICA) methods in terms of signal-to-noise ratios. In overcomplete cases, the true dictionary and sparse sources can be accurately recovered. Natural image tests show that overcomplete dictionaries have higher coding efficiency, with better compression and accuracy than complete dictionaries. The paper discusses the use of FOCUSS for sparse solutions to linear inverse problems, the development of algorithms for learning environmentally adapted dictionaries, and the application of Bayesian models with CSC priors for sparse representation. It also covers related work, including other methods for solving linear inverse problems, and discusses the properties of supergaussian priors and sparse coding. The paper concludes with a discussion of the FOCUSS algorithm, dictionary learning, and the use of CSC priors for sparse coding.