This paper addresses the problem of automatically recognizing pictured dishes by introducing a novel method to mine discriminative parts using Random Forests (RF). The method allows for the simultaneous mining of parts for all classes and shares knowledge among them. To improve efficiency, only patches aligned with image superpixels are considered, which are called components. The performance of the RF component mining for food recognition is evaluated using a new dataset of 101 food categories with 101,000 images. The model achieves an average accuracy of 50.76%, outperforming alternative classification methods except for CNN by 11.88% and existing discriminative part-mining algorithms by 8.13%. On the challenging MIT-Indoor dataset, the method compares favorably to other state-of-the-art component-based classification methods. The paper also introduces a novel, large-scale, publicly available dataset for real-world food recognition, named *Food-101*. The contributions of the paper include a novel discriminative part mining method based on RF, a superpixel-based patch sampling strategy, and a new dataset for food recognition.This paper addresses the problem of automatically recognizing pictured dishes by introducing a novel method to mine discriminative parts using Random Forests (RF). The method allows for the simultaneous mining of parts for all classes and shares knowledge among them. To improve efficiency, only patches aligned with image superpixels are considered, which are called components. The performance of the RF component mining for food recognition is evaluated using a new dataset of 101 food categories with 101,000 images. The model achieves an average accuracy of 50.76%, outperforming alternative classification methods except for CNN by 11.88% and existing discriminative part-mining algorithms by 8.13%. On the challenging MIT-Indoor dataset, the method compares favorably to other state-of-the-art component-based classification methods. The paper also introduces a novel, large-scale, publicly available dataset for real-world food recognition, named *Food-101*. The contributions of the paper include a novel discriminative part mining method based on RF, a superpixel-based patch sampling strategy, and a new dataset for food recognition.