This paper presents a novel approach to fruit detection using deep convolutional neural networks (DCNNs), specifically the Faster Region-based CNN (Faster R-CNN). The aim is to develop an accurate, fast, and reliable fruit detection system for autonomous agricultural robotic platforms, which are crucial for fruit yield estimation and automated harvesting. The authors adapt the Faster R-CNN model through transfer learning to combine multi-modal (RGB and Near-Infrared, NIR) imagery, exploring both early and late fusion methods. This results in a multi-modal Faster R-CNN model that achieves state-of-the-art performance, with an F1 score improving from 0.807 to 0.838 for sweet pepper detection. The approach is also quicker to deploy for new fruits, requiring bounding box annotation rather than pixel-level annotation. The model is retrained to detect seven different fruits, with the entire process taking four hours to annotate and train per fruit. The paper includes a detailed evaluation of the proposed method, comparing it to previous pixel-based methods and demonstrating its effectiveness in various experimental settings, including spatial-temporal independent condition experiments and detection of multiple fruits in the same scene. The findings are made available to the community through open datasets and tutorial documentation.This paper presents a novel approach to fruit detection using deep convolutional neural networks (DCNNs), specifically the Faster Region-based CNN (Faster R-CNN). The aim is to develop an accurate, fast, and reliable fruit detection system for autonomous agricultural robotic platforms, which are crucial for fruit yield estimation and automated harvesting. The authors adapt the Faster R-CNN model through transfer learning to combine multi-modal (RGB and Near-Infrared, NIR) imagery, exploring both early and late fusion methods. This results in a multi-modal Faster R-CNN model that achieves state-of-the-art performance, with an F1 score improving from 0.807 to 0.838 for sweet pepper detection. The approach is also quicker to deploy for new fruits, requiring bounding box annotation rather than pixel-level annotation. The model is retrained to detect seven different fruits, with the entire process taking four hours to annotate and train per fruit. The paper includes a detailed evaluation of the proposed method, comparing it to previous pixel-based methods and demonstrating its effectiveness in various experimental settings, including spatial-temporal independent condition experiments and detection of multiple fruits in the same scene. The findings are made available to the community through open datasets and tutorial documentation.