This paper introduces a novel hybrid segmentation technique, FERSPNET-50, for real-time detection of wheat rust diseases (WRD) in Punjab, India. Wheat is a significant staple crop in India, with over 50% of the country's wheat grown in Punjab. WRD causes a significant decline in wheat quality, leading to annual losses of more than 10%. The proposed method combines a GNet model for classifying images into three weather conditions and a Faster Region-Based Convolutional Neural Network (FRCNN) for detecting wheat leaves and stems. A semi-automatic annotation method is used to generate ground truth rust lines, and a pyramid scene parsing network (PSPNET) is employed to predict rust diseases at the local level. A deep CNN model is also used to determine the orientation of rust segments, distinguishing false positives. The experimental results show that the FERSPNET-50 approach has a high precision of 0.97, outperforming state-of-the-art models like YOLOV4 (0.88) and RetinaNet (0.82). The study highlights the importance of early detection of WRD to minimize grain quality loss and improve agricultural efficiency.This paper introduces a novel hybrid segmentation technique, FERSPNET-50, for real-time detection of wheat rust diseases (WRD) in Punjab, India. Wheat is a significant staple crop in India, with over 50% of the country's wheat grown in Punjab. WRD causes a significant decline in wheat quality, leading to annual losses of more than 10%. The proposed method combines a GNet model for classifying images into three weather conditions and a Faster Region-Based Convolutional Neural Network (FRCNN) for detecting wheat leaves and stems. A semi-automatic annotation method is used to generate ground truth rust lines, and a pyramid scene parsing network (PSPNET) is employed to predict rust diseases at the local level. A deep CNN model is also used to determine the orientation of rust segments, distinguishing false positives. The experimental results show that the FERSPNET-50 approach has a high precision of 0.97, outperforming state-of-the-art models like YOLOV4 (0.88) and RetinaNet (0.82). The study highlights the importance of early detection of WRD to minimize grain quality loss and improve agricultural efficiency.