A novel hybrid segmentation technique for wheat rust disease detection is proposed. The method, called FERSPNET-50, combines panoptic segmentation with deep learning models to detect wheat rust diseases in real-time images. A dataset was collected under various weather conditions in Punjab, India. The GNet model classifies the collected images into three weather classes, which are then input into a Faster Region-based Convolutional Network (FRCNN) to detect wheat leaves and stems. A semi-automatic annotation method is used to generate ground truth rust lines for training. A Pyramid Scene Parsing Network (PSPNET) is employed to predict rust diseases at the local level. A pretrained deep CNN model is used to determine the orientation of rust segments within each detected patch, and patches that do not support local level patches are considered false positives. After classifying each patch, the severity of rust in wheat is calculated. Experimental results show that the proposed approach has a high precision (0.97) compared to state-of-the-art models like YOLOV4 (0.88) and RetinaNet (0.82). The study highlights the importance of early detection of wheat rust diseases to prevent grain quality loss. Traditional image processing techniques and machine learning methods have limitations in detecting rust diseases in real-time images. Deep learning techniques, particularly CNNs, offer better classification accuracy. However, the orientation of images can affect the performance of CNNs. The proposed FERSPNET-50 approach combines semantic and instance segmentation to achieve accurate and efficient detection of wheat rust diseases.A novel hybrid segmentation technique for wheat rust disease detection is proposed. The method, called FERSPNET-50, combines panoptic segmentation with deep learning models to detect wheat rust diseases in real-time images. A dataset was collected under various weather conditions in Punjab, India. The GNet model classifies the collected images into three weather classes, which are then input into a Faster Region-based Convolutional Network (FRCNN) to detect wheat leaves and stems. A semi-automatic annotation method is used to generate ground truth rust lines for training. A Pyramid Scene Parsing Network (PSPNET) is employed to predict rust diseases at the local level. A pretrained deep CNN model is used to determine the orientation of rust segments within each detected patch, and patches that do not support local level patches are considered false positives. After classifying each patch, the severity of rust in wheat is calculated. Experimental results show that the proposed approach has a high precision (0.97) compared to state-of-the-art models like YOLOV4 (0.88) and RetinaNet (0.82). The study highlights the importance of early detection of wheat rust diseases to prevent grain quality loss. Traditional image processing techniques and machine learning methods have limitations in detecting rust diseases in real-time images. Deep learning techniques, particularly CNNs, offer better classification accuracy. However, the orientation of images can affect the performance of CNNs. The proposed FERSPNET-50 approach combines semantic and instance segmentation to achieve accurate and efficient detection of wheat rust diseases.