This paper presents a novel approach for polyp segmentation using a hybrid architecture combining residual transformer layers and a hybrid loss function. The proposed method, named ResTransSeg-Net, leverages the strengths of both residual learning and transformer layers to improve the accuracy and efficiency of polyp segmentation. The architecture uses a pre-trained ResNet101 as an encoder and a transformer with three decoding layers as the decoder. The hybrid loss function combines focal Tversky loss, binary cross-entropy, and Jaccard index to reduce image-wise and pixel-wise differences while enhancing regional consistencies. Experimental results on ten diverse datasets demonstrate the effectiveness of the proposed method, achieving high Dice similarity (0.9048), recall (0.9041), precision (0.9057), and F2 score (0.8993). comparisons with state-of-the-art methods show superior performance. The study highlights the benefits of combining residual and transformer architectures for polyp segmentation, particularly in improving the robustness and generalization of the model.This paper presents a novel approach for polyp segmentation using a hybrid architecture combining residual transformer layers and a hybrid loss function. The proposed method, named ResTransSeg-Net, leverages the strengths of both residual learning and transformer layers to improve the accuracy and efficiency of polyp segmentation. The architecture uses a pre-trained ResNet101 as an encoder and a transformer with three decoding layers as the decoder. The hybrid loss function combines focal Tversky loss, binary cross-entropy, and Jaccard index to reduce image-wise and pixel-wise differences while enhancing regional consistencies. Experimental results on ten diverse datasets demonstrate the effectiveness of the proposed method, achieving high Dice similarity (0.9048), recall (0.9041), precision (0.9057), and F2 score (0.8993). comparisons with state-of-the-art methods show superior performance. The study highlights the benefits of combining residual and transformer architectures for polyp segmentation, particularly in improving the robustness and generalization of the model.