This paper introduces the Object Relation Module (ORM), a novel module designed to model the relationships between objects in images. The ORM processes a set of objects simultaneously, integrating their appearance and geometric features to capture complex relationships. This module is lightweight, in-place, and does not require additional supervision, making it easy to integrate into existing deep learning frameworks. The authors demonstrate that the ORM significantly improves object recognition and duplicate removal steps in modern object detection pipelines, leading to the first fully end-to-end object detector. The ORM is shown to be effective across various state-of-the-art object detection architectures, including Faster R-CNN, FPN, and DCN. The paper also provides a detailed analysis of the ORM's effectiveness through ablation studies and comparisons with existing methods, highlighting its ability to learn meaningful object relationships and enhance detection accuracy. The code for the ORM is available on GitHub.This paper introduces the Object Relation Module (ORM), a novel module designed to model the relationships between objects in images. The ORM processes a set of objects simultaneously, integrating their appearance and geometric features to capture complex relationships. This module is lightweight, in-place, and does not require additional supervision, making it easy to integrate into existing deep learning frameworks. The authors demonstrate that the ORM significantly improves object recognition and duplicate removal steps in modern object detection pipelines, leading to the first fully end-to-end object detector. The ORM is shown to be effective across various state-of-the-art object detection architectures, including Faster R-CNN, FPN, and DCN. The paper also provides a detailed analysis of the ORM's effectiveness through ablation studies and comparisons with existing methods, highlighting its ability to learn meaningful object relationships and enhance detection accuracy. The code for the ORM is available on GitHub.