This paper introduces PIN, a lightweight module that enables object localization capabilities in a frozen Vision-Language Model (VLM). The authors analyze the limitations of caption-based VLMs in object localization, noting that these models struggle to provide spatial information due to their training on caption-only data without explicit spatial grounding. To address this, the authors propose a simple yet effective Positional Insert (PIN), a learnable spatial prompt that is inserted into the frozen VLM without altering its parameters. The PIN is trained using a synthetic dataset generated by overlaying objects on background images, allowing the model to learn spatial relationships without requiring supervised data. The PIN is then used to enhance the VLM's ability to localize objects in images. The authors evaluate their approach on several benchmark datasets, including COCO, PVOC, and LVIS, and demonstrate that their method significantly improves the VLM's object localization performance. The PIN module is trained with a simple next-token prediction task on synthetic data, and it does not require any additional heads or projection layers. The results show that the PIN module enables the VLM to generate accurate bounding boxes for objects in images, even without any explicit supervision. The authors also show that their approach is effective across a variety of image types, including paintings, comics, and unique scenarios. The PIN module is lightweight, with only around 1.2 million parameters, making it an efficient solution for enabling object localization in VLMs. The authors conclude that their approach provides a simple and effective way to unlock the localization capabilities of caption-based VLMs without requiring specialized components or large annotated datasets.This paper introduces PIN, a lightweight module that enables object localization capabilities in a frozen Vision-Language Model (VLM). The authors analyze the limitations of caption-based VLMs in object localization, noting that these models struggle to provide spatial information due to their training on caption-only data without explicit spatial grounding. To address this, the authors propose a simple yet effective Positional Insert (PIN), a learnable spatial prompt that is inserted into the frozen VLM without altering its parameters. The PIN is trained using a synthetic dataset generated by overlaying objects on background images, allowing the model to learn spatial relationships without requiring supervised data. The PIN is then used to enhance the VLM's ability to localize objects in images. The authors evaluate their approach on several benchmark datasets, including COCO, PVOC, and LVIS, and demonstrate that their method significantly improves the VLM's object localization performance. The PIN module is trained with a simple next-token prediction task on synthetic data, and it does not require any additional heads or projection layers. The results show that the PIN module enables the VLM to generate accurate bounding boxes for objects in images, even without any explicit supervision. The authors also show that their approach is effective across a variety of image types, including paintings, comics, and unique scenarios. The PIN module is lightweight, with only around 1.2 million parameters, making it an efficient solution for enabling object localization in VLMs. The authors conclude that their approach provides a simple and effective way to unlock the localization capabilities of caption-based VLMs without requiring specialized components or large annotated datasets.