This study investigates the effectiveness of three data drift detection methods in real-world medical imaging data, focusing on both natural and synthetic data drift. The research highlights the importance of monitoring input data for systemic changes in distributions, which can significantly impact model performance and patient safety. The study uses a dataset of 239,235 chest radiographs from Trillium Health Partners, including both pre- and post-COVID-19 periods. The methods evaluated include tracking model performance, image-data-based drift detection (TorchXRay Vision AutoEncoder), model output-based drift detection (Black Box Shift Detection), and a combined approach (TAE + BBSD). Key findings include: 1. **Model Performance as a Proxy**: Monitoring model performance alone is not an effective method for detecting data drift, as it fails to capture clinically obvious changes, such as the introduction of COVID-19 in X-rays. 2. **Sensitivity to Data Drift**: The sensitivity of drift detection methods depends on the size of the dataset and the specific features being enriched. Larger datasets generally improve sensitivity, but the effectiveness varies with the type of drift. 3. **Type of Drift**: Different types of synthetic drifts, such as changes in patient demographics and pathologies, affect the sensitivity of drift detection methods differently. For example, detecting changes in younger patients is more sensitive than changes in older patients. 4. **Practical Implications**: The study emphasizes the need for practical guidelines and best practices for monitoring data drift in clinical AI deployments, especially in high-stakes scenarios like healthcare. It also highlights the limitations of current methods, such as the lack of explainability and potential biases in drift detection algorithms. The research underscores the critical role of data drift detection in maintaining the reliability and safety of deployed AI models, particularly in the context of medical imaging.This study investigates the effectiveness of three data drift detection methods in real-world medical imaging data, focusing on both natural and synthetic data drift. The research highlights the importance of monitoring input data for systemic changes in distributions, which can significantly impact model performance and patient safety. The study uses a dataset of 239,235 chest radiographs from Trillium Health Partners, including both pre- and post-COVID-19 periods. The methods evaluated include tracking model performance, image-data-based drift detection (TorchXRay Vision AutoEncoder), model output-based drift detection (Black Box Shift Detection), and a combined approach (TAE + BBSD). Key findings include: 1. **Model Performance as a Proxy**: Monitoring model performance alone is not an effective method for detecting data drift, as it fails to capture clinically obvious changes, such as the introduction of COVID-19 in X-rays. 2. **Sensitivity to Data Drift**: The sensitivity of drift detection methods depends on the size of the dataset and the specific features being enriched. Larger datasets generally improve sensitivity, but the effectiveness varies with the type of drift. 3. **Type of Drift**: Different types of synthetic drifts, such as changes in patient demographics and pathologies, affect the sensitivity of drift detection methods differently. For example, detecting changes in younger patients is more sensitive than changes in older patients. 4. **Practical Implications**: The study emphasizes the need for practical guidelines and best practices for monitoring data drift in clinical AI deployments, especially in high-stakes scenarios like healthcare. It also highlights the limitations of current methods, such as the lack of explainability and potential biases in drift detection algorithms. The research underscores the critical role of data drift detection in maintaining the reliability and safety of deployed AI models, particularly in the context of medical imaging.