The paper introduces a novel method for detecting tracking failures in point tracking tasks, which is a common computer vision challenge. The proposed method, called Forward-Backward (FB) error, leverages the assumption that correct tracking should be independent of the direction of time-flow. Specifically, the FB error measures the consistency between a forward-tracked trajectory and a backward-tracked validation trajectory. If these trajectories differ significantly, it indicates a tracking failure. The FB error is evaluated on synthetic data and compared to the Sum-of-Square Differences (SSD) method. Results show that the FB error outperforms SSD in detecting correct trajectories, especially at small thresholds. The paper also proposes a brute-force selection mechanism using the FB error to choose reliable feature points for tracking over an entire video sequence, demonstrating its effectiveness in selecting points that are consistently tracked. Additionally, the paper introduces the Median Flow tracker, which combines the FB error with the Normalized Cross-Correlation (NCC) error to improve tracking performance. The Median Flow tracker is tested on several benchmark sequences and shows superior performance compared to state-of-the-art methods, achieving the best results in three out of six sequences. The paper concludes by highlighting the advantages of the FB error and its potential for integration into other tracking frameworks.The paper introduces a novel method for detecting tracking failures in point tracking tasks, which is a common computer vision challenge. The proposed method, called Forward-Backward (FB) error, leverages the assumption that correct tracking should be independent of the direction of time-flow. Specifically, the FB error measures the consistency between a forward-tracked trajectory and a backward-tracked validation trajectory. If these trajectories differ significantly, it indicates a tracking failure. The FB error is evaluated on synthetic data and compared to the Sum-of-Square Differences (SSD) method. Results show that the FB error outperforms SSD in detecting correct trajectories, especially at small thresholds. The paper also proposes a brute-force selection mechanism using the FB error to choose reliable feature points for tracking over an entire video sequence, demonstrating its effectiveness in selecting points that are consistently tracked. Additionally, the paper introduces the Median Flow tracker, which combines the FB error with the Normalized Cross-Correlation (NCC) error to improve tracking performance. The Median Flow tracker is tested on several benchmark sequences and shows superior performance compared to state-of-the-art methods, achieving the best results in three out of six sequences. The paper concludes by highlighting the advantages of the FB error and its potential for integration into other tracking frameworks.