This paper proposes a novel method for detecting tracking failures using Forward-Backward (FB) error. The method tracks a point forward and backward in time, then compares the two trajectories to detect discrepancies. The FB error is defined as the distance between the forward and backward trajectories. It is shown to reliably detect tracking failures and select reliable trajectories in video sequences. The method is complementary to normalized cross-correlation (NCC) and is applied to develop a new object tracker called Median Flow. The FB error is calculated by comparing the forward and backward trajectories of a point. The forward trajectory is obtained by tracking the point forward in time, while the backward trajectory is obtained by tracking the point backward from the last frame to the first. The distance between these two trajectories is used as the FB error. The main advantage of the FB error is that it can be applied to a wide range of trackers and is easy to implement. The method was evaluated on synthetic data, where it showed high performance in detecting correct tracking. The FB error was compared to SSD, and it was found to be significantly better in most cases. The method was also applied to select feature points from video sequences, where it was shown to be effective in identifying reliable points. The paper also proposes a new object tracker called Median Flow, which uses the FB error to improve tracking performance. The tracker uses a bounding box and a pair of images to estimate the motion of the bounding box. The tracker initializes a set of points on a rectangular grid within the bounding box and tracks them using the Lucas-Kanade tracker. The quality of the point predictions is estimated using the FB error, and the worst predictions are filtered out. The remaining predictions are used to estimate the displacement of the bounding box. The Median Flow tracker was compared to other state-of-the-art approaches on several benchmark sequences. It was shown to outperform these approaches in terms of the number of correctly tracked frames. The best results were achieved using a combination of FB and NCC errors. The paper concludes that the FB error is a reliable measure for tracking failure detection and that the Median Flow tracker achieves state-of-the-art performance on benchmark sequences.This paper proposes a novel method for detecting tracking failures using Forward-Backward (FB) error. The method tracks a point forward and backward in time, then compares the two trajectories to detect discrepancies. The FB error is defined as the distance between the forward and backward trajectories. It is shown to reliably detect tracking failures and select reliable trajectories in video sequences. The method is complementary to normalized cross-correlation (NCC) and is applied to develop a new object tracker called Median Flow. The FB error is calculated by comparing the forward and backward trajectories of a point. The forward trajectory is obtained by tracking the point forward in time, while the backward trajectory is obtained by tracking the point backward from the last frame to the first. The distance between these two trajectories is used as the FB error. The main advantage of the FB error is that it can be applied to a wide range of trackers and is easy to implement. The method was evaluated on synthetic data, where it showed high performance in detecting correct tracking. The FB error was compared to SSD, and it was found to be significantly better in most cases. The method was also applied to select feature points from video sequences, where it was shown to be effective in identifying reliable points. The paper also proposes a new object tracker called Median Flow, which uses the FB error to improve tracking performance. The tracker uses a bounding box and a pair of images to estimate the motion of the bounding box. The tracker initializes a set of points on a rectangular grid within the bounding box and tracks them using the Lucas-Kanade tracker. The quality of the point predictions is estimated using the FB error, and the worst predictions are filtered out. The remaining predictions are used to estimate the displacement of the bounding box. The Median Flow tracker was compared to other state-of-the-art approaches on several benchmark sequences. It was shown to outperform these approaches in terms of the number of correctly tracked frames. The best results were achieved using a combination of FB and NCC errors. The paper concludes that the FB error is a reliable measure for tracking failure detection and that the Median Flow tracker achieves state-of-the-art performance on benchmark sequences.