This paper presents a novel framework for jointly recognizing and linking visually ambiguous events. The approach combines observations along with linkage and global constraints in one probabilistic graphical model. The framework is based on a Bayesian network and uses Reversible Jump Markov Chain Monte Carlo (RJMCMC) to search the space of feasible explanations. The framework can be extended to multiple layers of linkage and is applied to the Bicycles problem, where the task is to recognize and link drop and pick events of bicycles in a rack over five days. The framework formulates the problem of recognizing and linking events as labeling a Bayesian network. The best global explanation is the Maximum a Posteriori (MAP) solution over a set of feasible explanations. The search space is sampled using RJMCMC, and a set of general move types is proposed that can be extended to multiple layers of linkage. Simulated annealing is used to find the MAP solution given all observations. The framework is evaluated on a challenging dataset consisting of seven sequences collected from two sites. The results show that the framework outperforms previous methods in terms of accuracy and convergence speed. The framework is able to correctly associate people to the bicycle they have dropped or picked and to link picks to earlier drops. The framework is also able to handle ambiguous situations where multiple individuals approach the racks, and the events are linked across temporal gaps. The framework is compared with other methods such as MHT and MCMC, and it is shown that the framework provides a more accurate and efficient solution. The framework is also able to handle the problem of linking events with a wide temporal gap, making it applicable to other domains. The framework is evaluated on the Bicycles problem and is shown to be effective in recognizing and linking events in a challenging dataset. The framework is also able to handle the problem of linking people entering a building to those departing. The framework is able to correctly associate people to the bicycle they have dropped or picked and to link picks to earlier drops. The framework is also able to handle ambiguous situations where multiple individuals approach the racks, and the events are linked across temporal gaps.This paper presents a novel framework for jointly recognizing and linking visually ambiguous events. The approach combines observations along with linkage and global constraints in one probabilistic graphical model. The framework is based on a Bayesian network and uses Reversible Jump Markov Chain Monte Carlo (RJMCMC) to search the space of feasible explanations. The framework can be extended to multiple layers of linkage and is applied to the Bicycles problem, where the task is to recognize and link drop and pick events of bicycles in a rack over five days. The framework formulates the problem of recognizing and linking events as labeling a Bayesian network. The best global explanation is the Maximum a Posteriori (MAP) solution over a set of feasible explanations. The search space is sampled using RJMCMC, and a set of general move types is proposed that can be extended to multiple layers of linkage. Simulated annealing is used to find the MAP solution given all observations. The framework is evaluated on a challenging dataset consisting of seven sequences collected from two sites. The results show that the framework outperforms previous methods in terms of accuracy and convergence speed. The framework is able to correctly associate people to the bicycle they have dropped or picked and to link picks to earlier drops. The framework is also able to handle ambiguous situations where multiple individuals approach the racks, and the events are linked across temporal gaps. The framework is compared with other methods such as MHT and MCMC, and it is shown that the framework provides a more accurate and efficient solution. The framework is also able to handle the problem of linking events with a wide temporal gap, making it applicable to other domains. The framework is evaluated on the Bicycles problem and is shown to be effective in recognizing and linking events in a challenging dataset. The framework is also able to handle the problem of linking people entering a building to those departing. The framework is able to correctly associate people to the bicycle they have dropped or picked and to link picks to earlier drops. The framework is also able to handle ambiguous situations where multiple individuals approach the racks, and the events are linked across temporal gaps.