The paper introduces the Bayes tree, a novel data structure that provides a foundation for understanding graphical model inference algorithms and their connection to sparse matrix factorization methods. The Bayes tree is directed and maps more naturally to the square root information matrix of the simultaneous localization and mapping (SLAM) problem. The authors highlight three key insights provided by the Bayes tree: a better understanding of matrix factorization in terms of probability densities, a simple editing of the Bayes tree for incremental updates, and a new algorithm called iSAM2 for sparse nonlinear incremental optimization. iSAM2 achieves improvements in efficiency through incremental variable re-ordering and fluid relinearization, eliminating the need for periodic batch steps. The paper also includes a detailed evaluation of iSAM2 and comparisons with other state-of-the-art SLAM algorithms, demonstrating its favorable performance in both quality and efficiency on a range of real and simulated datasets.The paper introduces the Bayes tree, a novel data structure that provides a foundation for understanding graphical model inference algorithms and their connection to sparse matrix factorization methods. The Bayes tree is directed and maps more naturally to the square root information matrix of the simultaneous localization and mapping (SLAM) problem. The authors highlight three key insights provided by the Bayes tree: a better understanding of matrix factorization in terms of probability densities, a simple editing of the Bayes tree for incremental updates, and a new algorithm called iSAM2 for sparse nonlinear incremental optimization. iSAM2 achieves improvements in efficiency through incremental variable re-ordering and fluid relinearization, eliminating the need for periodic batch steps. The paper also includes a detailed evaluation of iSAM2 and comparisons with other state-of-the-art SLAM algorithms, demonstrating its favorable performance in both quality and efficiency on a range of real and simulated datasets.