This paper revisits the problem of comparing representations in neural networks using canonical correlation analysis (CCA) and introduces a new similarity index called Centered Kernel Alignment (CKA). The authors argue that CCA, while useful, has limitations when applied to high-dimensional data, as it can only measure meaningful similarities if the dimensionality of the representations is less than the number of data points. CKA, on the other hand, is designed to overcome these limitations by measuring the relationship between representational similarity matrices, which do not suffer from the same constraints as CCA. The paper discusses the invariance properties of similarity indexes and their implications for measuring neural network representations. It highlights that invariance to orthogonal transformations is desirable for neural networks trained by gradient descent, as it preserves the symmetries of the networks. The authors also show that CKA can reliably identify correspondences between representations in networks trained from different initializations, which is a limitation of other similarity indexes like CCA and linear regression. The paper provides a detailed analysis of CKA, including its relationship to other similarity measures such as linear regression, CCA, and singular vector CCA (SVCCA). It demonstrates that CKA is equivalent to centered kernel alignment (CKA) and is closely related to CCA. The authors also show that CKA can reveal consistent relationships between layers in different network architectures and across different datasets, which other methods fail to do. Finally, the paper presents empirical results to validate the effectiveness of CKA in identifying correspondences and understanding the structure of neural network representations. It concludes by discussing the potential of CKA for further research and its advantages over previous methods in finding correspondences between learned representations in hidden layers of neural networks.This paper revisits the problem of comparing representations in neural networks using canonical correlation analysis (CCA) and introduces a new similarity index called Centered Kernel Alignment (CKA). The authors argue that CCA, while useful, has limitations when applied to high-dimensional data, as it can only measure meaningful similarities if the dimensionality of the representations is less than the number of data points. CKA, on the other hand, is designed to overcome these limitations by measuring the relationship between representational similarity matrices, which do not suffer from the same constraints as CCA. The paper discusses the invariance properties of similarity indexes and their implications for measuring neural network representations. It highlights that invariance to orthogonal transformations is desirable for neural networks trained by gradient descent, as it preserves the symmetries of the networks. The authors also show that CKA can reliably identify correspondences between representations in networks trained from different initializations, which is a limitation of other similarity indexes like CCA and linear regression. The paper provides a detailed analysis of CKA, including its relationship to other similarity measures such as linear regression, CCA, and singular vector CCA (SVCCA). It demonstrates that CKA is equivalent to centered kernel alignment (CKA) and is closely related to CCA. The authors also show that CKA can reveal consistent relationships between layers in different network architectures and across different datasets, which other methods fail to do. Finally, the paper presents empirical results to validate the effectiveness of CKA in identifying correspondences and understanding the structure of neural network representations. It concludes by discussing the potential of CKA for further research and its advantages over previous methods in finding correspondences between learned representations in hidden layers of neural networks.