This paper explores novel philosophical questions raised by recent progress in large language models (LLMs), focusing on interpretability, multimodal and modular extensions, consciousness, and reproducibility. The authors discuss the challenges of understanding LLMs' internal mechanisms, particularly through causal intervention methods, and the implications of these models for modeling human cognition. They critique the limitations of benchmarking methods, which often fail to accurately measure LLMs' capabilities due to issues like saturation, gamification, and data contamination. The paper also delves into the concept of mechanistic interpretability, aiming to uncover the causal structure underlying LLMs' behavior. This involves methods such as probing, attribution, and causal interventions, with a focus on identifying and validating the functional roles of specific internal features and circuits. The authors illustrate these concepts through case studies, including the discovery of 'induction heads' in language models, which demonstrate pattern completion and generalization capabilities. Overall, the paper highlights the need for more sophisticated methods to understand and interpret LLMs' internal processes and their potential applications in cognitive science.This paper explores novel philosophical questions raised by recent progress in large language models (LLMs), focusing on interpretability, multimodal and modular extensions, consciousness, and reproducibility. The authors discuss the challenges of understanding LLMs' internal mechanisms, particularly through causal intervention methods, and the implications of these models for modeling human cognition. They critique the limitations of benchmarking methods, which often fail to accurately measure LLMs' capabilities due to issues like saturation, gamification, and data contamination. The paper also delves into the concept of mechanistic interpretability, aiming to uncover the causal structure underlying LLMs' behavior. This involves methods such as probing, attribution, and causal interventions, with a focus on identifying and validating the functional roles of specific internal features and circuits. The authors illustrate these concepts through case studies, including the discovery of 'induction heads' in language models, which demonstrate pattern completion and generalization capabilities. Overall, the paper highlights the need for more sophisticated methods to understand and interpret LLMs' internal processes and their potential applications in cognitive science.