This paper presents a comprehensive survey of semantic communication networks (SemComNet), covering its architecture, security, and privacy aspects. SemComNet is a communication paradigm that aims to convey the essential meaning of source data rather than merely transmitting raw bits. It supports semantic-oriented transmission, efficient spectrum utilization, and flexible networking among collaborative agents. SemComNet is expected to enable intelligent applications such as autonomous driving and the Metaverse. However, due to its reliance on advanced technologies like AI and knowledge graphs, it introduces new and unexpected threats that hinder its development. Additionally, the heterogeneous nature of SemComNet introduces critical security and privacy challenges. The paper introduces a novel three-layer architecture of SemComNet for multi-agent interaction, comprising the control layer, semantic transmission layer, and cognitive sensing layer. It discusses the working modes and enabling technologies of SemComNet. Based on the layered architecture, the paper outlines a taxonomy of security and privacy threats in SemComNet, discussing state-of-the-art defense approaches. Finally, it presents future research directions, clarifying the path toward establishing intelligent, robust, and green SemComNet. The paper identifies three main security and privacy threats in SemComNet: 1) attacks on the control layer, such as sponge examples attacks and knowledge bases poisoning attacks; 2) attacks on the semantic transmission layer, such as semantic adversarial attacks and false data injection attacks; and 3) attacks on the cognitive sensing layer, such as semantic model theft and knowledge base desynchronization attacks. These threats pose significant challenges to the security and privacy of SemComNet. The paper discusses existing defense solutions and highlights the need for further research to address these challenges. The survey is the first to comprehensively cover the fundamentals of SemComNet, alongside a detailed analysis of its security and privacy issues.This paper presents a comprehensive survey of semantic communication networks (SemComNet), covering its architecture, security, and privacy aspects. SemComNet is a communication paradigm that aims to convey the essential meaning of source data rather than merely transmitting raw bits. It supports semantic-oriented transmission, efficient spectrum utilization, and flexible networking among collaborative agents. SemComNet is expected to enable intelligent applications such as autonomous driving and the Metaverse. However, due to its reliance on advanced technologies like AI and knowledge graphs, it introduces new and unexpected threats that hinder its development. Additionally, the heterogeneous nature of SemComNet introduces critical security and privacy challenges. The paper introduces a novel three-layer architecture of SemComNet for multi-agent interaction, comprising the control layer, semantic transmission layer, and cognitive sensing layer. It discusses the working modes and enabling technologies of SemComNet. Based on the layered architecture, the paper outlines a taxonomy of security and privacy threats in SemComNet, discussing state-of-the-art defense approaches. Finally, it presents future research directions, clarifying the path toward establishing intelligent, robust, and green SemComNet. The paper identifies three main security and privacy threats in SemComNet: 1) attacks on the control layer, such as sponge examples attacks and knowledge bases poisoning attacks; 2) attacks on the semantic transmission layer, such as semantic adversarial attacks and false data injection attacks; and 3) attacks on the cognitive sensing layer, such as semantic model theft and knowledge base desynchronization attacks. These threats pose significant challenges to the security and privacy of SemComNet. The paper discusses existing defense solutions and highlights the need for further research to address these challenges. The survey is the first to comprehensively cover the fundamentals of SemComNet, alongside a detailed analysis of its security and privacy issues.