This paper explores how semantic meaning is encoded in the representation spaces of large language models (LLMs). The authors address two fundamental questions: how categorical concepts are represented, and how hierarchical relations between concepts are encoded. They extend the linear representation hypothesis to show that simple categorical concepts are represented as simplices, and hierarchically related concepts are orthogonal in a specific sense. Complex concepts, which reflect hierarchical structures, are represented as polytopes constructed from direct sums of simplices. The theoretical findings are validated using the Gemma LLM, where representations for 957 hierarchically related concepts are estimated using WordNet data. The results provide a foundation for understanding how high-level semantic concepts are encoded in LLMs, with implications for model interpretability and control.This paper explores how semantic meaning is encoded in the representation spaces of large language models (LLMs). The authors address two fundamental questions: how categorical concepts are represented, and how hierarchical relations between concepts are encoded. They extend the linear representation hypothesis to show that simple categorical concepts are represented as simplices, and hierarchically related concepts are orthogonal in a specific sense. Complex concepts, which reflect hierarchical structures, are represented as polytopes constructed from direct sums of simplices. The theoretical findings are validated using the Gemma LLM, where representations for 957 hierarchically related concepts are estimated using WordNet data. The results provide a foundation for understanding how high-level semantic concepts are encoded in LLMs, with implications for model interpretability and control.