This paper introduces Bilevel Positional Encoding (BiPE), a novel positional encoding method designed to improve length extrapolation in language models. BiPE combines two distinct encodings for each position: an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the location within a segment, helping the model capture semantic information, while the inter-segment encoding specifies the segment index and models relationships between segments, enhancing extrapolation capabilities. Theoretical analysis shows that this disentanglement of positional information makes learning more effective. Empirical results demonstrate that BiPE outperforms existing methods in various tasks across different text modalities, showing superior length extrapolation capabilities. The paper also includes a theoretical justification of BiPE's parameter efficiency and extensive experiments to validate its effectiveness.This paper introduces Bilevel Positional Encoding (BiPE), a novel positional encoding method designed to improve length extrapolation in language models. BiPE combines two distinct encodings for each position: an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the location within a segment, helping the model capture semantic information, while the inter-segment encoding specifies the segment index and models relationships between segments, enhancing extrapolation capabilities. Theoretical analysis shows that this disentanglement of positional information makes learning more effective. Empirical results demonstrate that BiPE outperforms existing methods in various tasks across different text modalities, showing superior length extrapolation capabilities. The paper also includes a theoretical justification of BiPE's parameter efficiency and extensive experiments to validate its effectiveness.