The paper introduces a novel framework, the Protein Structure Transformer (PST), which enhances protein language models by integrating structural information. PST refines the self-attention mechanisms of pretrained language transformers by incorporating structural data using structure extractor modules. The model is pre-trained on a small protein structure database using masked language modeling. Empirical evaluations show that PST outperforms state-of-the-art protein language models, such as ESM-2, in various function and structure prediction tasks, demonstrating superior parameter efficiency and accuracy. The findings highlight the potential of integrating structural information into protein language models, offering new insights into the relationship between protein sequence, structure, and function. The code and pre-trained models are available at <https://github.com/BorgwardtLab/PST>.The paper introduces a novel framework, the Protein Structure Transformer (PST), which enhances protein language models by integrating structural information. PST refines the self-attention mechanisms of pretrained language transformers by incorporating structural data using structure extractor modules. The model is pre-trained on a small protein structure database using masked language modeling. Empirical evaluations show that PST outperforms state-of-the-art protein language models, such as ESM-2, in various function and structure prediction tasks, demonstrating superior parameter efficiency and accuracy. The findings highlight the potential of integrating structural information into protein language models, offering new insights into the relationship between protein sequence, structure, and function. The code and pre-trained models are available at <https://github.com/BorgwardtLab/PST>.