This paper introduces a new approach called Contrastive Sequence-Structure Pre-training (CSSP) to enhance antibody language models (PLMs) by incorporating structural information. Antibodies are proteins in the immune system with highly variable sequences, making them challenging for traditional protein language models (PLMs) to predict. The CSSP method uses a contrastive learning approach inspired by CLIP, aligning sequence and structure representations in a shared latent space. This approach improves the model's ability to capture structural similarity and enhances accuracy in downstream tasks like binding prediction. The study uses a combination of experimental and predicted antibody structures to train the model. The CSSP approach involves training a sequence encoder while using a pre-trained structure encoder. This method leads to better sequence representations that align with structural similarity, improving performance in binding prediction tasks. The model, AntiBERTa2-CSSP, is made available for non-commercial use. The results show that CSSP significantly improves the performance of PLMs in predicting antibody-antigen binding. The model achieves higher accuracy and data efficiency, especially when using a small dataset of experimental structures. The study also highlights the importance of structural information in understanding antibody function and demonstrates that CSSP can be applied to various PLMs, offering a flexible solution for antibody engineering. The paper also discusses the limitations of using predicted structures, showing that they do not provide the same level of information as experimental structures. This emphasizes the need for high-quality structural data in antibody discovery. Overall, the CSSP approach provides a valuable tool for improving the understanding and prediction of antibody function, with potential applications in resource-limited environments.This paper introduces a new approach called Contrastive Sequence-Structure Pre-training (CSSP) to enhance antibody language models (PLMs) by incorporating structural information. Antibodies are proteins in the immune system with highly variable sequences, making them challenging for traditional protein language models (PLMs) to predict. The CSSP method uses a contrastive learning approach inspired by CLIP, aligning sequence and structure representations in a shared latent space. This approach improves the model's ability to capture structural similarity and enhances accuracy in downstream tasks like binding prediction. The study uses a combination of experimental and predicted antibody structures to train the model. The CSSP approach involves training a sequence encoder while using a pre-trained structure encoder. This method leads to better sequence representations that align with structural similarity, improving performance in binding prediction tasks. The model, AntiBERTa2-CSSP, is made available for non-commercial use. The results show that CSSP significantly improves the performance of PLMs in predicting antibody-antigen binding. The model achieves higher accuracy and data efficiency, especially when using a small dataset of experimental structures. The study also highlights the importance of structural information in understanding antibody function and demonstrates that CSSP can be applied to various PLMs, offering a flexible solution for antibody engineering. The paper also discusses the limitations of using predicted structures, showing that they do not provide the same level of information as experimental structures. This emphasizes the need for high-quality structural data in antibody discovery. Overall, the CSSP approach provides a valuable tool for improving the understanding and prediction of antibody function, with potential applications in resource-limited environments.