CrystalFormer is a transformer-based autoregressive model designed for generating crystalline materials with space group symmetry. By leveraging the discrete and sequential nature of Wyckoff positions, CrystalFormer learns to predict the species and locations of symmetry-inequivalent atoms in the unit cell. It demonstrates advantages in symmetric structure initialization and element substitution compared to conventional methods. CrystalFormer also enables property-guided materials design through a plug-and-play approach. The model compresses the material dataset, enabling systematic exploration of crystalline materials by incorporating solid-state chemistry knowledge and heuristics. CrystalFormer's architecture is simple, general, and flexible, making it a promising foundation for crystalline materials modeling. The model is trained on the MP-20 dataset and can generate crystal samples with high accuracy. It is capable of generating crystals with given space group symmetry and can be used for property-guided materials design. CrystalFormer's ability to generate diverse and stable crystal structures reduces computational costs for downstream optimizations. The model is also effective in generating double perovskite structures and has been shown to outperform standard element substitution methods. CrystalFormer's plug-and-play approach allows for the integration of multiple conditions in materials design. The model's performance is comparable to other crystal graph convolutional neural networks. The model's ability to generate materials with specific properties is demonstrated through controlled generation of materials with target band gaps and formation energies. The model's likelihood-based exploration of the crystal space is more efficient compared to traditional sampling approaches based on physical energy functions. The model's ability to generate materials with given crystal lattice is also feasible. The model's performance is validated through various experiments and comparisons with existing methods.CrystalFormer is a transformer-based autoregressive model designed for generating crystalline materials with space group symmetry. By leveraging the discrete and sequential nature of Wyckoff positions, CrystalFormer learns to predict the species and locations of symmetry-inequivalent atoms in the unit cell. It demonstrates advantages in symmetric structure initialization and element substitution compared to conventional methods. CrystalFormer also enables property-guided materials design through a plug-and-play approach. The model compresses the material dataset, enabling systematic exploration of crystalline materials by incorporating solid-state chemistry knowledge and heuristics. CrystalFormer's architecture is simple, general, and flexible, making it a promising foundation for crystalline materials modeling. The model is trained on the MP-20 dataset and can generate crystal samples with high accuracy. It is capable of generating crystals with given space group symmetry and can be used for property-guided materials design. CrystalFormer's ability to generate diverse and stable crystal structures reduces computational costs for downstream optimizations. The model is also effective in generating double perovskite structures and has been shown to outperform standard element substitution methods. CrystalFormer's plug-and-play approach allows for the integration of multiple conditions in materials design. The model's performance is comparable to other crystal graph convolutional neural networks. The model's ability to generate materials with specific properties is demonstrated through controlled generation of materials with target band gaps and formation energies. The model's likelihood-based exploration of the crystal space is more efficient compared to traditional sampling approaches based on physical energy functions. The model's ability to generate materials with given crystal lattice is also feasible. The model's performance is validated through various experiments and comparisons with existing methods.