This paper explores cost-efficient methods to adapt pre-trained Large Language Models (LLMs) to new lower-resource languages, focusing on Estonian. Using the Llama 2 model, the authors investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. The results show that even a small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances performance on Estonian. The authors also demonstrate cross-lingual knowledge transfer from high-quality English instructions to Estonian, improving commonsense reasoning and multi-turn conversation capabilities. The best model, named LHAMMAS, is the first open-source instruction-following LLM for Estonian. Additionally, the authors publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in developing open-source LLMs for Estonian. The paper includes a detailed evaluation of the models on various tasks, such as question answering, machine translation, and grammatical error correction, and discusses the limitations and ethical considerations of the work.This paper explores cost-efficient methods to adapt pre-trained Large Language Models (LLMs) to new lower-resource languages, focusing on Estonian. Using the Llama 2 model, the authors investigate the impact of combining cross-lingual instruction-tuning with additional monolingual pretraining. The results show that even a small amount of additional monolingual pretraining followed by cross-lingual instruction-tuning significantly enhances performance on Estonian. The authors also demonstrate cross-lingual knowledge transfer from high-quality English instructions to Estonian, improving commonsense reasoning and multi-turn conversation capabilities. The best model, named LHAMMAS, is the first open-source instruction-following LLM for Estonian. Additionally, the authors publish Alpaca-est, the first general task instruction dataset for Estonia. These contributions mark the initial progress in developing open-source LLMs for Estonian. The paper includes a detailed evaluation of the models on various tasks, such as question answering, machine translation, and grammatical error correction, and discusses the limitations and ethical considerations of the work.