5 Mar 2024 | Zeyuan Ma, Hongshu Guo, Jiacheng Chen, Guojun Peng, Zhiguang Cao, Yining Ma, Yue-Jiao Gong
LLaMoCo is an instruction-tuning framework designed to adapt large language models (LLMs) for solving optimization problems in a code-to-code manner. The framework introduces a comprehensive instruction set containing well-described problem prompts and effective optimization codes. It employs a two-phase learning strategy, including a contrastive learning-based warm-up phase before instruction tuning, to enhance model convergence during fine-tuning. Experiments show that a CodeGen (350M) model fine-tuned by LLaMoCo outperforms GPT-4 Turbo and other competitors on both synthetic and realistic problem sets. LLaMoCo provides a user-friendly approach by allowing users to describe optimization problems through a stipulated prompt protocol, enabling the model to generate proper optimization code. The framework addresses limitations of existing methods, such as low efficiency, sensitivity to prompt design, and lack of domain-specific knowledge. LLaMoCo's two-phase training strategy improves latent space representations and enhances instruction tuning performance. The framework demonstrates superior optimization performance, with the fine-tuned model showing strong zero-shot generalization ability to realistic optimization tasks. The results indicate that even a relatively small LLM can achieve performance comparable to larger models like GPT-4 when fine-tuned with domain-specific knowledge. LLaMoCo's approach offers a more efficient and robust solution for optimization code generation compared to existing methods.LLaMoCo is an instruction-tuning framework designed to adapt large language models (LLMs) for solving optimization problems in a code-to-code manner. The framework introduces a comprehensive instruction set containing well-described problem prompts and effective optimization codes. It employs a two-phase learning strategy, including a contrastive learning-based warm-up phase before instruction tuning, to enhance model convergence during fine-tuning. Experiments show that a CodeGen (350M) model fine-tuned by LLaMoCo outperforms GPT-4 Turbo and other competitors on both synthetic and realistic problem sets. LLaMoCo provides a user-friendly approach by allowing users to describe optimization problems through a stipulated prompt protocol, enabling the model to generate proper optimization code. The framework addresses limitations of existing methods, such as low efficiency, sensitivity to prompt design, and lack of domain-specific knowledge. LLaMoCo's two-phase training strategy improves latent space representations and enhances instruction tuning performance. The framework demonstrates superior optimization performance, with the fine-tuned model showing strong zero-shot generalization ability to realistic optimization tasks. The results indicate that even a relatively small LLM can achieve performance comparable to larger models like GPT-4 when fine-tuned with domain-specific knowledge. LLaMoCo's approach offers a more efficient and robust solution for optimization code generation compared to existing methods.