BLADE is a novel framework designed to enhance Black-box Large Language Models (LLMs) with small Domain-specific Models. The framework consists of a general LLM and a small domain-specific LM, where the small LM preserves domain-specific knowledge and offers specialized insights, while the general LLM contributes robust language comprehension and reasoning capabilities. The method involves three steps: Domain-specific Pretraining (DP), Knowledge Instruction Tuning (KIT), and Bayesian Prompted Optimization (BPO). DP imparts domain-specific knowledge to the small LM, KIT enhances the small LM's ability to follow instructions, and BPO aligns the output of the small LM with the general LLM using derivative-free optimization. Extensive experiments on legal and medical benchmarks show that BLADE significantly outperforms existing approaches, demonstrating its effectiveness and cost-efficiency in adapting general LLMs for vertical domains.BLADE is a novel framework designed to enhance Black-box Large Language Models (LLMs) with small Domain-specific Models. The framework consists of a general LLM and a small domain-specific LM, where the small LM preserves domain-specific knowledge and offers specialized insights, while the general LLM contributes robust language comprehension and reasoning capabilities. The method involves three steps: Domain-specific Pretraining (DP), Knowledge Instruction Tuning (KIT), and Bayesian Prompted Optimization (BPO). DP imparts domain-specific knowledge to the small LM, KIT enhances the small LM's ability to follow instructions, and BPO aligns the output of the small LM with the general LLM using derivative-free optimization. Extensive experiments on legal and medical benchmarks show that BLADE significantly outperforms existing approaches, demonstrating its effectiveness and cost-efficiency in adapting general LLMs for vertical domains.