This paper presents a novel approach using evolutionary algorithms to automatically discover effective combinations of diverse open-source models for creating new foundation models. The method operates in both parameter space and data flow space, enabling optimization beyond just the weights of individual models. It facilitates cross-domain merging, generating models like a Japanese LLM with math reasoning capabilities. The Japanese Math LLM achieved state-of-the-art performance on various benchmarks, even surpassing models with significantly more parameters. A culturally-aware Japanese VLM also demonstrated effectiveness in describing Japanese culture-specific content, outperforming previous Japanese VLMs. The work contributes new state-of-the-art models to the open-source community and introduces a new paradigm for automated model composition, paving the way for exploring alternative, efficient approaches to foundation model development. The paper discusses the application of evolutionary algorithms to model merging, which involves combining multiple models into a single architecture. This approach is compared to traditional transfer learning, where a pre-trained model is further fine-tuned for a new task. Model merging aims to create a versatile and comprehensive model by combining the knowledge from multiple pre-trained models. The paper presents a methodology that leverages evolutionary algorithms to facilitate the merging of foundation models. The approach is distinguished by its ability to navigate both parameter space (weights) and data flow space (inference path), proposing a framework that integrates these two dimensions. The paper presents several key contributions to the field of foundation model development: automated model composition, cross-domain merging, state-of-the-art performance, high efficiency and surprising generalizability, and culturally-aware VLM. The work demonstrates that evolutionary algorithms can discover more effective model merging solutions, providing a path for automating the creation of more capable models. The paper also discusses the connection to evolutionary neural architecture search, where evolutionary algorithms can explore a vast space of possibilities, discovering novel and counter-intuitive combinations that traditional methods and human intuition might miss. The paper presents a method for merging models in both parameter space and data flow space. In parameter space, the weights of multiple foundational models are integrated into a unified entity with the same neural network architecture. In data flow space, the inference path that tokens follow as they traverse through the neural network is optimized. The paper shows that merging in the data flow space can lead to better performance by optimizing the inference path rather than just the weights of the individual models. The paper also discusses the application of evolutionary algorithms to model merging, which involves combining multiple models into a single architecture. The approach is compared to traditional transfer learning, where a pre-trained model is further fine-tuned for a new task. Model merging aims to create a versatile and comprehensive model by combining the knowledge from multiple pre-trained models. The paper presents a methodology that leverages evolutionary algorithms to facilitate the merging of foundation models. The approach is distinguished by its ability to navigate both parameter space (weights) and data flow space (inference path), proposing a framework that integrates these two dimensions.This paper presents a novel approach using evolutionary algorithms to automatically discover effective combinations of diverse open-source models for creating new foundation models. The method operates in both parameter space and data flow space, enabling optimization beyond just the weights of individual models. It facilitates cross-domain merging, generating models like a Japanese LLM with math reasoning capabilities. The Japanese Math LLM achieved state-of-the-art performance on various benchmarks, even surpassing models with significantly more parameters. A culturally-aware Japanese VLM also demonstrated effectiveness in describing Japanese culture-specific content, outperforming previous Japanese VLMs. The work contributes new state-of-the-art models to the open-source community and introduces a new paradigm for automated model composition, paving the way for exploring alternative, efficient approaches to foundation model development. The paper discusses the application of evolutionary algorithms to model merging, which involves combining multiple models into a single architecture. This approach is compared to traditional transfer learning, where a pre-trained model is further fine-tuned for a new task. Model merging aims to create a versatile and comprehensive model by combining the knowledge from multiple pre-trained models. The paper presents a methodology that leverages evolutionary algorithms to facilitate the merging of foundation models. The approach is distinguished by its ability to navigate both parameter space (weights) and data flow space (inference path), proposing a framework that integrates these two dimensions. The paper presents several key contributions to the field of foundation model development: automated model composition, cross-domain merging, state-of-the-art performance, high efficiency and surprising generalizability, and culturally-aware VLM. The work demonstrates that evolutionary algorithms can discover more effective model merging solutions, providing a path for automating the creation of more capable models. The paper also discusses the connection to evolutionary neural architecture search, where evolutionary algorithms can explore a vast space of possibilities, discovering novel and counter-intuitive combinations that traditional methods and human intuition might miss. The paper presents a method for merging models in both parameter space and data flow space. In parameter space, the weights of multiple foundational models are integrated into a unified entity with the same neural network architecture. In data flow space, the inference path that tokens follow as they traverse through the neural network is optimized. The paper shows that merging in the data flow space can lead to better performance by optimizing the inference path rather than just the weights of the individual models. The paper also discusses the application of evolutionary algorithms to model merging, which involves combining multiple models into a single architecture. The approach is compared to traditional transfer learning, where a pre-trained model is further fine-tuned for a new task. Model merging aims to create a versatile and comprehensive model by combining the knowledge from multiple pre-trained models. The paper presents a methodology that leverages evolutionary algorithms to facilitate the merging of foundation models. The approach is distinguished by its ability to navigate both parameter space (weights) and data flow space (inference path), proposing a framework that integrates these two dimensions.