The rising costs of training frontier AI models

The rising costs of training frontier AI models

31 May 2024 | Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej, David Owen
The costs of training frontier AI models have risen dramatically since 2016, with amortized costs increasing by 2.4 times per year (95% CI: 2.0 to 3.1). This study estimates training costs using three approaches: hardware and energy amortization, cloud rental prices, and total model development costs including R&D staff. The most expensive models, such as GPT-4 and Gemini Ultra, cost tens of millions of dollars to train, with GPT-4 at 40M and Gemini Ultra at 30M. Training costs are driven by AI accelerator chips, server components, cluster-level interconnect, and energy consumption. If the trend continues, the largest training runs could cost over a billion dollars by 2027, limiting access to only the most well-funded organizations. Hardware acquisition costs are one to two orders of magnitude higher than amortized costs, with AI accelerator chips accounting for 44% of amortized hardware and energy costs. R&D staff costs make up 29–49% of total amortized model development costs, with equity included. Excluding equity, R&D staff costs drop to 19–33%. The study highlights the economic challenges of AI development, including the high costs of hardware, energy, and R&D labor. The findings suggest that AI development will become increasingly expensive, raising concerns about access and responsible development. The study provides a detailed analysis of training costs, emphasizing the need for further research and data collection to better understand the economic implications of AI development.The costs of training frontier AI models have risen dramatically since 2016, with amortized costs increasing by 2.4 times per year (95% CI: 2.0 to 3.1). This study estimates training costs using three approaches: hardware and energy amortization, cloud rental prices, and total model development costs including R&D staff. The most expensive models, such as GPT-4 and Gemini Ultra, cost tens of millions of dollars to train, with GPT-4 at 40M and Gemini Ultra at 30M. Training costs are driven by AI accelerator chips, server components, cluster-level interconnect, and energy consumption. If the trend continues, the largest training runs could cost over a billion dollars by 2027, limiting access to only the most well-funded organizations. Hardware acquisition costs are one to two orders of magnitude higher than amortized costs, with AI accelerator chips accounting for 44% of amortized hardware and energy costs. R&D staff costs make up 29–49% of total amortized model development costs, with equity included. Excluding equity, R&D staff costs drop to 19–33%. The study highlights the economic challenges of AI development, including the high costs of hardware, energy, and R&D labor. The findings suggest that AI development will become increasingly expensive, raising concerns about access and responsible development. The study provides a detailed analysis of training costs, emphasizing the need for further research and data collection to better understand the economic implications of AI development.
Reach us at info@study.space
[slides and audio] The_rising_costs_of_training_frontier_AI_models