From a user perspective, some gaming enthusiasts have built their own PCs equipped with high-performance GPUs like the NVIDIA GeForce RTX 4090. Interestingly, this GPU is also capable of handling small-scale deep learning tasks. The RTX 4090 requires a 450W power supply, with a recommended total power supply of 850W (in most cases it doesn't need this and won't run at full load). If your task runs continuously for a week, that translates to 0.85 kW × 24 hours × 7 days = 142.8 kWh per week. In California, PG&E charges residential customers up to 50 cents per kWh, meaning you'd spend about $70 per week on electricity. Additionally, you will need a CPU and other components that work alongside your GPU, which will further increase electricity consumption. This means that the total cost of electricity can be even higher.
Now, your ai business will accelerate. According to the manufacturer, an H100 Tensor Core GPU has a maximum thermal design power (TDP) of around 700 watts, depending on the specific version. This is the power needed to cool the GPU under full workload. A reliable power supply unit for this high-performance deep learning tool is typically around 1600 W. If you use the NVIDIA DGX platform for your deep learning tasks, a single DGX H100 system, equipped with 8 H100 GPUs, consumes approximately 10 .2kW. For even higher performance, an NVIDIA DGX SuperPOD can include between 24 and 128 NVIDIA DGX nodes. With 64 nodes, the system could conservatively consume about 652.8 kW. While your startup could hope to purchase this equipment worth millions of dollars, the costs of both the cluster and the necessary facilities would be substantial. In most cases, it makes more sense to rent GPU clusters from cloud computing providers. Focusing on energy costs, commercial and industrial users generally benefit from lower electricity rates. If your average cost is around 20 cents per kWh, operating 64 DGX nodes at 652.8 kW 24/7 would result in 109.7 MWh per week. This could cost you approximately $21,934 per week.
According to rough estimates, a typical family in California would spend about 150 kWh per week on electricity. Interestingly, this is about the same cost you would incur if you ran a model training task at home using a high-performance GPU like the RTX 4090.
From this table we can see that operating a SuperPOD with 64 nodes could consume as much energy in a week as a small community.
Training ai models
Now, let's dive into some numbers related to modern ai models. OpenAI has never revealed the exact number of GPUs used to train ChatGPT, but a rough estimate suggests it could involve thousands of GPUs running continuously for several weeks or months, depending on the release date of each ChatGPT model. The energy consumption for such a task would easily be on the scale of megawatts, resulting in costs on the scale of thousands of MWh.
Recently, Meta released LLaMA 3.1described as its “most capable model to date.” According to Meta, this is their largest model yet, trained on over 16,000 H100 GPUs – the first LLaMA model trained at this scale.
Let's crunch the numbers: LLaMA 2 was released in July 2023, so it's reasonable to assume that LLaMA 3 was at least a year in the making. While it is unlikely that all GPUs will run 24/7, we can estimate the power consumption at a 50% utilization rate:
1.6 kW × 16,000 GPU × 24 hours/day × 365 days/year × 50% ≈ 112,128 MWh
At an estimated cost of $0.20 per kWh, this translates to approximately $22.4 million in energy costs. This figure represents GPUs only, excluding additional power consumption related to data storage, networking, and other infrastructure.
Training modern large language models (LLMs) requires megawatt-scale energy consumption and represents a multimillion-dollar investment. This is why modern ai development often excludes smaller players.
Operate ai models
Running ai models also incurs significant energy costs, as each query and response requires computational power. Although the energy cost per interaction is small compared to model training, the cumulative impact can be substantial, especially if your ai business achieves large-scale success with billions of users interacting with your advanced LLM daily. Many interesting articles discuss this topic, including ai-chatbots-energy-usage-of-2023s-most-popular-chatbots-so-far/#:~:text=The%20training%20time%20of%20GPT,%2Dhours%2C%20or%207%2C200%20MWh.” rel=”noopener ugc nofollow” target=”_blank”>Energy cost comparisons between companies operating ChatBots. The bottom line is that since each query could cost between 0.002 and 0.004 kWh, popular companies would currently spend hundreds to thousands of MWh per year. And this number continues to increase.
Imagine for a moment that a billion people use a ChatBot frequently, averaging around 100 queries per day. The energy cost for this use can be estimated as follows:
0.002 kWh × 100 consultations/day × 1e9 people × 365 days/year ≈ 7.3e7 MWh/year
This would require a power supply of 8,000 MW and could result in an energy cost of approximately $14.6 billion per year, assuming an electricity rate of $0.20 per kWh.
The largest power plant in the US is the Grand Coulee Dam in the state of Washington, with a capacity of 6,809 MW. The largest solar park in the US is solar star in California, which has a capacity of 579 MW. In this context, no power plant is capable of supplying all the electricity needed for a large-scale ai service. This becomes evident when considering the annual electricity generation statistics provided by EIA (Energy Information Administration),
The 73 billion kWh calculated above would represent approximately 1.8% of the total electricity generated annually in the United States. However, it is reasonable to believe that this figure could be much higher. According to some media reports, when considering all energy consumption related to ai and data processing, the impact could be around 4% of total US electricity generation.
However, this is the current energy usage.
Today, chatbots primarily generate text-based responses, but are increasingly capable of producing two-dimensional images, “three-dimensional” videos, and other forms of media. The next generation of ai will go far beyond simple chatbots, which can provide high-resolution images for spherical displays (for example, for Las Vegas Sphere), 3D modeling and interactive robots capable of performing complex tasks and executing deep logistics. As a result, power demands for both model training and deployment are expected to increase dramatically, far exceeding current levels. It remains an open question whether our existing energy infrastructure can support such advances.
On the sustainability front, carbon emissions from industries with high energy demands are significant. One approach to mitigating this impact involves using renewable energy sources to power energy-intensive facilities such as data centers and computing centers. A notable example is the collaboration between Fervo Energía and Googlewhere geothermal energy is used to power a data center. However, the scale of these initiatives remains relatively small compared to the overall energy needs anticipated in the coming ai era. There is still much work to do to address sustainability challenges in this context.
Correct the numbers if you consider them unreasonable.
