The Real Cost of Running AI at Scale
AI’s infrastructure bill looks simple because it is divided into tidy per-million-token prices. It is anything but simple. Every generated token also carries an invisible tail of electricity, cooling water, grid emissions, hardware depreciation, and engineering complexity—and that tail is getting longer.
Start with power. The International Energy Agency estimates that data centers consumed roughly 415 terawatt-hours of electricity in 2024, about 1.5% of global demand, and projects consumption to approach 945 terawatt-hours by 2030. That growth is not evenly distributed. EPRI estimates that U.S. data centers could consume between 6.7% and 12% of American electricity by 2028. AI accelerators are part of the reason: NVIDIA says a GB200 NVL72 rack can draw around 120 kilowatts. A 100-megawatt AI campus running flat out would use 876,000 megawatt-hours annually—enough to supply roughly 87,600 U.S. households at average annual consumption. Training one frontier model can concentrate that demand in one place. Google reported using 29.3 million TPU v3 GPU-hours to train Gemini Ultra; at a nominal $2 per accelerator-hour, that compute alone represents about $58.6 million.
Water is where neat generalizations collapse. There is no honest universal “liters per token” figure because consumption depends on climate, humidity, hardware, utilization, and cooling design. Air-cooled facilities consume little operational water, but evaporation can become part of the cooling process; closed-loop systems reduce water use while requiring more energy. That creates a local trade-off: the cheapest server room in a hot, dry region may quietly appropriate the region’s water, while a water-efficient design in a comparatively water-rich grid may produce more carbon. Utilities and large operators increasingly report both total water consumption and water usage effectiveness, which is the right direction, but those figures are difficult to compare without the site’s power mix and cooling configuration. The real problem is that token invoices disclose none of it.
Carbon follows the same uncomfortable logic. An inference request in a data center outside Seattle and the identical request in Oklahoma can have different emissions because the grids have different marginal generators and carbon intensity. Offsets can compensate for reported emissions, but they do not reduce heat, water demand, or strain on the local grid. AI’s efficiency gains therefore do not automatically make the overall footprint smaller. If better models generate more requests, or reasoning systems produce more intermediate tokens for each answer, efficiency can be swallowed by growth. The AWS Builders account of three agent orchestrators shows the trap in miniature: token cost rose 74.5% and 192.8% in two mixed runs, and a nested failure was still reported as 100% successful. More machine activity appeared to mean more efficiency until the bill—and the metric—caught up.
Money is where the physical costs become unavoidable. Operators must buy accelerators, networking, cooling, power capacity, and buildings years before demand is proven, then guess which models will remain valuable. That makes depreciation and utilization inseparable from price. A cheap API can still be supported by expensive capacity, while a spectacularly efficient system can fail commercially if it serves too few tokens per dollar of hardware. Open inference engines may widen competition, but software support remains part of the purchase. On the TrnRT discussion, bookernath put it neatly: “Tokens per minute is a strange metric, I usually see tok/s. A per-second figure would be more accessible in comparing this to alternatives.” The same applies beyond price lists: throughput means little without power draw, latency distribution, utilization, and total cost per completed task.
For teams building products, the practical response is unglamorous accounting. Track energy and carbon per completed request, not per generated token. Include failed agent loops, retries, tool calls, and tokens consumed by orchestration. Compare deployment regions by marginal electricity cost and carbon intensity. Reserve capacity for known traffic instead of pretending every possible model will be hot. And evaluate smaller models, cached responses, batch processing, and routing before reaching for the largest system. The background remover running entirely in the browser illustrates the alternative: moving inference onto the user’s device can avoid a server-side GPU and its full utility burden. It also reveals why “free” is often merely a change in who pays.
AI at scale is not powered by magic; it is powered by industrial infrastructure whose costs have been socialized, deferred, or hidden behind flattering token prices. The winners will not be the teams that spend the most on inference, but the teams that can measure the whole bill and make every watt, joule, and token answer for itself.
Comments
Leave a message below. Your comment saves to your browser.