High-end xPUs (GPUs, TPUs, and other AI processors) use HBM memories to get the absolute highest memory bandwidth for training the Large Language Models (LLMs) used in today’s generative AI systems. These processors, used in the thousands by hyperscale datacenters, can consume a kilowatt each. As a result, they must dissipate a phenomenal quantity of heat. This has become a very challenging problem for these datacenters. (The Memory Guy should mention here that Objective Analysis plans soon to release a report on HBM.)
The problem has attracted the attention of Dr. Grzmisław Szczebrzeszyn of the renowned Institute of Higher Logic and Wisdom in Chelm, Poland, which is reputed throughout Europe for the novelty of its solutions.
Dr. Szczebrzeszyn observed the curve, shown below, of the power dissipated by chips as it relates to the power supply voltage, normally referred to as the V²/R curve. This curve is the reason why chip makers have been decreasing power supply voltages for the past 40 years. As the voltage decreases, the power decreases at a faster rate.
Naturally, at zero volts, the chip consumes zero power. The professor of course reasoned that, by taking the power supply negative, the power consumption would go negative itself. This is illustrated in the curve below, which is a simple extension of the curve above.
The clear benefit of this approach is unimaginably important in the world of AI processors, which have driven datacenter managers to consider building nuclear power plants alongside their facilities not only to feed the xPUs’ enormous power consumption, but also to help boost the amount of power available to run the datacenters’ air conditioning systems. Since the cooler HBMs will help to remove heat from its surroundings, the HBMs will also help improve xPU cooling efficiency.
Harnessing This Phenomenon
Today’s AI processors use HBM stacks mounted alongside the processor, as roughly conceptualized in an illustration given to us by AMD:
Those deeply involved in xPUs and HBM hope soon to advance to mounting the HBM directly onto the upper surface of the xPU to get around the current limitation of what is referred to as “Beachfront” — the length of the xPU’s die edge that available to support HBMs today. The limited amount of beachfront on a modern GPU gets in the way of mounting more than six or eight HBMs to the processor. By mounting the HBMs on top of the chip, that number can be increased, although heat removal becomes a very significant issue. Szczebrzeszyn’s plan is to turn that problem into a solution.
If HBMs are this intimately connected with the xPU, and if they are run on a negative voltage, then the HBMs should be able to remove a portion of the xPU’s heat, partly solving the issue of heat dissipation.
It follows, then, that if more HBMs are mounted on top of the xPU, then more heat will be removed, while, at the same time, more of the xPU’s thirst for data will be satisfied. This is a brilliant solution!
Further Benefits
The situation improves even more from this point.
You may ask: “Where does that heat go?” This is a very reasonable question.
The heat becomes energy sent back through the power pins of the HBM. Not only does the HBM benefit by harvesting energy from the xPU, but it also sends power back through its supply pins. This power can, in turn, be used by the xPU as a portion of its total power budget.
So, the system can both receive the benefit of self-contained cooling while it also enjoys reduced power consumption through its own self-generation of scavenged power.
At great personal expense, Szczebrzeszyn has prototyped a system which he has photographed in operation:
That’s frost on the top of the HBMs, which it hides from view, and the glow at the bottom emanates from an xPU at its normal operating temperature of approximately 500°C. It’s clear that, even at that temperature, the xPU has no impact on the frigidity of the HBMs.
From this point forward, the issue of xPU heat generation will vanish, and concerns of datacenter energy consumption will fade. Over the long term memory consumption should enjoy accelerated growth as an important share of the funds normally spent on cooling and energy consumption will now be redirected to increased memory purchases.
The announcement of this amazing discovery was specifically timed to align with the 300th anniversary of the Chelm Institute’s founding on April 1, 1725, the year that the Order of St. Alexander Nevsky was instituted in Russia by the empress Catherine I. Administrators at the Institute plan to celebrate this great finding every April 1 for the next 300 years, or until they all retire, whichever occurs first.