Nvidia’s GPUs: Why the hottest items in the AI boom are in short supply

The “H100s” are brands of high-end chips called graphics processing units, or GPUs, which continue to face a massive supply crunch amid skyrocketing demand, even as shortages across most other chip categories are beginning to ease out.

There are two reasons for the shortage.

One, the generative AI boom has led to the demand for these specialised chips to skyrocket, given that GPUs have the computing power and operational efficiency to run the calculations that allow AI companies working on LLMs (or large language models), such as ChatGPT or Bard, to chomp down on massive volumes of data.

Two, there is only one major company that produces H100 type chips — US-based Nvidia Corporation, which has seen valuations top $1 trillion since the LLM boom and is now swamped with orders that it is struggling to deliver.

As a result, consortiums such as the San Francisco Compute Group are finding ways to get around the shortage by promising smaller startups and researchers limited access to these chips by scrounging the market for GPUs and buying whatever is available by raising funds using the procured computer chips as collateral.

GPU rush, Nvidia stranglehold

The pioneering graphics chipmaker is already one of the world’s most valuable companies, riding on its dominance in the gaming and AI sectors, and the potential that generative AI holds in reshaping the technology sector.

Nvidia’s market cap breached $1 trillion on May 30, making it the first US chipmaker to enter the elite trillion-dollar club ahead of storied competitors such as Intel and AMD.

Buoyant demand for chips in data centres is the primary reason why Nvidia has been surging, with the latest trigger being the generative AI rush.

Traditionally, the CPU, or central processing unit, has been the most important component in a computer or server, and Intel and AMD dominate the market. GPUs are relatively new additions to the computer hardware market, and were initially sold as cards that plugged into a personal computer’s motherboard to add computing power to an AMD or Intel CPU.

Nvidia’s main pitch over the years has been that graphics chips can handle the computation workload surge of the kind that is needed in high-end graphics for gaming or animation applications far better than standard processors. AI applications too require tremendous computing power and have been progressively getting GPU-heavy in their backend hardware.

Most advanced systems used for training generative AI tools now deploy as many as half a dozen GPUs to every one CPU used, completely changing the equation in which GPUs were seen as add-ons to CPUs. Nvidia dominates the global market for GPUs and is likely to maintain this lead well into the foreseeable future.

AI wave, data centre demand

If Taiwan-based foundry specialist TSMC is the most important backend player in the semiconductor chips business, Nvidia (with Intel, AMD, Samsung and Qualcomm) is at the front end. Since 1999, when Nvidia first popularised the term GPU with its GeForce 256 processor, the company’s chips have been coveted as shaping what is possible in graphics. Nvidia GPU chips such as the new ‘RTX’ range are now at the forefront of the generative AI boom based on LLMs.

Nvidia’s data centre business has recorded a growth of almost 15% during the first quarter of this calendar year versus flat growth for AMD’s data centre unit and a sharp decline of nearly 40% in Intel’s data centre business unit. Alongside the application of GPUs, the company’s chips are comparatively more expensive that most CPUs on a per unit basis, resulting in far better margins.

Analysts say Nvidia is ahead in the race for AI chips because of its proprietary software that makes it easier to leverage all of the GPU hardware features for AI applications. Nvidia also has the systems that back the processors up, and the software that runs all of it, making it a full stack solutions company.

In addition to GPU manufacturing, Nvidia offers an application programme interface (API) — a set of defined instructions that enable different applications to communicate with each other — called CUDA, which allows the creation of parallel programs using GPUs and are deployed in supercomputing sites around the world.

It also has a leg in the mobile computing market with its Tegra mobile processors for smartphones and tablets, as well as products in the vehicle navigation and entertainment systems.

High entry barriers

Nvidia’s resilience is a case study in a segment that has very high entry barriers and offers a phenomenal premium for specialisation. The global semiconductor chip industry is dominated by some countries and a handful of companies. Taiwan and South Korea make up about 80% of the global foundry base for chips. Only one firm, the Netherlands-based ASML, produces EUV (extreme ultraviolet lithography) devices, without which it is not possible to make an advanced chip. Cambridge, UK-based chip designer Arm is the world’s biggest supplier of chip design elements used in products from smartphones to games consoles (which Nvidia was keen to acquire).

It’s a nearly closed manufacturing ecosystem with very high entry barriers, as China’s SMIC, a national semiconductor champion that is now reportedly struggling to procure advanced chip-making equipment after a US-led blockade, is finding out. In this market, Nvidia, which comprehensively dominates the chips used for high-end graphics-based applications, has come to dominate multiple end-use sectors including gaming, crypto mining, and now AI.