The ever-increasing scale of artificial intelligence programs such as OpenAI’s ChatGPT is creating a crisis where the available memory chips can’t keep up with today’s fastest processors from Nvidia and others. Investors should look for opportunities among companies reinventing memory chips.

Note: This is a reprint of an article originally published on March 11th, 2024. If you would like more articles such as this one, considering subscribing.

The current infatuation with artificial intelligence has been fixated on the fastest processors that make AI possible, such as Nvidia’s GPU chips, and competing efforts from Advanced Micro Devices and Intel.

It’s clear, however, that the thing that will make or break AI going forward is memory, specifically, the circuits that hold the data that feeds data-hungry AI.

Executive summary: Memory circuitry such as DRAM is the key to AI. The increasing size of neural networks for artificial intelligence programs such as ChatGPT calls for more and more memory circuitry to store the data. That is producing a recovery in the DRAM market. However, a lot more innovation will be necessary to solve the problem of the increasing energy consumption and delay introduced by the burgeoning role of memory. Now is a good time to familiarize yourself with thirty-four names participating in the memory market.

The speed and the energy efficiency of memory circuits such as DRAM have not kept pace with that of “logic” chips such as GPUs, leading to a disconnect: The faster the chips go, the more that memory is holding everything back.

Is that an investment opportunity? Reason dictates it should be. The problem of how to have really big, really efficient memory is one of the key challenges of the AI age. Herewith, some thoughts on how to play the memory race.

TURING FORESAW THIS

The intellectual godfather of computing, Alan Turing, realized early on that memory circuits are the key to computing. Computers don’t perform calculations in a vacuum, they transform one set of data into another, and that data has to come from somewhere.

In his first design for a practical computer, the “Automatic Computing Engine,” in 1945, which was never actually built, Turing emphasized the need for tons of memory to keep the calculating part busy, far beyond the punched paper cards that were being used back then.

You could say we are living in a time that is still realizing the profundity of Turing’s insight.

Memory has only become more critical in an age of AI. Today’s AI programs are positively ballooning in scale and their memory appetite is growing apace. A cutting-edge program such as OpenAI’s GPT-4 can take up more than the available eighty gigabytes of DRAM memory of a top-of-the-line Nvidia GPU. There are hundreds of gigabytes more memory needed to house the data used to train AI programs.

Because of AI’s insatiable appetite for data, memory is always its “bottleneck,” its limiting factor.

THE MEMORY BOTTLENECK

Giving AI more memory only brings more problems. Additional memory increases the time that the main chip, the GPU or the microprocessor, spends trying to access that memory. Energy usage soars because most of the power consumption in an AI computer is the energy needed to go from a GPU to its companion memory chip and back.

By one industry estimate, provided by the technical journal Microprocessor Report, it takes a thousand times more power to move a single piece of data from DRAM to the GPU than it does to actually perform the mathematical operation on that piece of data once it’s inside the GPU. Computers can spend ninety percent of their energy just moving data.

The AI networking race: There is a parallel track to the memory race that is somewhat orthogonal to what’s discussed here, which is the battle between Nvidia and Arista Networks for control of the networking of AI. Both companies make the case that because memory is a bottleneck, computer networking needs to change. They’re offering competing technologies to try to win an AI networking race. You can think of that battle as a corollary of the AI memory race, though it is not the focus of this article. For more, see Arista’s Got Nvidia in its Sights.

A chip-industry veteran, Raj Talluri, CEO of tech startup Enovix, remarked in a recent interview I did with him that “A lot of people think generative AI is about compute,” when, in fact, “It actually is not so much about compute as it is about banging on the memory […] to get the data, and that is an inherently power-intensive problem.”

Every time the application processor has to retrieve something from memory, says Talluri, “when you hit on the LPDRAM [“low-power double-data-rate memory,” the most common type of DRAM memory chip in mobile devices], the power burns,” says Talluri. “When the signal travels outside the processor to memory, it just burns.”

Similar testimony comes from a scholar of chip technology, Naveen Verma, a professor in Princeton University’s Department of Electrical Engineering, whom I interviewed last week for ZDNet.

“AI models with a hundred billion to trillions of parameters need to put those parameters in memory,” explains Verma, “but, this movement between memory and processing ends up becoming a true bottleneck, one that those of us who study computer architecture and computer systems view with great concern.”

Verma’s semiconductor startup, EnCharge AI, is working on special kinds of memory chips to drastically reduce the power needed to run AI on energy-constrained devices such as smartphones.

A TIME FOR INNOVATION

What you have, then, is a growth industry in memory, driven by AI’s unstoppable quest to scale to ever-larger programs. But, alongside the need for more memory for AI, there is also a need for more efficient, more intelligent approaches to the design of memory chips.

That is leading to exotic types of memory unlike anything seen before. “High-bandwidth memory,” for example, also known as “HBM,” the state of the art for the most demanding computing, is an example of what’s called “3-D” memory. It’s made up of not one DRAM chip but multiple chips stacked on top of one another, sometimes as many as sixteen DRAM chips, all packed together in a single package.

In other words, you’re seeing the industry go to extremes to build novel designs to confront a crisis. That’s usually a good place to be in technology, when everyone is trying new things  rather than simply cranking out more of the same old thing.

A decade ago, the semiconductor industry started to be interesting again to investors as Moore’s Law, the pattern of constant improvement, sputtered and stopped after many decades of steady progress. What you see in the memory market now, I’m convinced, is the most interesting, most intense area of the resulting renaissance in semiconductors.

A GOOD TIME TO PROSPECT

It is also a good time to be looking at stocks in the memory market because industry sales of both NAND flash memory, the kind that holds files, and DRAM, the kind that makes up main memory in computers, are coming out of a steep decline in 2023 when piles of inventory had to be sold at low prices. The reversal of that situation this year and next should mean improving fortunes for many companies across the industry.

I’ve heard that view expressed many times in recent weeks, in the various conference calls held by companies large and small that play a role in the memory market. Sanjay Mehrotra, CEO of Micron Technology, one of the world’s biggest makers of DRAM and NAND chips, told analysts on the last earnings conference call, in December, that “we are at the very early stages of a multi-year growth phase catalyzed and driven by generative AI.”

There are plenty of other examples. On a call last week, Urs Gantner, the CEO of a specialty chip equipment company, VAT Group, was asked if HBM memory chips will help reinvigorate his company’s sales this year. Gantner’s company makes what are called “valves” that control the air vacuum in the semiconductor manufacturing process in order to increase the precision with which chemicals are deposited onto semiconductor wafers.

Gantner said his customers, the chip makers, “need not only logic chips, they need also the high bandwidth memory — they need these leading-edge chips, and this is what we expect will boost this recovery this year.” Added Gantner, “This is all driven by this AI.”

THE CONTENDERS

I’ve assembled below a table of thirty-four names to take a look at it.

Among contenders, the obvious first candidates are the makers of the memory chips. Micron Technology is one of the best in the industry, and a company that’s seen improving prospects following a downturn last year.

I’ve been writing since a 2015 Barron’s cover story that Micron is one of the most important companies on the planet because it’s a key supplier of memory alongside Hynix and Samsung Electronics, both of South Korea, its two biggest competitors.

Micron’s shares are easier to trade than Hynix and Samsung’s American Depository Receipts. Micron is one of the TL20 stocks worth considering, and its stock is up fifty-three percent since it was put in the group in July of 2022, outperforming the thirty-nine percent of the Nasdaq Composite Index.

If you like picking from left field, you could also consider the fourth-place memory producer, Nanya Technology of Taiwan. I wouldn’t advise doing so, but it’s one to keep in mind.

A second way to play memory are younger firms that have novel approaches to memory technology. One of them is Chandler, Arizona-based Everspin Technologies, which makes a kind of “non-volatile” memory chip, meaning, a chip that retains its data when the power is turned off.

Everspin is pitching that product as an alternative to DRAM and NAND made by Micron and the rest. Another specialty memory maker is Netlist. Both companies are pushing against conventional wisdom by trying to move the market to something very different. You could consider them the longest of long shots on this list.

Another young hopeful is Astera Labs of Silicon Valley, California, which is currently in registration to go public in an IPO lead by Morgan Stanley and JP Morgan. Astera makes specialized chips that ameliorate the memory bottleneck. Based on an industry standard called “Compute Express Link,” or, CXL, the technology allows DRAM memory to be more widely distributed throughout a data center while still moving data efficiently to the processors.

Astera is interesting because the CXL standard is also being embraced by more-established firms that have many other businesses. For example, Broadcom has some CXL products, and so does Smart Global Holdings. That means you can either bet on a “pure play” like Astera or you can be more conservative and go with proven names like the other two.

You’ll notice the list has more chip-equipment names on it than it does makers of chips. That’s because the DRAM titans, Micron and Samsung, spent decades consolidating the memory-chip business in order to bring economies of scale to a commodity business. The equipment makers, on the other hand, have been able to carve out niches in the different parts of memory-chip making and defend those niches.

If you listen to the executives of these equipment companies, they, too, see the whole memory market boosted by AI.

One of the big growth opportunities to which these equipment makers refer is the “AI personal computer,” the AI PC, an idea being promoted by Intel and AMD. I’ve been somewhat skeptical of the AI PC because I really think smartphones will capture more of the average individual’s AI usage.

But equipment makers have a good point: vendors of PCs such as Dell Technologies want to add memory to sell the public on the PC as being a new and improved device. As long as that’s true, PCs will probably see their memory content rise sharply.

Last month, Doug Lawson, the head of marketing for chip-equipment maker Axcelis, which gets twenty percent of its business from tools to make DRAM, told analysts that ”AI PCs look like they could be a big driver of DRAM. Microsoft is requiring sixteen gigabytes per AI PC for Windows 12.”

The chiefs of the equipment companies are even contending that the need for memory circuitry among their customers could be bigger than the need for the logic circuits of GPUs and microprocessors.

For example, Richard Blickman, CEO of BE Semiconductor, which makes tools to package together multiple chips, told analysts on the company’s conference call last month that the size of the opportunity for his company’s tools can be half again as big for memory as it is for microprocessors and GPUs because of the way HBM piles together multiple chips.

“We maintain the view that the volume in HBM is potentially significantly larger than for logic,” he said, “maybe a ratio of 1:4, 1:6 is what is often used — it can even be more than that.”

If I had to pick three names from this list, I’d pick the ones I’ve already picked for the TL20 group of stocks worth considering, Micron, Axcelis and BE Semi. I would pick Lam Research and KLA as close fourths and fifths, given their historical exposure to the memory-chip market.

But I have not comprehensively assessed how memory will affect the quarterly finances of each of the other companies, so I encourage you to do your own due diligence and come to your own conclusions.

With a recovering market for memory chips acting as a tide that lifts all ships, now is the time to become familiar with the next stage of innovation brought about by AI that may distinguish future leaders in the market.

Disclosure: Tiernan Ray owns no stock in anything that he writes about, and there is no business relationship between Tiernan Ray LLC, the publisher of The Technology Letter, and any of the companies covered. See here for further details.