Siamak Tavallaei — Big Data Feeds Big Compute through Memory
Register to attend the presentation on September 30
For most of the AI boom, the infrastructure conversation has revolved around compute: more GPUs, larger clusters, faster accelerators and ever-larger scale-up systems. Siamak Tavallaei believes the industry is reaching the point where that conversation has to change.
As AI systems become larger, and especially as agentic AI begins generating, retrieving and manipulating enormous quantities of data, the challenge increasingly becomes how quickly and efficiently data can be moved between memory and compute.
“The bottleneck is not the compute anymore,” Tavallaei says in his upcoming AI Leadership Forum presentation. “The interconnect to memory is—and the memory itself.”
That observation is the starting point for Tavallaei’s session, “Big Data Feeds Big Compute through Memory,” airing September 30 as part of Episode 1 of the Endless Memory with CXL series. And few people are better positioned to explain where this transition is headed.
Register to attend the presentation on September 30
Three Decades at the Intersection of Compute, Memory and Infrastructure
Tavallaei is Senior Principal Engineer at Samsung Semiconductor and a member of the CXL Consortium Board of Directors. His career spans more than three decades of work on server architecture, memory systems, interconnects and hyperscale infrastructure, including senior technical positions at Google Cloud, Microsoft Azure, HP and Compaq.
His involvement with CXL goes back to the Consortium’s formative years, including service as President, Board Director, Independent Advisor to the Board and founding Co-Chair of the Technical Task Force. That background gives his presentation an unusually broad perspective. Rather than beginning with a particular CXL device or implementation, Tavallaei begins with a fundamental question:
What happens when the growth of AI compute collides with the limits of memory capacity and data movement? His answer is essentially a new version of Parkinson’s Law: Compute grows to fill all the allocated memory.
From Big Data to Big Compute
Tavallaei draws an important distinction between the Big Data era and today’s Big Compute era. Big Data drove enormous growth in persistent storage—first hard drives and then NAND flash and SSDs. Much of that data was human generated, labeled and stored permanently.
AI changes the equation. Increasingly, data is machine-generated. AI systems need enormous working sets, scratchpad capacity, caches and longer-term memory that can be accessed quickly during execution. Tavallaei describes this as a need for increasingly large amounts of fast, temporary storage.
That changes the infrastructure challenge from simply storing more data to getting the right data to the right compute engine fast enough to keep increasingly expensive processors productive.
And that is where memory architecture and CXL enter the picture.
Samsung’s Research Points to What’s Possible
Tavallaei doesn’t make the case only conceptually. He presents results from Samsung research examining three fundamental constraints: latency, bandwidth and capacity.
The results he discusses are striking. Using CXL technologies, the research demonstrated a 40% latency reduction using hot-page monitoring, increased bandwidth through near-memory processing, and 50% more capacity from already-deployed memory using CXL memory pooling.
Those results illustrate something important about CXL: this isn’t simply about attaching more DIMMs to a server. It’s about changing the way memory can be expanded, shared, pooled and ultimately utilized across AI infrastructure.
Register to attend the presentation on September 30
AI Memory Doesn’t Have One Size
Tavallaei then applies the concept to a familiar AI workload: Retrieval-Augmented Generation, or RAG.
A RAG pipeline moves through search, databases, embeddings, vector databases, prompts, LLMs and responses. Different stages need different amounts and types of memory. Provision every compute node for its peak requirement and enormous amounts of expensive memory can sit idle.
Pooling changes that equation. Instead of provisioning each node for peak demand, Tavallaei describes compute nodes drawing from a shared memory pool as workloads change—allowing their effective memory footprints to “inflate or deflate” according to demand.
That is an important glimpse into where AI infrastructure may be heading: memory becoming a more dynamic infrastructure resource rather than a fixed quantity permanently attached to each processor.
Register to attend the presentation on September 30
A New AI Memory Hierarchy Is Emerging
Perhaps the most important part of Tavallaei’s presentation is his view of the emerging memory hierarchy. Traditional servers had a relatively straightforward stack: processor cache, CPU memory, local storage and network storage. AI infrastructure is creating something much richer.
HBM sits closest to accelerators. CPU-attached DRAM provides another tier. CXL can extend memory capacity. Additional tiers can incorporate pooled memory and storage. Scale-up fabrics connect accelerators, while networks connect increasingly large clusters. The challenge is connecting all of those resources without creating a collection of proprietary memory architectures.
Tavallaei summarizes CXL’s potential role in remarkably simple terms: “CXL is the link between the bulk memory and the AI/ML engines.”
The AI memory hierarchy is expanding from a handful of server-centric tiers to an infrastructure-wide
hierarchy spanning HBM, CPU memory, CXL memory, pooled memory and storage.
Why Standardization May Be the Key
The final part of Tavallaei’s argument goes beyond memory capacity. AI infrastructure is developing multiple scale-up fabrics and specialized interconnects. Building customized memory controllers and devices for every architecture would be expensive and could fragment the market.
Tavallaei proposes another direction: Scale up natively, then bridge to standard components. That is where CXL could become especially important. His architecture envisions hybrid switches bridging specialized scale-up fabrics with standardized CXL memory, potentially giving different AI architectures access to a common ecosystem of memory devices.
Instead of requiring a unique memory architecture for every accelerator fabric, CXL could provide a common memory interface across multiple architectures. Tavallaei describes CXL as a possible “next standard for memory interconnect beyond HBM, DDR or LPDDR.”
CXL Is Moving from Expansion Technology to AI Architecture
This may be the biggest reason to watch Tavallaei’s September 30 presentation. Much of the early discussion around CXL focused on relatively straightforward memory expansion. Tavallaei paints a considerably larger picture. CXL can potentially connect bulk memory to CPUs and AI engines, enable pooling, expand capacity, support a richer memory hierarchy and provide a standardized bridge between specialized scale-up fabrics and memory.
And standardization matters because an ecosystem only really accelerates when hardware vendors, software developers and customers can build around common interfaces. Tavallaei’s closing argument is that CXL provides exactly that foundation: a backward-compatible technology that protects supplier investment while encouraging software development around common hardware. The result could be a fundamental change in the way we think about AI infrastructure.
For years, the industry has concentrated on building bigger compute. The next challenge may be making sure those processors have access to enough memory—and that data can reach them fast enough to keep them working. Big Data feeds Big Compute through memory. And on September 30, Siamak Tavallaei will explain why CXL could become one of the critical technologies connecting them.
Join us September 30 for Episode 1 of the AI Leadership Forum’s Endless Memory with CXL series.
