AI Chip Race Shifts from Compute to Memory-Centric Design

Raja Koduri, CEO of OXMIQ Labs, took part in the fourth installment of SK hynix's 'AI Ecosystem' series, which explained that the center of gravity in AI computing is moving from compute units to data supply and memory. The series views AI not as models or algorithms alone but as an ecosystem where software, data center infrastructure, and semiconductor and memory technologies intertwine. It said that as workloads grow more advanced around inference and agentic AI, performance benchmarks are also changing. Now, not just computing power but where data is stored, how fast it moves, and how efficiently it is processed determine the performance and cost of AI systems.

The article points out that under Moore's Law, AI accelerators have reached petaflops-class performance, but the bandwidth that supplies data to compute units has not kept pace. Multiplication and addition operations consume energy on the order of a few femtojoules (fJ), while reading data from on-chip SRAM consumes more energy, and fetching from off-chip DRAM can cost hundreds to thousands of times more. As a result, the performance bottleneck is shifting toward data supply: compute resources limit large-scale training and large-batch inference, while memory bandwidth limits low-latency, small-batch inference.

Transformer architectures, which handle massive parameter counts and long contexts, are undermining the assumption of cache reuse. Model weights are difficult to fit entirely on-chip, and lower reuse forces systems to read data repeatedly from DRAM or HBM. The KV cache grows in proportion to context length, and when long-context agents run concurrently at scale, the existing DDR- and GDDR-centric structures struggle to sustain data flow, the article explains. It adds that optimization techniques such as FlashAttention, PagedAttention, and speculative decoding were developed to reduce memory access and data movement, but they cannot eliminate the physical cost of data movement.

HBM (High Bandwidth Memory) is a structural answer to this problem: it vertically stacks multiple DRAM dies and connects them to compute units through TSVs and interposers, reducing distance from centimeters to millimeters. According to the article, the author took part in developing a GPU with HBM at AMD in 2015, and at Intel led development of Ponte Vecchio, which integrated 47 chiplets and four HBM2e stacks in a single package. However, HBM bandwidth is constrained by interface area, signaling speed, power delivery, and thermal design, and as generations advance, simply adding bandwidth may not sustain performance gains.

Advanced packaging has become part of system architecture, co-designing compute dies, memory stacks, interconnects, power delivery, and heat dissipation paths rather than being mere back-end assembly. 2.5D and 3D integration, chiplets, interposers, and hybrid bonding all aim to reduce the physical distance between data and compute. Hybrid bonding directly joins copper to copper instead of using microbumps, creating more connections and wider data channels in the same area while lowering energy per bit. The article stresses that detailed issues such as PCB resonance, microbump fatigue, and thermal interference determine performance and reliability, so reflecting them precisely is important in memory-centric design.

Processing-in-Memory (PIM) performs some computation inside or near memory instead of moving data to compute units. The article sees PIM not as a replacement for CPUs or GPUs but as well suited to complementing existing accelerators in tasks where data movement costs exceed computation costs. For repeated simple operations on large tensors, it says energy consumption can be reduced by 10 to 100 times depending on the workload. However, bringing compute functions into memory stacks makes heat dissipation near temperature-sensitive DRAM a difficult challenge, requiring a system-level approach that coordinates memory, compute, packaging, thermal design, and software.

The article summarizes that the benchmark of the AI chip race has broadened from individual compute dies to entire packages and systems. If the CPU era was a contest of processor performance and the GPU era a contest of massively parallel processing, the AI era is one in which designs that tie compute, memory, packaging, interconnects, and software into a single system and minimize unnecessary data movement determine competitiveness. The author concludes that the bigger challenge is how efficiently the available computing power can be used: keeping data close to compute, reducing movement, and extracting more computational value from the same data make the difference. The next installment will address how this trend connects to physical AI and on-device AI.

Raja Koduri, CEO of OXMIQ Labs, took part in the fourth installment of SK hynix's 'AI Ecosystem' series, which explained that the center of gravity in AI computing is moving from compute units to data supply and memory. The series views AI not as models or algorithms alone but as an ecosystem where software, data center infrastructure, and semiconductor and memory technologies intertwine. It said that as workloads grow more advanced around inference and agentic AI, performance benchmarks are also changing. Now, not just computing power but where data is stored, how fast it moves, and how efficiently it is processed determine the performance and cost of AI systems.

The article points out that under Moore's Law, AI accelerators have reached petaflops-class performance, but the bandwidth that supplies data to compute units has not kept pace. Multiplication and addition operations consume energy on the order of a few femtojoules (fJ), while reading data from on-chip SRAM consumes more energy, and fetching from off-chip DRAM can cost hundreds to thousands of times more. As a result, the performance bottleneck is shifting toward data supply: compute resources limit large-scale training and large-batch inference, while memory bandwidth limits low-latency, small-batch inference.

Transformer architectures, which handle massive parameter counts and long contexts, are undermining the assumption of cache reuse. Model weights are difficult to fit entirely on-chip, and lower reuse forces systems to read data repeatedly from DRAM or HBM. The KV cache grows in proportion to context length, and when long-context agents run concurrently at scale, the existing DDR- and GDDR-centric structures struggle to sustain data flow, the article explains. It adds that optimization techniques such as FlashAttention, PagedAttention, and speculative decoding were developed to reduce memory access and data movement, but they cannot eliminate the physical cost of data movement.

HBM (High Bandwidth Memory) is a structural answer to this problem: it vertically stacks multiple DRAM dies and connects them to compute units through TSVs and interposers, reducing distance from centimeters to millimeters. According to the article, the author took part in developing a GPU with HBM at AMD in 2015, and at Intel led development of Ponte Vecchio, which integrated 47 chiplets and four HBM2e stacks in a single package. However, HBM bandwidth is constrained by interface area, signaling speed, power delivery, and thermal design, and as generations advance, simply adding bandwidth may not sustain performance gains.

Advanced packaging has become part of system architecture, co-designing compute dies, memory stacks, interconnects, power delivery, and heat dissipation paths rather than being mere back-end assembly. 2.5D and 3D integration, chiplets, interposers, and hybrid bonding all aim to reduce the physical distance between data and compute. Hybrid bonding directly joins copper to copper instead of using microbumps, creating more connections and wider data channels in the same area while lowering energy per bit. The article stresses that detailed issues such as PCB resonance, microbump fatigue, and thermal interference determine performance and reliability, so reflecting them precisely is important in memory-centric design.

Processing-in-Memory (PIM) performs some computation inside or near memory instead of moving data to compute units. The article sees PIM not as a replacement for CPUs or GPUs but as well suited to complementing existing accelerators in tasks where data movement costs exceed computation costs. For repeated simple operations on large tensors, it says energy consumption can be reduced by 10 to 100 times depending on the workload. However, bringing compute functions into memory stacks makes heat dissipation near temperature-sensitive DRAM a difficult challenge, requiring a system-level approach that coordinates memory, compute, packaging, thermal design, and software.

The article summarizes that the benchmark of the AI chip race has broadened from individual compute dies to entire packages and systems. If the CPU era was a contest of processor performance and the GPU era a contest of massively parallel processing, the AI era is one in which designs that tie compute, memory, packaging, interconnects, and software into a single system and minimize unnecessary data movement determine competitiveness. The author concludes that the bigger challenge is how efficiently the available computing power can be used: keeping data close to compute, reducing movement, and extracting more computational value from the same data make the difference. The next installment will address how this trend connects to physical AI and on-device AI.