AI Agents Need Inference Infrastructure That Can Keep Up with Growing Token Use

Author: Lucy.·Wednesday, November 4, 2026
AI Agents Need Inference Infrastructure That Can Keep Up with Growing Token Use

On November 4, I attended the AMD Advancing AI World Tour & AI DevDay Seoul at The Westin Seoul Parnas.

A central theme throughout DevDay was the infrastructure needed to make AI widely useful in everyday work. Alongside AMD’s hardware and software strategy, I heard how South Korean companies including Moreh, NAVER Cloud, Clevi, Nota AI, and Lablup approach operations and optimization.

The more work we give AI agents, the more tokens they consume, and the more important it becomes to have infrastructure that can handle that inference quickly and reliably. As the age of AI agents approaches, choosing the right model also means thinking about where and how to run it, and at what cost.

The Growing Token Demands of AI Inference

When we ask a chatbot questions and receive answers, token consumption grows with the length and number of exchanges. AI agents that autonomously carry out long, multistep tasks work differently. To act on a single instruction, they repeatedly make plans, find information, use tools, check results, and decide what to do next.

Coding agents are a good example. An AI generates code, runs it, reads the errors, makes changes, and tests it again. The user may have typed just one line, “Build this feature,” but producing the result involves many model calls and input and output tokens. Tokens, the units of information processed by AI, accumulate rapidly across the entire workflow.

As we assign more work to agents and run more of them at the same time, demand for inference grows. Even if better model efficiency lowers the price per token, total costs can rise when usage grows faster. That is why expanding AI use requires us to consider inference economics and design for them.

AMD forecast of monthly token demand growth for general AI tasks, consumer agents, and enterprise agents from 2024 to 2030Chart recreated from the token-demand forecast presented by AMD at the event.


Moreh on the Economics of Token Services

Moreh described a shift in customer demand from renting GPU servers to using APIs that connect directly to existing agents. Customers want a service that gives them quick access to the models they need, delivers responses at a suitable speed, and makes token costs predictable.

Differences between hardware do not disappear in this model. Managing those differences becomes the service operator’s responsibility. Operators need to support new models, optimize computation for their hardware, distribute incoming requests, and reduce memory and network bottlenecks. Even on the same hardware, software and operating choices affect the throughput a service can deliver and its cost.

One particularly useful distinction was between the speed an individual user experiences and the total throughput of a single GPU. Batching requests can raise overall throughput while slowing responses for individual users. Conversely, concentrating resources on fewer requests can speed up responses while reducing the hardware’s total throughput.

The competitiveness of inference infrastructure is not determined by peak speed alone. How reliably a service delivers the required quality and throughput within the time customers are willing to wait shapes both its pricing and its long-term viability.


Agent Workloads Extend Beyond the GPU

AMD’s EPYC session highlighted the CPU’s role in the age of agents. Generating code and actually running that code are different tasks. File processing, compilation, testing, and database access all require CPUs, memory, storage, and networking.

The presentation also treated sandboxes, which run agent-generated code and tools in isolated environments, as a distinct part of the infrastructure. As the number of agents grows, so does the importance of providing space and resources for concurrent execution, managing task permissions, and handling failed runs.

From this perspective, faster GPU token generation is only part of the picture. Even if a model responds quickly, slow searches or tests can keep users waiting longer than expected. Model response speed and the time an agent takes to finish a task need to be measured separately.

NAVER Cloud’s case study extended this discussion to day-to-day operations. Adding servers requires planning for rack space, power, and cooling as well as processors. The presentation described a process of qualifying each new CPU generation, validating it on management nodes and within the operations platform, and then rolling it out to services. It showed why validating and deploying hardware in real workloads matters as much as choosing good equipment.


Local AI Needs Optimization to Be Cost-Effective

As token use grows, local AI that handles some tasks on company servers or PCs becomes an important option. It can replace recurring API spending with hardware, electricity, and operating costs, while keeping sensitive data in-house. But simply putting a model on a machine does not resolve cost and performance challenges.

South Korean companies at the event addressed different parts of this challenge.

  • Clevi focused on input documents. Extracting only the text from HWP documents (HWP is a Korean word-processing format), PDFs, or scanned documents can lose relationships within tables, the placement of images, and page structure. Converting documents into a form AI can read while preserving the original source material is a starting point for agents to make sound judgments.
  • Nota AI showed how to divide work across devices. Its example split input-context processing and answer generation between an NPU and a GPU, while sharing model weights to improve memory efficiency. The approach optimizes not only the model but also how it runs and how work is assigned to hardware.
  • Lablup showed how the same model can perform differently in different execution environments. Specifications such as unified memory support or the ability to run low-precision models do not tell us the actual speed. Computation paths and memory usage need to suit the hardware.

These examples share a common lesson: there is a great deal to solve between a model and the work it needs to do. Software that reads documents correctly, uses memory efficiently, and reduces device-specific bottlenecks is what turns hardware performance into value for a service.


Businesses Should Compare the Cost of a Completed Task

Applying these presentations to business AI adoption gives us a clear basis for decisions. Beyond token prices and hardware costs, we need to look at the total cost of completing a task at the required quality. The “token maxing” trend that attracted attention earlier this year has evolved in just a few months toward a “token economy” focused on maximizing the work delivered for a given token spend. AMD has reportedly also moved from unlimited monthly token use per employee to a capped allowance.

  1. Quality: Does the result meet actual business requirements? What is missing or incorrect, and how much time does a person need to spend correcting it?
  2. Time and cost: How long does it take to finish, not just produce the first response? What is the combined cost of token usage, retries, hardware, and operations?
  3. Data and operations: What information leaves the organization? Can the same standard be maintained as concurrent workloads grow or models change?

The results can guide a choice between cloud, local, or a combination of the two. Running frequent, easy-to-verify tasks locally and sending tasks that need higher performance to the cloud is another approach to assess against actual quality and cost. The goal should be to manage costs while maintaining task success rates, rather than simply increasing the share of work done locally.


The Foundation for Sustained AI Use

Taken together, the day’s presentations showed how the spread of AI agents increases model calls while also expanding the execution, data, and operational infrastructure around them. As we entrust more work to agents, token consumption and inference infrastructure will become increasingly important.

AIB needs this perspective when comparing AI and finding ways to apply it to customers’ challenges. Alongside evaluating which model produces good answers, we need to examine how quickly and reliably it completes real work, at a cost the business can sustain. Building that foundation will be an important competitive advantage in the age of AI agents.

Event programs: AMD ADVANCING AI WORLD TOUR SEOUL 2026 · AMD AI DEVDAY SEOUL 2026