NVIDIA Shows Two-Station GLM-5.3 Serving

NVIDIA’s November 13, 2026 tutorial shows how to serve GLM-5.3 NVFP4 across two DGX Stations with vLLM because its roughly 464 GB of model weights exceed the 252 GB of HBM3e on one GB300 GPU. The model has 753 billion parameters, with 40 billion activated per token. Although this sparse activation reduces computation, all weights must remain available because each token can be routed to different experts. CPU weight offload is disabled.

The two Stations connect over ConnectX-8 RDMA (a high-speed link for direct data transfer) and run tensor parallelism with TP=2, splitting model weights and computation across both GB300 GPUs. Station A hosts the API and returns answers through one OpenAI-compatible endpoint; Station B runs a worker. The tutorial requires both Stations to have at least 464 GB of free disk space for their checkpoint copies. It downloads the pinned checkpoint on A and copies it to B; vLLM partitions weights across the GPUs at load time, without further quantization.

The distributed vLLM service enables multi-token prediction, a form of speculative decoding in which the model drafts four tokens per step and the full model verifies them. The number accepted varies by prompt. It also stores the KV cache in FP8, which the tutorial says halves cache memory compared with 16-bit storage, and enables prefix caching to avoid reprocessing prompt prefixes already seen by the server. The launch settings include a 98,304-token context ceiling for input plus output, 93% GPU-memory utilization, and a maximum of 16 sequences.

AIPerf tested the service on the SPEED-Bench Coding dataset. Output throughput peaked at 607.8 tokens per second at concurrency 16; at concurrency 32, it fell to 385.0 tokens per second, with mean request latency of 453.5 seconds. NVIDIA recommends connecting and validating the Stations before deployment, consulting its Local AI recipe repository for configurations, and rerunning AIPerf with custom prompts to find suitable concurrency for a workload. The tutorial warns that the unauthenticated API, launched on all Station A interfaces, including management, should be firewalled to intended clients; authentication and TLS are advised before wider access.

NVIDIA’s November 13, 2026 tutorial shows how to serve GLM-5.3 NVFP4 across two DGX Stations with vLLM because its roughly 464 GB of model weights exceed the 252 GB of HBM3e on one GB300 GPU. The model has 753 billion parameters, with 40 billion activated per token. Although this sparse activation reduces computation, all weights must remain available because each token can be routed to different experts. CPU weight offload is disabled.

The two Stations connect over ConnectX-8 RDMA (a high-speed link for direct data transfer) and run tensor parallelism with TP=2, splitting model weights and computation across both GB300 GPUs. Station A hosts the API and returns answers through one OpenAI-compatible endpoint; Station B runs a worker. The tutorial requires both Stations to have at least 464 GB of free disk space for their checkpoint copies. It downloads the pinned checkpoint on A and copies it to B; vLLM partitions weights across the GPUs at load time, without further quantization.

The distributed vLLM service enables multi-token prediction, a form of speculative decoding in which the model drafts four tokens per step and the full model verifies them. The number accepted varies by prompt. It also stores the KV cache in FP8, which the tutorial says halves cache memory compared with 16-bit storage, and enables prefix caching to avoid reprocessing prompt prefixes already seen by the server. The launch settings include a 98,304-token context ceiling for input plus output, 93% GPU-memory utilization, and a maximum of 16 sequences.

AIPerf tested the service on the SPEED-Bench Coding dataset. Output throughput peaked at 607.8 tokens per second at concurrency 16; at concurrency 32, it fell to 385.0 tokens per second, with mean request latency of 453.5 seconds. NVIDIA recommends connecting and validating the Stations before deployment, consulting its Local AI recipe repository for configurations, and rerunning AIPerf with custom prompts to find suitable concurrency for a workload. The tutorial warns that the unauthenticated API, launched on all Station A interfaces, including management, should be firewalled to intended clients; authentication and TLS are advised before wider access.