SageMaker Adds GPU Partitioning

Amazon SageMaker Inference now supports GPU partitioning for model endpoints that use inference components. The feature uses NVIDIA Multi-Instance GPU (MIG) technology to divide a physical accelerator into hardware-isolated partitions, each with dedicated compute and memory. Models can occupy separate partitions on one GPU, and users can select a fraction of a newer accelerator instead of reserving an entire device. Single-model endpoints, serverless endpoints and heterogeneous instance pools are not supported.

Customers previously had three options when they had more models than accelerators: run fewer, larger endpoints and batch requests, use multi-model endpoints, or manage Kubernetes themselves. Batching works when models are interchangeable; multi-model endpoints share a GPU without isolation and incur a loading delay when an unloaded model is first requested. Kubernetes supports MIG but requires users to manage drivers, partition settings, device plugins and scheduling. SageMaker lets users select a partition profile while the service handles configuration, placement and scaling. Salesforce helped define requirements and validate the initial implementation, and said it is preparing to test the solution with representative production workloads.

MIG can split a GPU into as many as seven instances, depending on the accelerator. Dedicated memory bandwidth is intended to keep one model’s traffic spikes from affecting another model’s p99 latency; hardware boundaries also contain crashes and memory errors. Models can remain loaded in their own partitions, avoiding load penalties and eviction, while partition performance supports load testing and mapping GPU costs to a model, team or tenant. A layout is selected when an endpoint is created, and MIG only allows layouts to change while partitions are idle. Models needing more than one GPU still require whole devices.

The feature adds two optional API fields: AcceleratorPartitionLayouts on a production variant in CreateEndpointConfig, and AcceleratorPartitionType on an inference component. Each component copy uses one partition of the selected type; increasing CopyCount puts more copies on free partitions. NumberOfAcceleratorDevicesRequired remains 1 and refers to one partition when a type is selected. CPU cores and host memory remain specified separately. Requests through InvokeEndpoint do not change. Provisioning the endpoint includes partitioning the accelerators, which added roughly 17 seconds in the company’s testing.

Available profiles depend on the GPU. P4d instances with 40 GB A100s and P5 instances with 80 GB H100s support up to seven partitions; G7 instances with 32 GB RTX PRO 4500 Blackwell GPUs support two, and G7e instances with 96 GB RTX PRO 6000 Blackwell Server Edition GPUs support four. A profile’s name states its compute units and GPU memory, and the sum of profiles in a layout cannot exceed either resource available on one device. A P4d using mig-1g.10gb needs an R525 or newer NVIDIA driver; the default image uses driver 470.256.02, while al2023-ami-sagemaker-inference-gpu-4-1 supplies a 580 driver.

Choosing a profile depends on workload demands. Compute supports prefill, which processes the prompt, while GPU memory supports token generation and limits KV cache capacity, context length and concurrent sequences. An 8B model at FP16 uses about 16 GB for weights, in addition to KV cache and 1–2 GB of runtime overhead. The article says the model can serve an 8,000-token context on a 40 GB partition.

The post identifies portfolios of small or mid-size models, per-customer model serving, multi-stage pipelines, small request-path classifiers, and canary models as possible uses. GPU partitions isolate GPU memory and throughput, but models on the same instance still share host CPU, system memory, storage, networking and endpoint routing. For a performance comparison, AWS tested one mig-3g.40gb H100 partition against a whole, unpartitioned A100-40GB, using genai-bench to generate load from the same client. The supplied article text ends before reporting the comparison’s results.

Amazon SageMaker Inference now supports GPU partitioning for model endpoints that use inference components. The feature uses NVIDIA Multi-Instance GPU (MIG) technology to divide a physical accelerator into hardware-isolated partitions, each with dedicated compute and memory. Models can occupy separate partitions on one GPU, and users can select a fraction of a newer accelerator instead of reserving an entire device. Single-model endpoints, serverless endpoints and heterogeneous instance pools are not supported.

Customers previously had three options when they had more models than accelerators: run fewer, larger endpoints and batch requests, use multi-model endpoints, or manage Kubernetes themselves. Batching works when models are interchangeable; multi-model endpoints share a GPU without isolation and incur a loading delay when an unloaded model is first requested. Kubernetes supports MIG but requires users to manage drivers, partition settings, device plugins and scheduling. SageMaker lets users select a partition profile while the service handles configuration, placement and scaling. Salesforce helped define requirements and validate the initial implementation, and said it is preparing to test the solution with representative production workloads.

MIG can split a GPU into as many as seven instances, depending on the accelerator. Dedicated memory bandwidth is intended to keep one model’s traffic spikes from affecting another model’s p99 latency; hardware boundaries also contain crashes and memory errors. Models can remain loaded in their own partitions, avoiding load penalties and eviction, while partition performance supports load testing and mapping GPU costs to a model, team or tenant. A layout is selected when an endpoint is created, and MIG only allows layouts to change while partitions are idle. Models needing more than one GPU still require whole devices.

The feature adds two optional API fields: AcceleratorPartitionLayouts on a production variant in CreateEndpointConfig, and AcceleratorPartitionType on an inference component. Each component copy uses one partition of the selected type; increasing CopyCount puts more copies on free partitions. NumberOfAcceleratorDevicesRequired remains 1 and refers to one partition when a type is selected. CPU cores and host memory remain specified separately. Requests through InvokeEndpoint do not change. Provisioning the endpoint includes partitioning the accelerators, which added roughly 17 seconds in the company’s testing.

Available profiles depend on the GPU. P4d instances with 40 GB A100s and P5 instances with 80 GB H100s support up to seven partitions; G7 instances with 32 GB RTX PRO 4500 Blackwell GPUs support two, and G7e instances with 96 GB RTX PRO 6000 Blackwell Server Edition GPUs support four. A profile’s name states its compute units and GPU memory, and the sum of profiles in a layout cannot exceed either resource available on one device. A P4d using mig-1g.10gb needs an R525 or newer NVIDIA driver; the default image uses driver 470.256.02, while al2023-ami-sagemaker-inference-gpu-4-1 supplies a 580 driver.

Choosing a profile depends on workload demands. Compute supports prefill, which processes the prompt, while GPU memory supports token generation and limits KV cache capacity, context length and concurrent sequences. An 8B model at FP16 uses about 16 GB for weights, in addition to KV cache and 1–2 GB of runtime overhead. The article says the model can serve an 8,000-token context on a 40 GB partition.

The post identifies portfolios of small or mid-size models, per-customer model serving, multi-stage pipelines, small request-path classifiers, and canary models as possible uses. GPU partitions isolate GPU memory and throughput, but models on the same instance still share host CPU, system memory, storage, networking and endpoint routing. For a performance comparison, AWS tested one mig-3g.40gb H100 partition against a whole, unpartitioned A100-40GB, using genai-bench to generate load from the same client. The supplied article text ends before reporting the comparison’s results.