Together AI Details GPU Cluster Auto-Repair

Together AI describes an automatic repair system for its GPU clusters that detects hardware and software faults, recommends a repair, and lets customers choose which actions need approval and when running workloads may be interrupted. The company says customers and its internal inference platform have used the capability since it announced the features earlier this year. The system is generally available with health checks, repair recommendations, per-fault and per-node approval policies, wait-for-idle controls, escalation rules, safety limits, and repair through the API or console. Node-label remediation contracts for customer detection systems remain in preview.

The system checks for faults both inside virtual machines and on physical hosts. In a VM, exporters monitor kernel logs for Xid errors and GPUs disconnecting from the PCIe bus, DCGM for double-bit ECC errors, row-remap failures, PCIe replay rates and thermal throttling, and Node Problem Detector for read-only filesystems, XFS shutdowns, kernel deadlocks, and repeated kubelet or containerd restarts. A Slurm exporter reports nodes marked DOWN or DRAIN. On hosts, checks track InfiniBand links and host reachability. Alerts include plain-language descriptions and machine-readable evidence; the control plane passes that information unchanged to the console, API and notifications. Alerts without a mapped repair are logged and sent to support, while issues such as disk pressure are recorded without triggering a reboot.

A failed health check is required before a repair can proceed. Customers can leave recommendations awaiting approval or enable automatic action for selected faults, including consent for individual Xid codes. For example, they can automatically approve reboots for Xid 119 and 95 while requiring approval for migrations. Repairs can wait until a node has no GPU-requesting Kubernetes pods or no Slurm allocation. The default maximum wait is 2 hours, capped at 24 hours; customers can instead choose not to interrupt running jobs, removing the time limit. Each repair records who requested and approved it, the rule used, and the time.

Customers can request repairs through the API, console, or, in preview, node labels and annotations written by their own detectors. Requests support Reboot, Quick reprovision, or Migrate; Remove remains a human action. Detector name and request ID prevent repeated scans from triggering the same repair twice. These requests still face evidence checks, capacity checks and per-cluster limits. Each repair follows cordon, wait, drain, action and rejoin. Reboot restarts the VM while preserving local scratch and temporary data; quick reprovision rebuilds it from a clean image and loses local data; migration rebuilds it on different hardware and sends the old host for testing; removal sends the node for repair or replacement.

The escalation ladder moves up after the same fault returns on the same machine following two attempts at the same action: a third reboot recommendation becomes reprovisioning, and a third reprovision recommendation becomes migration. Attempts older than a week, cancelled recommendations and faults that clear themselves do not count. Migration requires a critical alert currently active or one recorded on the same machine within the last ten days; without that evidence, the system uses quick reprovision. Host power cycling is in preview for faults such as Xid 79, which may require cutting GPU power. Hot-spare capacity can be reserved for migrations.

To limit damage during correlated failures, alerts must persist for a configured duration, repairs trigger roughly 30 minutes of cooldown, and concurrency budgets are based on cluster size, with one repair at a time per node pool. A circuit breaker stops bursts of automatic repairs; each repair can target only one node, and recommendations disappear if their alert clears first. Writes are idempotent, so two control-plane replicas acting on the same node create one repair.

After migration or removal, the vacated physical machine automatically runs GPU, NCCL, InfiniBand and storage checks before returning to the pool. A passed result returns it without a ticket; an explicit failure files a hardware ticket with test results and logs, while an aborted check restores the machine's previous state. Together says the repair process is designed to work across accelerator vendors, with vendor-specific detection and diagnostic checks. Customers can go

Together AI describes an automatic repair system for its GPU clusters that detects hardware and software faults, recommends a repair, and lets customers choose which actions need approval and when running workloads may be interrupted. The company says customers and its internal inference platform have used the capability since it announced the features earlier this year. The system is generally available with health checks, repair recommendations, per-fault and per-node approval policies, wait-for-idle controls, escalation rules, safety limits, and repair through the API or console. Node-label remediation contracts for customer detection systems remain in preview.

The system checks for faults both inside virtual machines and on physical hosts. In a VM, exporters monitor kernel logs for Xid errors and GPUs disconnecting from the PCIe bus, DCGM for double-bit ECC errors, row-remap failures, PCIe replay rates and thermal throttling, and Node Problem Detector for read-only filesystems, XFS shutdowns, kernel deadlocks, and repeated kubelet or containerd restarts. A Slurm exporter reports nodes marked DOWN or DRAIN. On hosts, checks track InfiniBand links and host reachability. Alerts include plain-language descriptions and machine-readable evidence; the control plane passes that information unchanged to the console, API and notifications. Alerts without a mapped repair are logged and sent to support, while issues such as disk pressure are recorded without triggering a reboot.

A failed health check is required before a repair can proceed. Customers can leave recommendations awaiting approval or enable automatic action for selected faults, including consent for individual Xid codes. For example, they can automatically approve reboots for Xid 119 and 95 while requiring approval for migrations. Repairs can wait until a node has no GPU-requesting Kubernetes pods or no Slurm allocation. The default maximum wait is 2 hours, capped at 24 hours; customers can instead choose not to interrupt running jobs, removing the time limit. Each repair records who requested and approved it, the rule used, and the time.

Customers can request repairs through the API, console, or, in preview, node labels and annotations written by their own detectors. Requests support Reboot, Quick reprovision, or Migrate; Remove remains a human action. Detector name and request ID prevent repeated scans from triggering the same repair twice. These requests still face evidence checks, capacity checks and per-cluster limits. Each repair follows cordon, wait, drain, action and rejoin. Reboot restarts the VM while preserving local scratch and temporary data; quick reprovision rebuilds it from a clean image and loses local data; migration rebuilds it on different hardware and sends the old host for testing; removal sends the node for repair or replacement.

The escalation ladder moves up after the same fault returns on the same machine following two attempts at the same action: a third reboot recommendation becomes reprovisioning, and a third reprovision recommendation becomes migration. Attempts older than a week, cancelled recommendations and faults that clear themselves do not count. Migration requires a critical alert currently active or one recorded on the same machine within the last ten days; without that evidence, the system uses quick reprovision. Host power cycling is in preview for faults such as Xid 79, which may require cutting GPU power. Hot-spare capacity can be reserved for migrations.

To limit damage during correlated failures, alerts must persist for a configured duration, repairs trigger roughly 30 minutes of cooldown, and concurrency budgets are based on cluster size, with one repair at a time per node pool. A circuit breaker stops bursts of automatic repairs; each repair can target only one node, and recommendations disappear if their alert clears first. Writes are idempotent, so two control-plane replicas acting on the same node create one repair.

After migration or removal, the vacated physical machine automatically runs GPU, NCCL, InfiniBand and storage checks before returning to the pool. A passed result returns it without a ticket; an explicit failure files a hardware ticket with test results and logs, while an aborted check restores the machine's previous state. Together says the repair process is designed to work across accelerator vendors, with vendor-specific detection and diagnostic checks. Customers can go