KDnuggets published a September 9, 2026 technical guide describing 7 ways machine learning teams can train LLMs on limited hardware such as dual or quad workstation GPUs, including RTX 4090s, A10Gs, and L40Ss. The guide says full pre-training or full fine-tuning of multi-billion-parameter foundation models normally requires clusters of H100s linked by 3.2 Tbps InfiniBand, while smaller teams often face PCIe bandwidth limits and 24 GB to 48 GB of VRAM per device.

The guide says a naive 16-bit setup with AdamW can fail before step 1 on a 7B parameter model. Static weights alone use 14 GB of VRAM in FP16/BF16, AdamW first and second moment states require 8 bytes per parameter in FP32, or about 56 GB for a 7B model, and backward gradients add another 14 GB before activation memory grows with context length. Engineers are told to distinguish static memory overhead from dynamic transient memory overhead, and to identify whether training is compute-bound or memory bandwidth-bound.

For fine-tuning 7B to 70B parameter models on single or dual 24 GB GPUs, the guide recommends QLoRA and DoRA. QLoRA freezes base weights in 4-bit NormalFloat, uses Double Quantization to save another 0.37 bits per parameter, dequantizes weights into BF16 during the forward pass, and trains low-rank update matrices. DoRA separates magnitude and direction updates, but the guide says on-the-fly dequantization can reduce training throughput by 20% to 35% versus native 16-bit training.

For full-parameter learning under memory limits, the guide describes GaLore, which projects high-dimensional gradients into a low-rank subspace (smaller representation of updates) and stores optimizer states only for projected matrices. It says standard AdamW stores two FP32 states per trainable parameter, while GaLore updates projections every T steps to reduce SVD overhead. The risk is step-latency spikes and instability if the update frequency T or rank cutoff r is poorly chosen.

For models larger than total GPU memory, the guide lists FSDP and ZeRO-Stage 3 host offloading. Each GPU holds only 1/N of the model state during idle periods, All-Gather communication reconstructs layer weights before computation, and CPU RAM can store inactive shards. The tradeoff is severe PCIe Gen4/Gen5 I/O bottlenecks; if host-to-device transfers lag behind compute, GPU utilization can drop below 30%, and NVMe dataloading can be starved.

For long context windows, the guide recommends selective activation checkpointing, which drops memory-heavy activations and recomputes them during backpropagation. It targets operations such as GeLU/SwiGLU activations, layer norms, and dropout masks, usually around transformer block boundaries. The guide says full activation recomputation adds roughly 30% computational overhead per training step, and poor allocation profiling can cause CUDA out-of-memory errors even when gross VRAM appears below limits.

The guide also names FlashAttention-2, fused operations, FP8 mixed precision, and RingAttention. FlashAttention-2 avoids writing the full N x N attention matrix to HBM by tiling Query, Key, and Value matrices into L1 cache/SRAM blocks, while fused kernels combine operations to reduce memory traffic. FP8 uses E4M3 with 1 sign bit, 4 exponent bits, and 3 mantissa bits for activations and weights, and E5M2 with 1 sign bit, 5 exponent bits, and 2 mantissa bits for gradients; on Ada Lovelace, RTX 4090, L40S, or Hopper H100 hardware, FP8 Tensor Cores can double compute throughput and halve activation VRAM. RingAttention splits 64k+ context sequences across K devices and passes KV blocks in a ring, but PCIe or 1GbE/10GbE latency can erase gains when transfer time exceeds compute time.

KDnuggets published a September 9, 2026 technical guide describing 7 ways machine learning teams can train LLMs on limited hardware such as dual or quad workstation GPUs, including RTX 4090s, A10Gs, and L40Ss. The guide says full pre-training or full fine-tuning of multi-billion-parameter foundation models normally requires clusters of H100s linked by 3.2 Tbps InfiniBand, while smaller teams often face PCIe bandwidth limits and 24 GB to 48 GB of VRAM per device.

The guide says a naive 16-bit setup with AdamW can fail before step 1 on a 7B parameter model. Static weights alone use 14 GB of VRAM in FP16/BF16, AdamW first and second moment states require 8 bytes per parameter in FP32, or about 56 GB for a 7B model, and backward gradients add another 14 GB before activation memory grows with context length. Engineers are told to distinguish static memory overhead from dynamic transient memory overhead, and to identify whether training is compute-bound or memory bandwidth-bound.

For fine-tuning 7B to 70B parameter models on single or dual 24 GB GPUs, the guide recommends QLoRA and DoRA. QLoRA freezes base weights in 4-bit NormalFloat, uses Double Quantization to save another 0.37 bits per parameter, dequantizes weights into BF16 during the forward pass, and trains low-rank update matrices. DoRA separates magnitude and direction updates, but the guide says on-the-fly dequantization can reduce training throughput by 20% to 35% versus native 16-bit training.

For full-parameter learning under memory limits, the guide describes GaLore, which projects high-dimensional gradients into a low-rank subspace (smaller representation of updates) and stores optimizer states only for projected matrices. It says standard AdamW stores two FP32 states per trainable parameter, while GaLore updates projections every T steps to reduce SVD overhead. The risk is step-latency spikes and instability if the update frequency T or rank cutoff r is poorly chosen.

For models larger than total GPU memory, the guide lists FSDP and ZeRO-Stage 3 host offloading. Each GPU holds only 1/N of the model state during idle periods, All-Gather communication reconstructs layer weights before computation, and CPU RAM can store inactive shards. The tradeoff is severe PCIe Gen4/Gen5 I/O bottlenecks; if host-to-device transfers lag behind compute, GPU utilization can drop below 30%, and NVMe dataloading can be starved.

For long context windows, the guide recommends selective activation checkpointing, which drops memory-heavy activations and recomputes them during backpropagation. It targets operations such as GeLU/SwiGLU activations, layer norms, and dropout masks, usually around transformer block boundaries. The guide says full activation recomputation adds roughly 30% computational overhead per training step, and poor allocation profiling can cause CUDA out-of-memory errors even when gross VRAM appears below limits.

The guide also names FlashAttention-2, fused operations, FP8 mixed precision, and RingAttention. FlashAttention-2 avoids writing the full N x N attention matrix to HBM by tiling Query, Key, and Value matrices into L1 cache/SRAM blocks, while fused kernels combine operations to reduce memory traffic. FP8 uses E4M3 with 1 sign bit, 4 exponent bits, and 3 mantissa bits for activations and weights, and E5M2 with 1 sign bit, 5 exponent bits, and 2 mantissa bits for gradients; on Ada Lovelace, RTX 4090, L40S, or Hopper H100 hardware, FP8 Tensor Cores can double compute throughput and halve activation VRAM. RingAttention splits 64k+ context sequences across K devices and passes KV blocks in a ring, but PCIe or 1GbE/10GbE latency can erase gains when transfer time exceeds compute time.