Anna Villarreal successfully deployed Ollama on a Jetson Orin Nano by building the software from source to optimize resource management. Building from source took approximately 30 minutes and allowed for explicit configuration of GPU acceleration, which proved necessary as Docker containers were too resource-intensive for the device's 8GB of RAM. The process involved installing the Cuda-toolkit-13 and Go, followed by configuring paths to ensure the system utilized the GPU, specifically targeting the sm_87 architecture.

During initial testing, the application suffered from severe performance issues, with generation speeds of only 1.2 to 1.35 tokens per second when using the Q8_0 model quantization. This configuration only loaded 3–9 of the 17 available GPU layers, forcing the CPU to handle the remainder. By switching to the Q4_K_M quantization format, the model size decreased from 1.5 GB to 808 MB. This allowed 17 of 17 GPU layers to load entirely into memory, resulting in a generation speed of approximately 30.7 tokens per second. This shift enabled a 971-token test to complete in 35 to 45 seconds, compared to the 13 minutes and 20 seconds required by the Q8_0 version, representing a roughly 25x speedup.

The performance gains of the Q4_K_M quantization introduced reliability concerns, specifically regarding malformed output. Testing revealed a failure rate of 50–65% for valid JSON generation, with 3 out of 6 attempts producing malformed brackets or incorrect field names. To resolve this without reverting to the slower Q8_0 model, the app was updated with automatic retry logic. The system now attempts to generate a response up to 3 times if the initial attempt is malformed. Given the high speed of the Q4 quantization, the cumulative time for 3 attempts remains significantly faster than a single Q8_0 request, effectively raising the success rate for structured outputs to 85–95%.

Anna Villarreal successfully deployed Ollama on a Jetson Orin Nano by building the software from source to optimize resource management. Building from source took approximately 30 minutes and allowed for explicit configuration of GPU acceleration, which proved necessary as Docker containers were too resource-intensive for the device's 8GB of RAM. The process involved installing the Cuda-toolkit-13 and Go, followed by configuring paths to ensure the system utilized the GPU, specifically targeting the sm_87 architecture.

During initial testing, the application suffered from severe performance issues, with generation speeds of only 1.2 to 1.35 tokens per second when using the Q8_0 model quantization. This configuration only loaded 3–9 of the 17 available GPU layers, forcing the CPU to handle the remainder. By switching to the Q4_K_M quantization format, the model size decreased from 1.5 GB to 808 MB. This allowed 17 of 17 GPU layers to load entirely into memory, resulting in a generation speed of approximately 30.7 tokens per second. This shift enabled a 971-token test to complete in 35 to 45 seconds, compared to the 13 minutes and 20 seconds required by the Q8_0 version, representing a roughly 25x speedup.

The performance gains of the Q4_K_M quantization introduced reliability concerns, specifically regarding malformed output. Testing revealed a failure rate of 50–65% for valid JSON generation, with 3 out of 6 attempts producing malformed brackets or incorrect field names. To resolve this without reverting to the slower Q8_0 model, the app was updated with automatic retry logic. The system now attempts to generate a response up to 3 times if the initial attempt is malformed. Given the high speed of the Q4 quantization, the cumulative time for 3 attempts remains significantly faster than a single Q8_0 request, effectively raising the success rate for structured outputs to 85–95%.