Baseten Speeds Qwen3-ASR Batch Transcription

Baseten says it optimized Alibaba’s open-weight Qwen3-ASR 1.7B for batch transcription, achieving up to 7x more audio per second than unmodified vLLM on a single RTX PRO 6000 at 64 concurrent requests. For hour-long files, the service sustains 3,128x real-time speed and costs $0.0013 per audio hour, with no meaningful change in word error rate. The optimized deployment is the default in Baseten’s Model Library and can be called through an OpenAI-compatible transcription client.

Qwen3-ASR 1.7B supports more than 30 languages and 22 Chinese dialects. Baseten reports a 5.4% average word error rate on Hugging Face’s Open ASR Leaderboard, placing it on the Pareto frontier (models offering the best trade-offs between accuracy and size). Its 5.40% score is 0.07 percentage points above NVIDIA Nemotron 3.5 Transcribe 30B-A4B’s 5.33%, while the Qwen model has a fraction of that model’s size.

Baseten replaced vLLM’s Python-based audio decoding with torchcodec, which sends files to FFmpeg for decoding and resampling in one native call. On an hour-long file, preprocessing takes well under a second. For one-minute files, the original decoder took 76 milliseconds alone and 2,195 milliseconds when decoding 8 files at once; the replacement took 45 milliseconds alone and 48 milliseconds with 8 files. Parallel processing across time ranges also lets long recordings be joined with bit-exact correctness.

For high-concurrency processing, Baseten stores the attention KV cache in FP8, halving the data read per step and allowing twice as many sequences in GPU memory. Because vLLM’s default FlashAttention-2 backend did not support an FP8 cache on the RTX PRO 6000, Baseten switched to FlashInfer kernels; memory utilization for the attention kernel rose from 50% to 80%. Model weights remain in BF16 because weight quantization raised word error rates in Baseten’s tests.

Baseten also kept CPU copies of sequence lengths and clip-length values to avoid waiting for the GPU to send them back before preparing work. Under load, GPU utilization rose from 86% to 97%. Four API-server processes and twice the preprocessing thread pool help decoding and feature extraction keep pace with the GPU at 64 concurrent requests.

In Baseten’s tests, throughput for 10-second clips rose from 149x to 602x real time, a 4.0x gain; for 30-second clips, from 346x to 1,358x, or 3.9x; for 1-minute clips, from 358x to 1,861x, or 5.2x; and for 5-minute clips, from 380x to 2,645x, or 7.0x. Unmodified vLLM could not handle 30-minute or 1-hour clips in one request; the optimized stack reached 2,688x for 30-minute files and 3,128x for hour-long files, both at 16 concurrent requests.

For 5-minute clips, latency fell from 5.1 seconds to 1.5 seconds for a single request, and from 50.4 seconds to 7.3 seconds at 64 concurrent requests, a 6.9x reduction at the higher load. Baseten estimates its deployment costs $0.0013 per audio hour, compared with $0.011 for the same model on unmodified vLLM and $0.21–$0.36 for popular closed-weight providers, making its estimated cost around 277x lower. The estimate uses Baseten’s list price for a dedicated RTX PRO 6000 and published pay-as-you-go prices for providers’ flagship batch models.

Baseten says it optimized Alibaba’s open-weight Qwen3-ASR 1.7B for batch transcription, achieving up to 7x more audio per second than unmodified vLLM on a single RTX PRO 6000 at 64 concurrent requests. For hour-long files, the service sustains 3,128x real-time speed and costs $0.0013 per audio hour, with no meaningful change in word error rate. The optimized deployment is the default in Baseten’s Model Library and can be called through an OpenAI-compatible transcription client.

Qwen3-ASR 1.7B supports more than 30 languages and 22 Chinese dialects. Baseten reports a 5.4% average word error rate on Hugging Face’s Open ASR Leaderboard, placing it on the Pareto frontier (models offering the best trade-offs between accuracy and size). Its 5.40% score is 0.07 percentage points above NVIDIA Nemotron 3.5 Transcribe 30B-A4B’s 5.33%, while the Qwen model has a fraction of that model’s size.

Baseten replaced vLLM’s Python-based audio decoding with torchcodec, which sends files to FFmpeg for decoding and resampling in one native call. On an hour-long file, preprocessing takes well under a second. For one-minute files, the original decoder took 76 milliseconds alone and 2,195 milliseconds when decoding 8 files at once; the replacement took 45 milliseconds alone and 48 milliseconds with 8 files. Parallel processing across time ranges also lets long recordings be joined with bit-exact correctness.

For high-concurrency processing, Baseten stores the attention KV cache in FP8, halving the data read per step and allowing twice as many sequences in GPU memory. Because vLLM’s default FlashAttention-2 backend did not support an FP8 cache on the RTX PRO 6000, Baseten switched to FlashInfer kernels; memory utilization for the attention kernel rose from 50% to 80%. Model weights remain in BF16 because weight quantization raised word error rates in Baseten’s tests.

Baseten also kept CPU copies of sequence lengths and clip-length values to avoid waiting for the GPU to send them back before preparing work. Under load, GPU utilization rose from 86% to 97%. Four API-server processes and twice the preprocessing thread pool help decoding and feature extraction keep pace with the GPU at 64 concurrent requests.

In Baseten’s tests, throughput for 10-second clips rose from 149x to 602x real time, a 4.0x gain; for 30-second clips, from 346x to 1,358x, or 3.9x; for 1-minute clips, from 358x to 1,861x, or 5.2x; and for 5-minute clips, from 380x to 2,645x, or 7.0x. Unmodified vLLM could not handle 30-minute or 1-hour clips in one request; the optimized stack reached 2,688x for 30-minute files and 3,128x for hour-long files, both at 16 concurrent requests.

For 5-minute clips, latency fell from 5.1 seconds to 1.5 seconds for a single request, and from 50.4 seconds to 7.3 seconds at 64 concurrent requests, a 6.9x reduction at the higher load. Baseten estimates its deployment costs $0.0013 per audio hour, compared with $0.011 for the same model on unmodified vLLM and $0.21–$0.36 for popular closed-weight providers, making its estimated cost around 277x lower. The estimate uses Baseten’s list price for a dedicated RTX PRO 6000 and published pay-as-you-go prices for providers’ flagship batch models.