New AA-WER Streaming Benchmark Evaluates Voice AI Performance

Wednesday, June 3, 2026
  • •Artificial Analysis released the AA-WER Streaming benchmark measuring STT model latency and transcription accuracy.
  • •Cartesia Ink-2 and ElevenLabs Scribe v2 Realtime demonstrate leading performance on Final and Partial transcription benchmarks.
  • •DeepgramAI Flux achieved the fastest recorded transcription times, reaching 0.019s latency for partial event returns.

Artificial Analysis launched the AA-WER Streaming benchmark on June 2, 2026, to evaluate streaming Speech to Text (STT) models based on accuracy and latency. The benchmark tests models across 8 hours of audio using three datasets: AA-AgentTalk, Earnings22-Cleaned-AA, and VoxPopuli-Cleaned-AA. Measurements focus on two metrics: First Final Transcription, tracking the Word Error Rate (WER) and latency of the first final-denoted transcript returned; and First Partial Transcription, tracking the first transcript-bearing event returned after speech detection.

In the Final accuracy category, Cartesia Ink-2 (semantic endpoints) achieved a 3.59% WER at 0.21s latency, followed by ElevenLabs Scribe v2 Realtime (3.64%, 0.14s). For First Partial accuracy, ElevenLabs Scribe v2 Realtime led with 3.65% WER at 0.13s latency. DeepgramAI Flux proved the fastest overall, recording 0.020s latency on final outputs and 0.019s on partials, both at 7.36% WER.

Performance varies by dataset, with no single model leading across all metrics. For example, ElevenLabs Scribe v2 Realtime leads on the AA-AgentTalk set, while AssemblyAI U3 Realtime Pro leads on VoxPopuli final transcriptions and Google Chirp 3 Streaming on partials. On average, Final transcription accuracy is approximately 0.7pp higher than First Partial accuracy, though most models reach peak accuracy early. Notable exceptions include Voxtral Mini Transcribe Realtime, which improved from 9.6% to 5.3% WER when waiting for final output. Pricing for tested models ranges from $2 to $17 per 1,000 minutes, with Cartesia and ElevenLabs offering competitive performance-to-price ratios.

Subscribe to the AIB Newsletter

The fastest, most reliable AI news, straight to your inbox

Blog posts related to this article