You can get both fast batched inference and accurate context-aware transcription by carefully managing segment boundaries—this matters for production speech systems that need both speed and quality.
This paper improves speech transcription by combining the speed of batched processing with the accuracy benefits of maintaining context. The key innovation is using voice activity detection to safely batch audio segments while keeping historical context, which helps Whisper produce better punctuation and proper nouns without sacrificing inference speed.