You can improve RAG retrieval without retraining by combining multiple chunk sizes and weighing them based on uncertainty—fine-grained chunks find relevant content, larger chunks provide context.
This paper proposes UMG-RAG, a training-free retrieval system that combines dense and sparse retrievers across multiple chunk sizes to improve RAG quality. Instead of choosing one chunk size, it intelligently fuses results from different granularities based on confidence estimates, and optionally returns larger parent chunks for better context while using fine-grained chunks for precise retrieval.