Increasing RAG Accuracy with Multi-Level Chunking

Notes from Stop Chunking like it’s 2022

The argument: how well a query gets answered depends on what chunk size you picked.

• Highly specific queries might want smaller chunks.
• While some queries might like the wider context.

This is because of the fixed size embedding where you either lose fine-grained details or semantic context.

The oracle experiment

They back this claim with an oracle experiment, comparing:

• Retrievers with a single fixed chunk size.
• An oracle retriever which always knows the best chunk size per question
  • probably the best fixed retriever per question.

They find there is room for improvement if we just know optimal chunk size for each query.

• ~1-3% on MTEB
• up to 37% on one of the datasets

Multi-level chunking

They suggest we do multi-level chunking (by taking multiple chunk sizes)
At query time, we do parallel searches across all the levels and then combine the results using RRF at a document level. (Or a larger chunk level)

Cost

Storage and Compute increase by the number of different sizes we store.

Other Approaches

Other approaches tackling similar problems include Contextual RAG, Late Chunking and RAPTOR, which tend to add a lot of complexity