Topic 06 / 8

Advanced RAG Techniques

~15 min read // LLM Engineering Series // Coding India

The Limits of Naive RAG

The basic RAG pipeline (embed document → search vector database → feed to LLM) works well for simple prototypes but quickly fails in production. It might retrieve irrelevant context or miss crucial details spread across multiple documents.

Semantic Chunking

If you split a 100-page PDF arbitrarily every 500 words, you might cut a sentence in half, destroying its meaning. Semantic chunking involves splitting documents by logical boundaries—paragraphs, markdown headers, or even using a smaller NLP model to group sentences that discuss the same topic.

Hybrid Search

Vector search (Dense search) is amazing at understanding meaning. If you search for “dog”, it will find “puppy”. But it’s terrible at exact keyword matching. If you search for “Error Code 404B”, it might struggle. Hybrid Search combines traditional keyword search (like BM25) with vector search to give you the best of both worlds.

Re-Ranking

Vector databases are fast but their distance metrics (like cosine similarity) are relatively crude. A common advanced pattern is to retrieve the top 20 results from the vector database, and then pass them through a specialized Cross-Encoder Re-ranker (like Cohere’s Rerank API). This model is much slower but highly accurate, re-sorting the 20 results so the absolute best context is sent to the LLM.