Chapter 1: Understanding RAG

When developing impactful applications with large language models (LLMs), the retrieval-augmented generation (RAG) technique plays a crucial role. This method allows the integration of external information that wasn't part of the LLM's training dataset into its text generation process, significantly minimizing errors and enhancing the relevance of responses.

The core concept of RAG is straightforward: retrieve the most pertinent text segment and incorporate it into the LLM's original prompt. This way, the LLM can utilize these reference texts to formulate responses. However, achieving a reliable and high-quality RAG pipeline that meets production standards can be quite challenging.

This article delves into various strategies to enhance RAG outcomes for your LLM applications, ranging from foundational concepts to advanced techniques. Real-world examples and insights from my own experience in building RAG-powered products will also be shared.

Pre-retrieval Techniques

The pre-retrieval strategies focus on optimizing the indexing step or enhancing the search for chunks in the database.

The first strategy involves improving the quality of indexed data. The adage "garbage in, garbage out" is very relevant here. Many overlook this crucial initial step, concentrating instead on subsequent optimizations. It's vital not to indiscriminately add every document to your vector database and expect favorable outcomes. Instead, enhance the quality of your indexed data by:

1. Removing irrelevant documents based on your specific use case.
2. Formatting your indexed data to reflect potential end-user queries.
3. Adding metadata to documents to facilitate efficient retrieval.

For instance, when dealing with math problems, two seemingly similar problems may test different concepts. By tagging them with relevant metadata, you can ensure that the retrieval process considers the correct context.

Another common issue occurs when segments lose critical information during splitting. For example, an article's initial sentences may introduce entities by name, while later sentences refer to them only with pronouns. If these segments are split, the resultant chunks may lose semantic meaning. Replacing pronouns with actual names can enhance the semantic relevance of these segments.

The second strategy focuses on chunk optimization. Depending on your downstream task, you should determine the optimal chunk size and necessary overlap. If chunks are too small, they may lack the information required for the LLM to answer queries accurately. Conversely, overly large chunks may introduce irrelevant details that confuse the LLM or exceed context limits.

From experience, it's unnecessary to use a singular chunk optimization method throughout the pipeline. For instance, larger chunks could be effective for summarization tasks, while smaller chunks might be more suitable for coding references.

Another helpful technique involves rephrasing the user's query to better align with the content found in your vector database. The Query2Doc technique generates pseudo-documents that expand the query, while the HyDE (Hypothetical Document Embeddings) method creates hypothetical documents relevant to the query.

These hypothetical documents can be generated as follows:

# For blog articles

prompt = f"Please create a paragraph from a blog article about {user_query}"

# For code documentation in markdown

prompt = f"Please draft a code documentation for {user_query} in markdown format."

A common pitfall when using Query2Doc or HyDE is that the generated documents may contradict actual documents or be entirely irrelevant. To mitigate this, retrieve documents both with and without hypothetical documents, allowing for post-retrieval techniques to determine the best reference text.

If a user query is complex and requires multiple reference texts, break it down into simpler sub-queries. For example, a user asking about the differences between ChromaDB and Weaviate could be rephrased into two separate questions: "What is ChromaDB?" and "What is Weaviate?".

To further enhance the query handling process, consider using query routing to direct user queries to different RAG processes based on the task at hand.

The first video, Systematically Improving RAG Applications, provides in-depth insights on optimizing RAG processes and techniques.

Retrieval Techniques

Once you have your query prepared, you can enhance retrieval results in the second stage of the RAG pipeline.

A frequently overlooked technique is the use of alternative search methods, which can either replace or complement vector similarity searches. Although vector similarity searches are generally effective, in some cases, full-text searches or structured queries may yield better results. For instance, if your dataset contains many semantically similar chunks differing only by a few keywords, precise keyword matching could be more effective.

Another important consideration is to experiment with various embeddings for your specific task. Many users stick to default embedding options without realizing that different models capture various semantic information. For example, the Instructor Embedding model allows for tailored embedding instructions based on your data and task requirements.

In addition, implementing adaptations such as small-to-big retrieval, recursive searches, or context-aware retrieval can significantly improve relevance. These techniques begin by retrieving smaller, highly specific data chunks, then expanding to include larger text blocks for context.

Hierarchical retrieval can also be beneficial. By creating a two-layer structure containing both original chunks and summaries, you can first conduct a search on the summary layer to filter out irrelevant documents, then delve deeper into the specific chunks.

The second video, Evaluating Your RAG Applications, offers strategies for assessing and enhancing the effectiveness of RAG implementations.

Post-retrieval Techniques

After retrieving relevant chunks, several strategies can further enhance generation quality. Depending on your task's nature and the text chunk format, you can utilize one or more of the following techniques:

For tasks focused on a single chunk, reranking or scoring is a commonly employed technique. A high score in a vector similarity search does not always guarantee relevance, so a secondary round of scoring can help select the most useful text segments for generating responses. You can ask the LLM to rank document relevance or use methods like keyword frequency or metadata matching to refine selections.

In contrast, if your task involves multiple chunks, such as summarization or comparison, consider compressing information as a post-processing step. This could involve summarizing, paraphrasing, or extracting key points from each chunk before passing the consolidated information to the LLM for generation.

Conclusion

In this article, I presented numerous techniques to enhance your RAG pipeline within LLM-powered applications, including:

• Basic RAG process: indexing, retrieval, and generation.
• Pre-retrieval techniques: improving indexed data quality, chunk optimization, and query rewriting.
• Retrieval techniques: utilizing alternative search methods, experimenting with different embeddings, and implementing hierarchical retrieval.
• Post-retrieval techniques: reranking and scoring retrieved chunks, and information compression.
• Strategies for balancing quality and latency: using smaller models, parallel processing, making choices instead of generating text, and caching.

You can apply one or more of these techniques to improve the accuracy and efficiency of your RAG pipeline. I hope these insights assist you in creating a more effective RAG framework for your applications.