Legion RAG: Performance That Doesn’t Break the Bank

Published on Oct 30, 2024

Legion’s AI summary

You'll learn how Legion's Retrieval Augmented Generation (RAG) system delivers high performance at a fraction of the cost.

  • Dynamic chunking and embedding: Legion's novel approach ensures superior retrieval while keeping costs low
  • Contextual Retrieval comparison: See how Legion's RAG system stacks up against Anthropic's technique on the Codebases dataset
  • Detailed cost analysis: Explore the significant cost savings of Legion's ingestion method compared to Contextual Retrieval

Read on to uncover the power and efficiency of Legion's RAG platform for enterprise AI assistants.

At Legion, we are in the trade of building helpful AI Agents for enterprises that (a) generate answers grounded on THEIR massive proprietary knowledge base, (b) scale to a large number of end-knowledge workers in the enterprise; and (c) respect and mirror access controls previously established in the enterprise. We accomplish this using a multi-faceted approach, which involves building a powerful and robust Retrieval Augmented Generation (RAG) system to support natural language queries.

RAG is a powerful technique which enables Large Language Models (LLMs) to answer questions about a user’s private knowledge base without actually having to train these models on private data. In a typical RAG system, a user's knowledge base is broken down (chunked) into smaller text pieces and each piece is embedded into a vector space using an embedding model. These embeddings capture the semantic information present in the chunks and enable a direct comparison against a natural language query (by embedding this query into the same vector space) to enable retrieving the most relevant chunks for a given query.

One of the major challenges with chunking and embedding is that chunks embedded independent of each other can lack appropriate context. For example, let’s consider a knowledge base that comprises the following text: “The Golden Gate Bridge is a historical landmark in the city of San Francisco. Its red color makes it easily recognizable from a distance”. If our chunking technique ends up separating the two sentences, it would be impossible to disambiguate the “It” in the second sentence without additional context. Many techniques have been developed to address these kinds of shortcomings, like generic document summaries to chunks, hypothetical document embedding and summary-based indexing.

Recently, Anthropic introduced a new technique called Contextual Retrieval to better address this issue. In contextual retrieval, every chunk is enhanced by a concise chunk specific context that explains the chunk using the overall document.

In this post, we make the following contributions:

  • We evaluate and compare Legion’s chunking and embedding strategy against Anthropic’s Contextual Retrieval technique on the Codebases dataset released by Anthropic; and show that for the recall@5 metric, both systems deliver comparable performance (~93%).
  • Additionally, we share a detailed cost analysis comparison between the two methods and demonstrate that the Legion ingestion method is 1/300th the cost of the Contextual retrieval method.
  • And finally, we also highlight many nuances that one must contend with when using new benchmarking datasets to evaluate a RAG platform.

Legion’s RAG and Contextual Retrieval: Examining the Approaches

Details on the Codebases dataset

The Codebases dataset is divided into three different components:

  1. Documents to be ingested: There are 90 documents which are essentially files containing code (in languages like Python, Rust, C++) pulled from different repositories.
  2. User queries: There are 248 queries that a user might ask against these documents.
  3. Golden chunks: A set of chunks called the golden chunks are provided for each query. These are essentially the parts of the code files that they deem are necessary to answer a particular query.

More statistics concerning the Codebases dataset can be found in the appendix.

Evaluating Legion

For evaluating Legion’s RAG platform on Codebases dataset we ingest the 90 documents present in the dataset. The platform then uses its custom chunking and embedding technique to store the documents in a manner that enables superior retrieval performance. Following this, we issue the platform with the 248 queries present in the dataset and collect top-5 chunks per query that our platform deems are most relevant in answering those queries.

To remediate this we take a two-step approach. First, we evaluate an edit-distance between the top-5 chunks retrieved by our platform with the corresponding golden chunks. Second, we use a human-in-the-loop evaluation.

Our Results: On running our two step evaluation on the Codebases benchmark we found the following:

  1. Edit-distance based evaluation: On comparing our top-5 retrieved chunks with golden chunks using edit-distance we get a score of 55%.
  2. Human-in-the-loop evaluation: For the queries for which our system didn’t do well in the first step of our evaluation process we run a human evaluation.
    • Golden chunks are not truly golden: Human evaluators identified that close to 7% of the queries in the dataset have inappropriate golden chunks associated with them.
    • Quality of queries: Human evaluators noted that close to 5% of the queries present in the dataset are of very poor quality.
    • Golden chunks contain irrelevant information: Close to 25% of the queries have associated golden chunks that contain a lot of irrelevant information.

After aggregating the results from the above evaluations, we report performance metrics and insights from our findings.

Ingestion cost comparison: Legion RAG and Contextual Retrieval

In this section, we evaluate the two techniques for their cost to the end customer, comparing various costs associated with model deployment, API calls, context generation, and running embedding models.

For a medium-sized enterprise (100-1k employees),Legion’s ingestion and dynamic chunking methodology would cost ~360$ while using an approach akin to Contextual Retrieval would require ~$109k! Almost 300x more!

Takeaways

In this post we compare Legion’s dynamic chunking and embedding strategy to Anthropic’s new Contextual Retrieval technique. Key takeaways include:

  1. Legion’s system performs on par with this new technique on a benchmarking dataset.
  2. Many nuances, like aligning chunks, sifting generic questions, and removing irrelevant information need to be kept in mind when working with “golden” chunks.
  3. Techniques like Contextual Retrieval might improve performance but do so by increasing cost prohibitively.
  4. Legion’s dynamic chunking and embedding strategy is at par with SOTA while keeping cost per ingestion low at ~$3.50.

Appendix

Codebases dataset Statistics

Statistical significance of the performance scores is provided via a two-tailed P value, demonstrating no statistically significant difference between the performance of the Legion RAG and the Contextual Retrieval systems.

Details of our cost analysis

The analysis covers both models, focusing on API calls, model deployment, and other associated costs, summarizing the financial implications for different sizes of enterprises.