12 minute read

RAG Systems vs. LCW: Performance and Cost Trade-offs

Published on Sep 3, 2024

Guruprasad Raghavan
Lead Research Scientist and Founder

Legion’s AI summary

Discover the performance and cost trade-offs between Retrieval-Augmented Generation (RAG) systems and Large Context Window (LCW) models:

  • Biographies benchmark: RAG systems outperform LCW models in retrieving needles from haystacks up to 2M tokens.
  • Multi-needle retrieval: Legion RAG system excels at finding multiple needles regardless of their position in the haystack.
  • Compute requirements: RAG systems are significantly more cost-effective, requiring fewer GPU resources compared to LCW models.

Large language models (LLMs) with extremely long context windows (LCW) have become a focal point of discussion, particularly for enterprise adoption of GenAI. Long context window LLMs such as Gemini, GPT-4o, Llama-1-405B, and Mixtral have further fueled this interest.

With the emergence of LCW models, came the development of academic benchmarks to quantify the effectiveness of these large systems. The popular benchmarks designed for LCW evaluation are as follows:

  1. Needle in a Haystack (and its variants)
  2. Infinity Bench
  3. ZeroSCROLLS

In this article, we conduct a comparative analysis between LCW models, utilizing in-context learning, and retrieval-augmented generation (RAG) systems paired with smaller context window models. This comparison spans both ends of the AI architectural size spectrum and is applied to two variants of the Needle in a Haystack (NIAH) benchmark.

Our major contributions in this study are twofold:

  1. We demonstrate that RAG systems paired with small context window models are more performant and cost effective than Long context window (LCW) models for use-cases that involve retrieval and analysis of specific information from a very large corpus of documents.
  2. Academic benchmarks (like Needle in a haystack) aren't well suited for demonstrating the unique abilities of LCW models in Enterprise AI scenarios. This necessitates the push to develop better benchmarks that can uniquely highlight abilities of LCW models that aren’t available in smaller systems. Additionally, the NIAH assays are not in touch with the reality of enterprise information as most haystacks are not neatly formatted text, but are instead trapped within documents like pdfs, ppts, and other document formats.

What is the Needle in a Haystack benchmark?

The Needle in a Haystack benchmark, as the name suggests, is a synthetic document wherein a “needle” (i.e. a relevant piece of information) is hidden within a large “haystack” (i.e. large corpus of text); And models (or systems of models) are evaluated with a query that aims to retrieve that particular needle.

The NIAH benchmark is highly relevant for Enterprise AI use cases that involve accessing and analyzing specific pieces of information from a large corpus of text and documents within enterprises.

We evaluate the two systems (RAG and LCW models) on two variants of the needle-in-a-haystack benchmark, namely:

  1. Biographies (NIAH) benchmark: Here, a single needle (that is very closely related in content to the haystack) is inserted into the haystack at an arbitrary position. Using a test set of 140 biographical questions for every haystack size, we compare and contrast different systems, using haystacks ranging from 2k to 2M tokens.

  2. Multi-needle in a haystack: Here, three needles are inserted into the haystack at different positions within the large corpus of text; and the model (or system of models) is evaluated on its ability to retrieve all the needles from the haystack given a relevant query.

Results

Performance:

Biographies (NIAH) benchmark

The biographies benchmark expects the model (or system of models) to retrieve the needle (i.e. natural language text pertaining to a particular individual) from the haystack and perform reasoning on it in order to report the following pieces of information, when available:

  1. Date of birth: in the format of day, month, and year
  2. Date of death: in the same format as above
  3. Nationality: as a string. If not provided in the description, it should be reported as “unknown.”
  4. Politician: as a boolean. To infer if the individual is a politician based on their description.
  5. Sportsperson: as a boolean. To infer if the individual is a sportsperson based on their description.

The RAG systems evaluated are:

  1. Legion RAG (end-to-end) system coupled with open-source Llama-3-8B-instruct, which has a maximum context window of 8k tokens.
  2. Contextual AI’s RAG 2.0.

On the other hand, the LCW model evaluated is GPT-4 that has a maximum context window of 32k tokens.

As demonstrated in Figure 2, we find that RAG systems coupled with smaller context window models (blue and red curves) are significantly more performant than the LCW models (yellow curve). Additionally, it is noteworthy that the performance of RAG systems are almost constant as the size of the haystack is varied from 2k to 2M tokens, while the LCW models demonstrate a sharp drop in accuracy as the haystack size increases.

GPU compute requirements / compute costs

Having demonstrated that the raw accuracy of using smaller RAG systems coupled with smaller context window models is more capable than in-context learning using very long context window models, we want to highlight the compute requirements and costs associated with each of the systems.

Legion RAG (end-to-end) combined with Llama-3-8B-instruct requires up to 2 A10 GPUs for single-user operations and can scale to support 50 concurrent users with 4 A10 GPUs. While the RAG approach involves additional retrieval components, we emphasize GPU requirements as they continue to be the primary cost driver.

On the other hand, long context window models can require a minimum of 40 A10 GPUs for their inference (for a single user).

Conclusions and future work:

Conclusions:

We have demonstrated that RAG systems are way more performant than LCW models on popular academic benchmarks developed to test the effectiveness of long context window models. Moreover, RAG systems can easily scale to large document corpora without any degradation in performance or accuracy, making them a perfect candidate for enterprise adoption of Generative AI.