Understanding retrieval-augmented generation with OpenAI

Models like GPT-4 are remarkable at generating human-like responses. There's a catch: they are limited by the data they were trained on, and they have no real-time access to anything else. That's where retrieval-augmented generation comes in. It extends a language model by folding external knowledge retrieval into the generation step.

What RAG actually is

RAG combines a language model with information retrieval:

• Retrieval: instead of relying only on the model's pre-trained knowledge, query external sources (databases, documents, web APIs) for information relevant to the user's input.
• Generation: pass the retrieved information to the model, which uses the extra context to produce a more accurate answer.

This matters whenever the model needs up-to-date or domain-specific information that wasn't available at training time.

Why bother

1. Overcome training limitations. GPT models know the world up to a cutoff. With RAG you query the latest information from databases, APIs or indexed documents.
2. More accurate, more relevant. With retrieval in place the model works from current and precise data, which matters most in knowledge-heavy domains.
3. Scalability. Rather than fine-tuning a model for every new knowledge domain, the model retrieves dynamically, which scales across applications.

How it works

RAG systems typically run in two phases:

1. Retrieval. The user's query pulls relevant documents or snippets from a knowledge base or document store.
2. Generation. Those documents become additional input to the model, which generates a contextually richer response.

A simple example

Two pieces: retrieve documents from an external source based on a query, then use them to generate an informed response.

1. The retrieval mechanism

A mock database of articles, searched for the ones relevant to the input:

# Sample article database (could be a real database or indexed documents)
articles = {
    1: "Venom: Last Dance, released in October 2024, is the final installment of the Venom trilogy.",
    2: "Gather AI, a Slack bot developed by Codemancers, introduces a new feature for creating mindmaps.",
    3: "Next.js 15 introduces the @next/codemod CLI for easily upgrading to the latest Next.js and React versions."
}
 
# Function to simulate document retrieval
def retrieve_documents(query):
    query = query.lower().split()
 
    relevant_docs = []
 
    # In real-world, you'd use more advanced search, Here, we are just matching substrings for simplicity
    for id, content in articles.items():
        content_lower = content.lower()
        if any(word in content_lower for word in query):
            relevant_docs.append(content)
 
    return relevant_docs

2. Generate a response from the retrieved data

from openai import OpenAI
 
client = OpenAI(
    api_key = "your-open-ai-key"
)
 
def generate_response(query, documents):
    context = "\n\n".join(documents)
     # Combine retrieved documents with the original query
    prompt = f"Answer the following question based on the context provided:\n\n{context}\n\nQuestion: {query}"
 
    messages = [
        {"role": "user", "content": prompt}
    ]
 
 
    response = client.chat.completions.create(
        model="gpt-4",
        messages=messages
    )
 
    return response.choices[0].message.content
 
 
user_query = "Whats new with Next.js 15?"
retrieved_docs = retrieve_documents(user_query)
 
if retrieved_docs:
    response = generate_response(user_query, retrieved_docs)
    print(response)
else:
    print("No relevant documents found.")
 
# Response : Next.js 15 introduces the @next/codemod CLI for easily upgrading to the latest Next.js and React versions.

What happened:

1. retrieve_documents() searches a small set of articles and returns those relevant to the query.
2. The retrieved documents go to GPT-4 as part of the prompt, and the model answers with that context in hand.

For the query "Whats new with Next.js 15?" the system retrieves the Next.js document and passes it along. The response is more detailed and more accurate than what the model would produce unaided.

Where to take it next

This example is deliberately small. Ways to grow it:

1. Better retrieval. Use FAISS for embedding search or Elasticsearch to search large document stores efficiently.
2. Real-time data. Integrate live APIs or databases (Wolfram Alpha, news APIs, product catalogues) to retrieve the latest information.
3. Better integration. In production, something like LangChain manages retrievals and the query-to-retrieval workflow for you. That is exactly what the follow-up post does.

Conclusion

RAG combines document retrieval with text generation to produce answers that are accurate and contextually grounded. Chatbots, content generators, question-answering systems: anywhere the model needs to know something it was never trained on, this is the pattern.

Resources

• OpenAI: optimizing LLM accuracy with RAG
• Red Hat: introduction to RAG