Lecture on Contextual Retrieval

Introduction

• Anthropic has released a new retrieval mechanism called contextual retrieval.
• Combines with re-ranking for state-of-the-art performance.
• Considered more of a chunking strategy than a novel retrieval technique.

Traditional Retrieval Mechanism

• Involves chunking documents into sub-documents and computing embeddings.
• Embeddings stored in a vector store.
• User queries generate embeddings for retrieving relevant chunks.
• Common failures in semantic search lead to combining with keyword-based search like BM25.

Limitations of Traditional Systems

• Loss of contextual information when returning chunks.
• Example: Financial information and revenue growth queries lacking context.

Introduction to Contextual Retrieval

• Anthropic suggests using contextual retrieval for better results.
• Include contextual information in each chunk.
• Use LLMs like Haiku to automate context addition.

Implementation Steps:

1. Provide the whole document as input.
2. Chunk the document.
3. Utilize Haiku to add contextual information to chunks.
4. Store embeddings in a vector database and update BM25 index.

Cost and Performance

• Cost of adding context is minimal ($1.02 per million document tokens).
• Significant improvement in retrieval failure rates:
  • Contextual embedding reduces failure from 5.7% to 3.7%.
  • Combined with contextual BM25, failure rate reduces to 2.9%.

Recommendations

• Use keyword-based search, query rewriter, and re-ranker.
• Chunking strategy and embedding models are application-dependent.
• Consider dense embeddings or Colbert-based multi-vector representations.

Performance Improvements

• Contextual retrieval reduces retrieval error rates.
• Re-ranking further improves retrieval accuracy.

Code Example

• Anthropic’s repo provides implementation examples.
• Comparison between basic RAG and contextual retrieval shows performance gains.

Conclusion

• Contextual retrieval shows promise in improving RAG systems.
• Importance of integrating contextual retrieval in applications.
• Encourage further exploration and learning about RAG and LLM agents.