The Core Illusion: Simplicity vs. Reality

The promise of "chat with your documents" is deceptively simple. Behind that user-friendly interface lies a complex orchestration of systems. A production-grade Retrieval-Augmented Generation (RAG) application isn't a single tool; it's a sophisticated pipeline where multiple specialized components work in concert. This is the anatomy of such a system, built from the ground up and entirely self-hosted.

The project, dubbed myRAG, demonstrates a full RAG stack. It features a FastAPI backend, a React frontend, and crucially, three distinct storage engines: Qdrant for vector similarity search, PostgreSQL for metadata, and Neo4j for knowledge graph representation. Orchestration is handled via Docker Compose, allowing for a self-contained, reproducible deployment. This walkthrough details each stage of the pipeline and the underlying concepts, drawing from the actual code used in the project.

Architecture at a Glance

The RAG pipeline begins with raw documents and transforms them through several stages before they can be queried by an LLM. The process can be visualized as a flow:

Diagram showing document ingestion, chunking, embedding, and storage in Qdrant, Neo4j, and PostgreSQL.

Raw documents are ingested and processed by a series of modules:

  • Docling: This initial stage handles document loading and preprocessing. It ensures that various document formats (PDF, TXT, DOCX, etc.) are parsed into a consistent, usable text format.
  • Chunker: Large documents are broken down into smaller, manageable chunks. This is critical because LLMs have context window limitations, and smaller chunks improve the precision of retrieval. The chunking strategy can significantly impact retrieval quality.

Multi-Modal Retrieval Strategies

Once documents are chunked, they are prepared for retrieval using multiple methods, each serving a different purpose:

  • Dense Embeddings: Using models like those accessible via OpenRouter, text chunks are converted into high-dimensional numerical vectors. These embeddings capture semantic meaning, allowing for similarity searches based on conceptual understanding rather than just keyword matching. This is the cornerstone of most modern RAG systems.
  • Sparse BM25: Traditional keyword-based search, like BM25 implemented with libraries such as `fastembed`, provides a complementary retrieval mechanism. While dense embeddings excel at semantic understanding, BM25 is effective for retrieving exact matches or when specific keywords are crucial.
  • LLM Triple Extraction: This advanced technique uses an LLM to extract structured data, specifically subject-predicate-object triples, from the text chunks. This transforms unstructured text into a knowledge graph, enabling more complex relational queries.

Integrated Storage for Diverse Needs

The outputs from these retrieval strategies are stored in specialized databases, each suited for different query types:

  • Qdrant (Named Vectors): This vector database is optimized for storing and querying dense embeddings. It enables fast similarity searches, allowing the system to find document chunks semantically similar to a user's query. The concept of "named vectors" allows for storing multiple embedding types (e.g., different models) for the same chunk.
  • Neo4j (Knowledge Graph): The extracted triples are stored in Neo4j, a native graph database. This allows for querying relationships between entities, providing a more structured and inferential retrieval capability beyond simple text similarity.
  • PostgreSQL (Metadata): A traditional relational database like PostgreSQL is used to store metadata associated with the documents and chunks. This includes information like document source, author, creation date, chunk ID, and any other descriptive attributes that aid in filtering or contextualizing search results.

Orchestration with Docker Compose

Bringing these disparate components together requires robust orchestration. Docker Compose simplifies the deployment and management of the entire stack. Each service—FastAPI backend, React frontend, Qdrant, PostgreSQL, Neo4j, and any embedding model services—runs in its own container. Docker Compose defines the services, their dependencies, network configurations, and volumes, ensuring that the entire application can be spun up or down with a single command. This is crucial for development, testing, and self-hosting, providing a consistent environment across different machines.

The RAG Pipeline in Action

When a user submits a query:

  1. The query is first processed by the FastAPI backend.
  2. It might be embedded using a dense model and/or processed by BM25.
  3. The system queries Qdrant for semantically similar chunks and potentially Neo4j for related entities based on graph traversals.
  4. PostgreSQL is queried for metadata to filter or refine results.
  5. The retrieved relevant chunks, along with the original query, are then passed to a Large Language Model (LLM).
  6. The LLM synthesizes an answer based on the provided context and its own knowledge.

The surprising detail here is not the complexity of each component, but how effectively they integrate. Each database and retrieval method plays a distinct, non-redundant role, creating a system far more powerful than the sum of its parts. This multi-pronged approach to retrieval ensures that the LLM receives the most relevant and contextualized information, leading to more accurate and comprehensive answers.

Beyond the Basics: Future Considerations

While this stack provides a robust foundation, further enhancements are possible. These include implementing more sophisticated chunking strategies, exploring different embedding models and their trade-offs, adding user authentication and access control, and developing more advanced prompt engineering techniques to guide the LLM's synthesis process. The choice of LLM itself is also a critical factor, influencing the quality of both the triple extraction and the final answer generation.

This self-hosted approach offers complete control over data, privacy, and infrastructure. It's a blueprint for developers and organizations looking to build custom RAG solutions without relying on third-party APIs for core components, demonstrating that a powerful, adaptable RAG system is achievable with careful architectural design.