Deep-Dive Technical Post: GitHub Codebase RAG

Standard Retrieval-Augmented Generation (RAG) implementations—typically relying on character-count chunking and pure cosine-similarity vector lookups—fall short when applied to source code repositories. Code relies on strict hierarchical structure, scoping rules, dependency graphs, and exact string matches for function and variable identifiers.

To overcome these constraints, GitHub Codebase RAG separates key components into a decoupled, asynchronous microservices architecture built around Abstract Syntax Tree (AST) parsing, hybrid vector/keyword search, cross-encoder re-ranking, and strict tenant isolation.

Architectural Breakdown

The system separates synchronous API operations, background indexing workloads, and heavy machine learning inference into independent services communicating over async protocols and shared data backplanes.

Client (curl / API)
        │
        ▼
┌───────────────────┐
│   FastAPI Server  │  handles HTTP, query execution, evaluation
│   (api service)   │
└────────┬──────────┘
         │ enqueue job (arq)
         ▼
┌───────────────────┐
│       Redis       │  job queue + hash state storage
└────────┬──────────┘
         │ poll + dequeue
         ▼
┌───────────────────┐
│    ARQ Worker     │  executes fetch, chunk, embed, store pipeline
│ (worker service)  │
└────────┬──────────┘
         │ HTTP / JSON (batched requests)
         ▼
┌───────────────────┐
│ Embedding Service │  dense + sparse embeddings, cross-encoder reranking
│ (embedding svc)   │  models pre-loaded in memory
└───────────────────┘
         │
         ▼
┌───────────────────┐
│   Qdrant Cloud    │  vector storage, payload filtering, RRF hybrid search
└───────────────────┘

Pipeline Implementation Details

1. Structure-Aware Chunking via Tree-Sitter ASTs

Splitting source code by static line counts or arbitrary character limits breaks function signatures, isolates variable declarations from their call sites, and disrupts lexical scope.

The ingestion engine uses tree-sitter and tree-sitter-language-pack to parse source files into Abstract Syntax Trees (ASTs). The parser walks the generated AST nodes to extract discrete code blocks based on language-specific grammar definitions:

• Supported Languages: TypeScript, TSX, JavaScript, Python, Go, Rust, Java, C, and C++.
• Node Targets: Scoped to top-level constructs such as function_declaration, method_declaration, class_declaration, interface_declaration, and impl_item.
• Metadata Attachment: Every chunk captures operational context, preserving its surrounding metadata:
• File path (path)
• Inclusive line boundaries (start_line, end_line)
• Associated code payload (code)
• Calculated SHA-256 hash (hash)
• Fallback Strategy: Files with unsupported extensions or invalid syntaxes default to line-based sliding-window chunking with configured overlap.

2. Hybrid Retrieval Protocol (Dense + Sparse + RRF)

Code queries present two opposing search demands:

1. Semantic Queries: "How is user session validation handled?"
2. Exact-Match Queries: "Where is validate_jwt_bearer_token declared?"

Single-vector strategies struggle to satisfy both simultaneously. This system runs a hybrid retrieval pipeline combining dense semantic search, sparse lexical keyword search, and Reciprocal Rank Fusion (RRF).

                            User Query
                                │
          ┌─────────────────────┴─────────────────────┐
          ▼                                           ▼
Dense Embedding Vector                     Sparse Feature Map
(all-MiniLM-L6-v2)                       (BM42 / FastEmbed)
          │                                           │
          ├─────────────────────┬─────────────────────┘
          ▼                     ▼
    Dense Vector Search    Sparse Keyword Search
    (Cosine Similarity)     (Inverted Index)
          │                               │
          └─────────────────────┬─────────┘
                                ▼
                    Reciprocal Rank Fusion (RRF)
                                │
                                ▼
                 Cross-Encoder Re-Ranker
             (ms-marco-MiniLM-L-6-v2)
                                │
                                ▼
                       Top-K Chunks to LLM

• Dense Embeddings: all-MiniLM-L6-v2 produces a 384-dimensional dense vector representing semantic concepts.
• Sparse Embeddings: Qdrant/bm42-all-minilm-l6-v2-attentions (FastEmbed) computes token importance weights derived from self-attention maps, outperforming traditional static BM25 term frequencies for code syntax.
• Reciprocal Rank Fusion (RRF): Qdrant merges top hits from both dense and sparse indices into a consolidated candidate pool using reciprocal rank scoring:

$$RRF\Score(d \in D) = \sum{m \in M} \frac{1}{k + r_m(d)}$$

Where $M$ is the set of search modalities (dense and sparse), \(r_m(d)\) is the rank position of document $d$ within modality $m$, and $k$ is a smoothing constant (typically $60$).

3. Cross-Encoder Re-Ranking Stage

Vector distance calculations score query vectors against document vectors independently. To account for joint token dependencies between the input question and candidate code, candidates undergo a second pass via a cross-encoder model.

• Candidate chunks retrieved during the initial vector search stage (e.g., \(K_{\text{candidate}} = 20\)) are sent to the embedding service.
• A cross-encoder/ms-marco-MiniLM-L-6-v2 transformer processes the concatenated sequence [CLS] Query [SEP] Code_Chunk [SEP] through cross-attention mechanisms.
• Chunks are re-scored, sorted, and truncated down to the final context window (\(K_{\text{final}} = 5\)).
• The refined context is passed to Gemini 2.5 Flash with strict instructions to generate answers grounded in retrieved lines and cite exact file spans.

Infrastructure and Architectural Decisions

Isolated Embedding Microservice

Loading PyTorch/ONNX models directly inside the FastAPI process or ARQ worker pool causes significant issues: memory duplications across multi-worker deployments, high startup latency, and resource contention on CPU/GPU hardware.

• Single-Instance Loading: The embedding_service container pre-loads models (all-MiniLM-L6-v2, BM42, and ms-marco-MiniLM-L-6-v2) into shared memory once on startup.
• Non-Blocking Async Execution: Dense generation, sparse encoding, and cross-encoder inference execute within threadpool executors via Python's asyncio.to_thread.
• Microservice Communication: The API server and background workers communicate with the embedding service via persistent asynchronous HTTP connections managed by httpx.

Distributed Queue and Incremental Indexing

Repository indexing requires high-throughput I/O and processing phases: cloning or pulling via GitHub API, AST parsing, batched embedding generation, and Qdrant upserts.

• Asynchronous Jobs: Requests sent to POST /api/index push job payload parameters onto a Redis-backed ARQ queue and return a job_id immediately.
• Granular Progress Telemetry: The worker updates real-time counters in Redis (total_files, fetched_files, chunked_files, embedded_chunks, stored_chunks) accessible to clients polling GET /api/index/{job_id}.
• Incremental Synchronization: File hashes are stored in a Redis hash set mapped to each repository:

1. During re-indexing, incoming files are hashed using SHA-256.
2. Unchanged files (matching stored hashes) bypass chunking and embedding entirely.
3. Modified files update existing vector payloads in Qdrant.
4. Deleted repository files are identified and cleared from vector storage via Qdrant payload filters.

Single Collection Tenant Isolation

To avoid overhead and collection-management constraints in Qdrant, all repositories share a single vector collection. Isolation is enforced at the storage engine level:

• Every point payload includes an indexed field: "repo": "owner/repository".
• A Keyword Payload Index is configured specifically for the repo field.
• Every retrieval operation appends a hard Qdrant filter constraint:

{
  "filter": {
    "must": [
      {
        "key": "repo",
        "match": { "value": "owner/repository" }
      }
    ]
  }
}

Observability & Automated Evaluation

RAG validation requires tracing execution across all execution steps and continuously measuring pipeline performance.

       Query Execution Trace
                 │
                 ▼
┌────────────────────────────────┐
│      @observe (DeepEval)       │
│  Agent ──► Retriever           │
│         ──► Reranker ──► LLM   │
└────────────────┬───────────────┘
                 │
                 ▼
┌────────────────────────────────┐
│     Confident AI Dashboard     │
└────────────────────────────────┘

• Execution Tracing: Query execution paths are annotated with DeepEval's @observe decorator, streaming telemetry (retrieval latencies, top-k candidate payloads, reranker scores, and final LLM outputs) directly to the Confident AI dashboard.
• Automated Quality Testing: A dedicated endpoint (POST /api/evaluate) runs real-time automated metrics against the execution pipeline using LLM-as-a-Judge paradigms:

Metric	Target Component	Measurement Focus	Evaluation Method
Faithfulness	Generator (LLM)	Quantifies claims in the generated output that directly align with provided source context (detects hallucinations).	Evaluates factual consistency against context statements.
Answer Relevancy	Generator (LLM)	Evaluates how directly the answer addresses the user's input prompt.	Extracts answer claims and computes ratio of query-relevant statements.
Contextual Precision	Retriever & Reranker	Checks if relevant chunks are ranked at the top of the context window.	Scores position-weighted relevance of retrieved context blocks.
Contextual Recall	Chunker & Retriever	Evaluates if the context includes all information needed to form the response.	Compares retrieved context against reference ground-truth answers.

Repository Access

Full deployment configurations, Docker Compose specifications, requirements, and server implementations are available on GitHub:

Source Repository: github.com/kVarunkk/GitHub-Codebase-RAG