∫
∑
∆
∂
Back to Projects
Local AI / Infrastructure

Privacy-First Local RAG

Zero cloud dependencies. 100% privacy compliance. Significant cost reduction by eliminating token-based pricing for high-volume internal document analysis.

Source Code

Summary

The Challenge

Sending sensitive data to cloud LLMs is a legal bottleneck for regulated industries (GDPR, HIPAA). Cloud processing is expensive at scale and introduces network latency that breaks real-time workflows.

The Strategic Solution

Built an end-to-end RAG pipeline that stays 100% offline. Combines ChromaDB for vector search with Ollama for local model orchestration. Handles text, complex PDFs, and scanned images via Tesseract OCR.

Technical Implementation

Data Sovereignty & Offline Intelligence

The primary goal: analysis without the internet.

System Flow

Problem 1: Context Fragmentation

Standard chunking loses semantic coherence. If a sentence is split, the model loses the context.

Fix: Recursive Overlap Chunking Implemented a 1,800-character window with a 200-character overlap. This ensures that every knowledge snippet contains enough preceding context to be meaningful during retrieval.

OCR Pipeline

Problem 2: Scanned Document Gap

Many enterprise 'documents' are just flat images inside PDFs. Standard parsers return empty strings.

Fix: Tesseract OCR Pipeline Architected a fallback mechanism. If a page yields zero text, the system triggers image preprocessing (grayscale + thresholding) and passes the buffer to Tesseract.

def process_scanned_page(page_image):
    # Denoising and thresholding for higher OCR accuracy
    processed = cv2.threshold(page_image, 127, 255, cv2.THRESH_BINARY)[1]
    return pytesseract.image_to_string(processed)

Problem 3: Semantic Retrieval Noise

Searching the entire database is noisy. Top-K retrieval needs to be precise.

Search Interface

Fix: Local Embedding Fusion Used all-MiniLM-L6-v2 for vectorization. It’s optimized for local hardware, producing 384-dimensional embeddings that fit in RAM while maintaining high cosine similarity accuracy.

Storage & Persistence

Data is managed in a local /chroma_db directory. No external clusters. No subscription fees.

System Stats

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection(name="local_knowledge_base")

Infrastructure

FastAPI: Provides the /query, /upload, and /stats endpoints. Async support handles ingestion while the LLM is busy with inference.

Local Inference: Ollama manages the model weight lifecycle. Inference speed is optimized via quantization, allowing 7B+ parameter models to run on standard workstation GPUs.

Key Learnings

  1. Local > Cloud: For privacy, hardware-bound AI is the only defensible architecture.
  2. Preprocessing is 80% of RAG: Better chunking and OCR beats a better LLM every time.
  3. Quantization is Essential: Running local models requires aggressive weight optimization to stay within VRAM limits.