AI Engineer | Transformers & LLMs Enthusiast

Summary

The Challenge

Legal teams spend significant time manually auditing privacy policies for DPDP Act compliance. Inaccuracies in these audits carry substantial regulatory risks and potential penalties.

The Strategic Solution

Developed a 4-agent orchestration pipeline to automate requirement decomposition, semantic retrieval, and cross-validation. This system ensures high-accuracy citations and verification, moving from manual 60-hour audits to streamlined 2-minute processing.

Technical Implementation

Sequential Processing Bottleneck

First implementation: 38 requirements × 5s per LLM call = 190s. Unacceptable.

Fix: FastAPI Async

async def evaluate_all(requirements):
    tasks = [evaluate_single(req) for req in requirements]
    return await asyncio.gather(*tasks, return_exceptions=True)

38x speedup. But now 38 API calls fail simultaneously.

LLM Reliability

OpenAI has rate limits, timeouts, malformed JSON. Needed defensive engineering.

Fix: Retry + Defensive Parsing

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
async def call_llm(prompt: str) -> dict:
    try:
        response = await openai_client.chat.completions.create(...)
        return parse_llm_response(response.content)
    except OpenAIError:
        return {"status": "UNKNOWN", "reasoning": "LLM unavailable"}

GPT-4 sometimes wraps JSON in markdown fences:

def parse_llm_response(raw: str) -> dict:
    clean = raw.strip()
    if clean.startswith("```"):
        clean = clean.split("```")[1]
        if clean.startswith("json"):
            clean = clean[4:]
    return json.loads(clean)

Vector Search Misses Legal Phrases

Pure semantic search fails. Policy says "we protect data" (similar to "encryption") but doesn't mention encryption. That's non-compliant.

Fix: Hybrid Search

def hybrid_search(query: str, top_k: int = 10):
    vector_results = vector_search(query, top_k)  # pgvector
    bm25_results = bm25_search(query, top_k)      # exact phrases
    return reciprocal_rank_fusion(vector_results, bm25_results)

Catches both concepts and exact terminology.

Database Design for Court

Every audit must be defensible. Immutable logs, precise citations.

Schema:

CREATE TABLE document_chunks (
    id UUID PRIMARY KEY,
    document_id UUID REFERENCES documents(id),
    text TEXT NOT NULL,
    page_number INTEGER,
    bbox JSONB,  -- {x0, y0, x1, y1}
    embedding vector(1536),
    created_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_embeddings_cosine 
ON document_chunks 
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

bbox enables citations like "Page 7, Section 3.2, (120, 450)".

Testing Against Production

Tests passed locally with SQLite. Crashed in production—pgvector's Vector(1536) doesn't exist in SQLite.

Fix: GitHub Actions with real Postgres container.

Multi-Agent Design

PLANNER: Breaks DPDP Act into atomic tasks
RETRIEVER: Hybrid search for relevant clauses
REASONER: Evaluates compliance per requirement
VERIFIER: Catches hallucinations before delivery

Each agent is testable, upgradeable, observable.

pgvector over Pinecone: For 10K docs: $0 cost, <5ms latency, full data control.

Docker Compose: docker-compose up starts Postgres+pgvector, runs migrations, seeds DPDP framework, starts API.

Learnings

Async is non-negotiable for LLM orchestration
Hybrid search > pure vector for legal text
Verification layers reduce hallucinations
Test against production stack (SQLite lies)
Immutable audit logs for regulated industries

Production AI is 20% prompts, 80% systems engineering.