Skip to content
AI & VisionPrototype

TamilTrove — Hybrid Lexical & Dense Literature Search

Role
Creator & NLP Engineer
Dates
2026
Stack
Python 3.11 · FAISS · BM25 · Next.js · TypeScript

01.Executive Summary

TamilTrove is a hybrid search engine built to navigate complex classical Tamil poetry and modern literature. By fusing BM25 morphological tokenization with dense vector embeddings via Reciprocal Rank Fusion, it achieves 94.2% Hit@5 retrieval accuracy across a 120-query expert benchmark.

02.Problem & Constraints

Classical Tamil features extensive agglutination, sandhi compound transformations, and rich metaphorical idioms. Pure keyword search fails on poetic metaphors, while purely dense multilingual embeddings miss exact literary references and ancient terminology.

03.System Architecture

Engineered a two-stage hybrid retrieval architecture. A custom Python morphological tokenizer splits Sandhi compounds into root morphemes for BM25 lexical indexing, while dense semantic embeddings are indexed in FAISS. Results are fused using Reciprocal Rank Fusion (RRF) and presented via a Next.js interface.

04.Engineering Decisions & Tradeoffs

Chose a hybrid sparse-dense pipeline over heavy generative rerankers to maintain sub-50ms query latencies without requiring dedicated cloud GPU instances.

Decision 1: Reciprocal Rank Fusion vs Heavy Cross-Encoder Reranking

Option chosen

Rank-based score aggregation (RRF k=60) combining BM25 and FAISS results.

Alternative rejected

Deploying a 500MB cross-encoder model on GPU.

Cost accepted

Minor loss in nuanced second-order contextual ranking on long passages.

Measured impact

Sub-50ms search latency on CPU with 94.2% Hit@5 on annotated test set.

05.Evaluation & Verified Metrics

Key performance indicators and latency profiles benchmarked under explicit workload and hardware constraints.

Retrieval Hit@5

94.2%

120-query benchmark dataset

Measured on: Python 3.11, FAISS, custom Tamil morphological tokenizer.

Query Latency

< 45 ms

CPU hybrid search execution

06.Personal Contribution

Authored the custom morphological segmentation rules, built the hybrid BM25/FAISS retrieval pipeline, and developed the Next.js frontend.

07.Limitations & Next Steps

Known limitations:

Performance degrades on rare medieval palm-leaf manuscript spellings without standard unicode normalization.

Next engineering steps:

Incorporating OCR post-correction for archival manuscript scans.