TamilTrove — Hybrid Lexical & Dense Literature Search
- Role
- Creator & NLP Engineer
- Dates
- 2026
- Stack
- Python 3.11 · FAISS · BM25 · Next.js · TypeScript
01.Executive Summary
TamilTrove is a hybrid search engine built to navigate complex classical Tamil poetry and modern literature. By fusing BM25 morphological tokenization with dense vector embeddings via Reciprocal Rank Fusion, it achieves 94.2% Hit@5 retrieval accuracy across a 120-query expert benchmark.
02.Problem & Constraints
Classical Tamil features extensive agglutination, sandhi compound transformations, and rich metaphorical idioms. Pure keyword search fails on poetic metaphors, while purely dense multilingual embeddings miss exact literary references and ancient terminology.
03.System Architecture
Engineered a two-stage hybrid retrieval architecture. A custom Python morphological tokenizer splits Sandhi compounds into root morphemes for BM25 lexical indexing, while dense semantic embeddings are indexed in FAISS. Results are fused using Reciprocal Rank Fusion (RRF) and presented via a Next.js interface.
04.Engineering Decisions & Tradeoffs
Chose a hybrid sparse-dense pipeline over heavy generative rerankers to maintain sub-50ms query latencies without requiring dedicated cloud GPU instances.
Decision 1: Reciprocal Rank Fusion vs Heavy Cross-Encoder Reranking
Rank-based score aggregation (RRF k=60) combining BM25 and FAISS results.
Deploying a 500MB cross-encoder model on GPU.
Minor loss in nuanced second-order contextual ranking on long passages.
Sub-50ms search latency on CPU with 94.2% Hit@5 on annotated test set.
05.Evaluation & Verified Metrics
Key performance indicators and latency profiles benchmarked under explicit workload and hardware constraints.
Retrieval Hit@5
120-query benchmark dataset
Measured on: Python 3.11, FAISS, custom Tamil morphological tokenizer.
Query Latency
CPU hybrid search execution
06.Personal Contribution
Authored the custom morphological segmentation rules, built the hybrid BM25/FAISS retrieval pipeline, and developed the Next.js frontend.
07.Limitations & Next Steps
Known limitations:
Performance degrades on rare medieval palm-leaf manuscript spellings without standard unicode normalization.
Next engineering steps:
Incorporating OCR post-correction for archival manuscript scans.