The challenge
How do you find the most relevant documents among tens of thousands, and how do you know whether a change actually makes search better? The goal was to implement the central parts of a search system and measure every change against relevance judgements rather than gut feeling.
The solution
The system consists of four parts that build on each other:
- Baseline: an inverted index with separate title and body fields and a simple term-frequency score.
- Classical ranking models: BM25, fielded BM25F, a query-likelihood language model with smoothing, and a fielded variant.
- Learning-to-rank: a pointwise model intended to rerank BM25 results using query–document features.
- Query expansion: pseudo-relevance feedback that weights new query terms from the top-ranked documents.
How it was built
The work builds on a provided framework with interfaces and tests; I implemented the logic of the four parts. The index is stored in SQLite and covers the first 50,000 documents of the Washington Post collection used in TREC. Learning-to-rank uses gradient boosting (scikit-learn) on precomputed features, and query expansion uses RM3-style weighting of terms from the top BM25 documents. Quality is measured with P@10, MAP, MRR and nDCG@10 against official relevance judgements.
Result
BM25 gave a clear improvement over the baseline: nDCG@10 rose from 0.048 to 0.144 on the 50 official queries (top 100 results, same index). The fielded variants performed at about the same level as BM25, while the plain language model scored lower. Neither learning-to-rank nor query expansion gave a reliable additional improvement in this setup.
Context
The figures come from a later, reproducible evaluation. The index covers only part of the collection (about 6% of the documents judged relevant are in the index), so absolute values are low and not comparable with full-collection results; the comparisons use the same index and the same queries. The integrated reranking in practice used a single feature instead of the trained model, and when evaluated separately the trained model did not rank better than BM25 either. The effect of query expansion was negligible.