Search Engineering Language Exercises

An inverted index is the core data structure of a full-text search engine. It maps each unique term in a corpus to the list of documents (and typically positions within those documents) where that term appears — the inverse of a document-to-terms mapping. This structure allows a search engine to find all documents containing a query term in O(1) time by looking up the term in the index. Engineers discuss inverted index design in terms of postings lists, term frequencies, document frequencies, and positional data for phrase queries.

BM25 (Best Match 25) is the standard probabilistic relevance ranking function used in modern search engines including Elasticsearch and Solr. It scores documents based on term frequency (TF), inverse document frequency (IDF), and document length normalisation. The two key parameters are k1 (controls term frequency saturation — typical value 1.2–2.0) and b (controls document length normalisation — typical value 0.75). Engineers tune these parameters based on corpus characteristics, discussing "b=0 disables length normalisation, useful for short-field queries like product names."

Relevance tuning is the iterative process of adjusting ranking signals and configuration to improve the quality of search results for a given query set. Engineers use relevance judgements — explicit ratings of document quality for a query (relevant, partially relevant, not relevant) — to measure improvements. In relevance review sessions, vocabulary includes precision (fraction of returned results that are relevant), recall (fraction of relevant documents returned), NDCG (Normalised Discounted Cumulative Gain), and MRR (Mean Reciprocal Rank). Phrases like "we're optimising for head queries" and "tail query recall is suffering" are standard.

Lexical search (keyword search) matches documents based on exact or stemmed term overlap between the query and document. Semantic search uses dense vector representations (embeddings) to find documents that are conceptually similar even when they share no common terms. Engineers explain: "lexical search handles exact product codes well but fails when users describe what they want in their own words — that's where semantic search adds value." Hybrid search combines both: a lexical retrieval pass followed by a semantic re-ranking stage.

Query understanding encompasses the processing steps that transform a raw user query into a structured search request. Stages include tokenisation, spelling correction, query classification (navigational, informational, transactional), entity recognition, synonym expansion, and intent detection. Engineers discuss "query rewriting" (transforming "iphone 15 pro max cover" into a structured product search) and "query relaxation" (broadening a query that returns no results by removing constraints). Strong query understanding is particularly important for e-commerce and enterprise search.

Search quality metrics vocabulary includes: Click-Through Rate (CTR) — fraction of impressions that result in a click; Position-Adjusted CTR — CTR normalised for rank position; Zero-Result Rate (ZRR) — fraction of queries returning no results; Search Abandonment Rate — users who leave without clicking a result; and nDCG (normalised Discounted Cumulative Gain) — a graded relevance metric weighted by rank position. Engineers also discuss A/B test metrics for ranking changes, using phrases like "the new model shows a statistically significant +3% improvement in nDCG@10 on our query evaluation set."

Faceted search allows users to filter search results by structured attributes (facets) such as brand, price range, colour, or category. Engineers implement facets using aggregations in Elasticsearch or Solr that count matching documents per attribute value alongside the main search query. Key design decisions include which facets to surface, whether to use post-filter (facet counts reflect main query) or global aggregation (facet counts are query-independent), and how to handle multi-select facets. Vocabulary includes facet counts, drill-down navigation, and hierarchical facets.

Embeddings are dense numerical vector representations of text, images, or other content, typically generated by a neural language model. In search, document and query embeddings are compared using cosine similarity or dot product to find semantically related content. Engineers discuss bi-encoder models (separate encoders for query and document, fast at retrieval) versus cross-encoder models (query and document processed together, more accurate but slower, used for re-ranking). Key vocabulary includes ANN (Approximate Nearest Neighbour) search, HNSW (Hierarchical Navigable Small World) index, and vector store.

Index refresh latency describes the delay between a document being written to the source system and becoming searchable. Engineers discuss "near-real-time (NRT) search" where new documents become visible after a configurable refresh interval (default 1 second in Elasticsearch) versus full re-indexing that rebuilds the index from scratch. Vocabulary includes segment merging, refresh interval, flush, and index alias swapping (used for zero-downtime re-indexing: build a new index, then atomically switch the alias). Indexing throughput and query latency are the primary trade-off levers.

Learning to rank (LTR) is a machine learning approach where a model is trained on relevance signals to predict the optimal ranking of search results for a query. Features fed to an LTR model include BM25 score, document freshness, click-through rate, user engagement signals, and entity match scores. Engineers discuss "pointwise", "pairwise", and "listwise" LTR approaches, and deploy LTR as a re-ranking stage after an initial candidate retrieval. Common vocabulary includes feature importance, training query set, offline evaluation (nDCG on held-out queries), and online evaluation (A/B test metrics).