The Index Is a Bet You Made at Write Time
Search feels instantaneous.
We type a few words and hit enter. Millions of documents collapse into a neat list of results in less than a second. It gives the illusion of a system that heard your request, sprinted through billions of pages and found the exact results the moment you asked.
It does feel like magic.
But that's not what happened. By the time you typed your query, most of the important work had already been done. Long before you searched for anything, the system had already broken documents apart, stripped words down, discarded some terms entirely, reorganised the remaining data, and built a structure designed to make retrieval cheap later. It did all of this without knowing the query you would eventually type. Search feels reactive, but much of the work that makes it fast happened in advance.
However, that preparation is not neutral. To build a useful representation of the data before the query exists, the system has to make choices about what is worth preserving, what can be normalised, and which distinctions future queries are likely to need. Some of those choices will be wrong.
A few months ago, I joined a new team at work. During one of my onboarding conversations, my Senior Engineering Manager, Vladimir, mentioned our usage of inverted indexes. The idea sounds simple: reorganise data ahead of time so retrieval becomes cheap later.
For a few days, the concept quietly gnawed at me. Early in your career, you just marvel at how fast a system returns a result. But once you've had to debug enough production systems, you learn to distrust free lunches. You look at speed with suspicion, because making one operation cheap usually means paying for it somewhere else.
And here, we pay before the query arrives. We decide which properties of the data are worth preserving for retrieval without knowing exactly what someone will eventually ask of it.
And that raised a question I could not stop thinking about:
How do you decide what to preserve when you don't yet know what someone will ask?
The Naive System
Let's imagine building search in the most straightforward way possible.
You type "distributed systems". The system immediately begins a linear scan. It opens the first document and checks if it contains "distributed" and "systems".
Then the second document. And you keep going, repeating the same process across millions of documents.
At a small scale, it works. If you only have a few hundred documents, scanning everything at query time is simple and probably fast enough. The system does not need complicated data structures.
Now, imagine building Google Search. Imagine doing that at web scale. At a few hundred documents, O(N) is nothing but an implementation detail. At web scale, it is a non-starter.
If every single query required the system to repeatedly open documents, scan text, compare tokens, and search through enormous amounts of data in real time, the universe would end before your results loaded.
Every new query sends us back through the corpus (set of documents) again. The documents may not have changed, but we repeatedly rediscover where their words are.
It is an architectural dead end. At some point, you'd stop asking the question of how to search documents faster and start asking, “Why are we searching the documents at query time at all?"
We already have the documents. Why don't we inspect them before anyone searches, organise the information we find, and build a structure that makes retrieval cheaper when a query eventually arrives.
In other words, we can move some of the work earlier.
That is where the inversion begins.
The Inversion
The solution is to flip the relationship entirely and organise it by word, not by document. Instead of storing Document → Words, we invert the relationship:
Word → Documents
Every time a document enters the system, it is broken apart into tokens. At its simplest, the index records which terms appear in which documents. Depending on how it's built, it can also record how often a term appears and where it appears within the document, allowing us to tell how close two words are to each other.
So instead of scanning the entire corpus every time someone searches for "distributed systems", the system can jump directly to documents containing "distributed", documents containing "systems", and find the overlap, classic Venn diagram style.
Search stops being a repeated traversal problem and becomes a lookup problem. We have moved much of the work of discovering where terms occur out of the query path and into indexing, making retrieval much cheaper when the query eventually arrives.
But there is an important consequence to doing that.
The search engine is no longer starting from the documents themselves. It is consulting a representation we built from them in advance.
This optimisation is a form of commitment. And in software architecture, as in life, commitment always demands a sacrifice.
Our slow, naive system never had to make quite the same commitment. It went back to the source material for every query. The inverted index gets its speed precisely by avoiding that work.
We are searching a precomputed map of the original data, and a map has to decide what is worth drawing. We can build extra representations for different kinds of retrieval, preserve positions, or keep the raw text around, but every additional representation is another tax on storage and indexing work. Eventually, we have to decide which distinctions are worth paying to make efficiently retrievable.
What the Index Assumes
Human languages are messy, ambiguous, and full of edge cases. To make them efficiently searchable, we transform them into a structure a machine can work with.
We decide where tokens begin and end, collapse different forms into shared representations, and discard terms we think contribute little to retrieval. None of these transformations is neutral. Every step embeds an assumption about which distinctions in the original text are actually going to matter later.
What counts as a word?
While we rarely think about this question, search systems cannot avoid it. Is state-of-the-art one word or four? Does the apostrophe make don't one word or two? What should happen to something like C++, where the symbols attached to the letter are part of the word's identity?
Before we can build our index, we have to turn those answers into rules. This process of breaking text into smaller units called tokens is called tokenisation. It is not just splitting text on spaces. It determines where one searchable unit ends and another begins. If our tokeniser only recognises sequences of letters and numbers, C++ might simply become C.
Consider the sentence: "The distributed systems book was surprisingly interesting."
After tokenisation, it might become:["the", "distributed", "systems", "book", "was", "surprisingly", "interesting"]
The index can still preserve additional information about those tokens, including their positions in the original text. But the tokenisation itself has already made a decision: what counts as a searchable unit.
Which Words Matter?
Open almost any document, and certain words like "the", "and", "is", or "of" appear everywhere. They occur in enormous numbers of documents and often do little to distinguish one document from another. We might designate these as stop words and exclude them from the index because we expect them to contribute little to retrieval.
Our array of tokens:
["the", "distributed", "systems", "book", "was", "surprisingly", "interesting"]
might become
["distributed", "systems", "book", "surprisingly", "interesting"].
For many searches, this is perfectly reasonable. The trouble starts when one of the words we decided was unimportant becomes the thing someone actually cares about.
A historian might want to know how often a particular US Senator uses the word "the". The original speeches may still contain every occurrence, but our index has nothing useful to tell them. We decided not to make that distinction efficiently retrievable.
Now consider Shakespeare's famous dilemma under an aggressive stop-word list: "To be or not to be." If every word in that query has been excluded from the index, there will be nothing left to look up.
This particular bet assumes that some words will contribute little to retrieval. Someone made that call before any query existed. And now, nobody really thinks of it as a decision anymore; just how things are.
When Are Two Words The Same?
We are creative with our vocabulary. We conjugate, pluralise, and change tense. We rarely struggle to see the relationship between run, running, runs and ran. But a search system processing their character sequences does not get that relationship for free. Without any further processing, run and running are simply two different tokens. If we want a search for one form to retrieve documents containing another, we need some way of bringing those forms together.
Two common ways for collapsing different grammatical forms towards a common representation are stemming and lemmatisation.
Starting from the same array of tokens:
["distributed", "systems", "book", "surprisingly", "interesting"]
Run it through a standard stemmer like Porter, and they become:
["distribut", "system", "book", "surprisingli", "interest"]
Exactly what comes out depends on which algorithm you pick. Porter, Snowball, and Lancaster can produce different stems.
Some of those look broken. Distribut and surprisingli aren't real English words, but the stemmer doesn't care. It isn't trying to write prose; it’s applying deterministic rules to chop off or replace common suffixes, producing consistent keys for fast lookups.
Lemmatisation takes a more linguistic approach, reducing a word to its dictionary form, or lemma. Our tokens might instead look more like
["distribute", "system", "book", "surprisingly", "interesting"]
The difference is clearer with irregular words. Ran cannot be reduced to run simply by chopping off a suffix, but a lemmatiser can recognise their grammatical relationship.
Whichever approach we choose, we are deciding that some differences in the original language are not important enough to keep separate for retrieval. Treating system and systems as the same word usually seems like the right call. Nobody wants literature on distributed systems to vanish just because they typed a singular noun into the search box.
But what happens when the suffix-chopping goes too far? A stemmer operates on character patterns, not meaning, so it can collapse words whose modern meanings we would rather keep separate. This is called overstemming. Run universe, university and universal through Porter, and every one collapses down to the stem univers.
Buried inside the useful behaviour of collapsing word variations is another assumption: that a distinction we collapse today will not turn out to matter to a future query.
There is one other search decision worth separating from all of this.
If we search for "distributed systems", the index might return millions of documents containing both words. Some of the documents mention the topic in passing, some are academic papers, some are spam, some are conference talks, and some will come from this blog. Here's the problem, we can't read everything.
It has to decide what to show first. That decision is ranking.
Ranking works differently from the decisions we've been examining. Tokenisation, stop words and stemming shape the representation available before the query arrives. Ranking usually combines available signals into an order for this query at query time. Some of those signals may have been computed earlier, but changing how they're combined may not require rebuilding the representation underneath them.
The bet is still there, but it lives somewhere different, closer to the surface, easier to revise, easier to A/B test on a Tuesday than a decision already materialised across the index.
What the Index Forgets
So far, we've made several decisions about which distinctions are worth preserving for retrieval. None of those choices is inherently a mistake. To be frank, most of them are what make the index useful.
The trouble starts when a future query needs one of the distinctions we chose not to preserve.
Take two sentences:
The database guarantees consistency.
The database does not guarantee consistency.
To anyone reading the original text, these are opposite claims. But run them through a pipeline that removes not as a stop word and normalises stems, and both can collapse into the same handful of tokens.
["databas", "guarante", "consist"]
The word not, the one token carrying the difference between the two claims, is gone from this representation. And if our index stores the terms without their positions, we've discarded another property of the original text: how those terms were arranged. The cost of the missing distinction is invisible until someone needs it.
The index is not the corpus. It is a representation of the corpus, built for a particular purpose. What it remembers depends on what we decided was worth preserving when we built it. We could store positions and preserve stop words. We could keep the original forms alongside the normalised ones. Search systems often do some combination of these things, but each one comes at a cost.
Changing the rules used to build an index does not automatically change the index we have already built. Those decisions have already been materialised across the corpus, so making a previously discarded distinction retrievable may mean processing those documents again and building a new representation.
Another problem with forgetting is that the index has no inherent way of knowing what it has forgotten. When a query asks for a distinction you stripped out five years ago, the index doesn't warn you that the information is still present in the source text but was not preserved during indexing.
It simply searches the representation it has.
Maybe it hands back irrelevant results. Maybe it hands back nothing at all.
From the search box, "there is nothing here" and "I cannot see what you are looking for" can look identical. One is a statement about the underlying information. The other is a limitation of the representation we built.
The index can only answer questions using the distinctions we chose to make available to it.
When the Bet Ages Badly
The previous section's problem was that the index could not tell you what it had forgotten. Time introduces a second problem: an assumption can stop being useful without anything in the index obviously breaking.
When engineers talk about a stale index, we usually mean it has fallen behind the data it represents. A new document hasn't appeared yet, a deleted one is still searchable, or a job is lagging somewhere behind the source of truth.
We know how to reason about this. The source changed, and the index hasn't caught up. Once the pipeline catches up, the problem will go away.
But an index can contain every document in the corpus and still be out of date.
Just imagine an English search index we happened to build in 2005. For our particular corpus, we decided to include go in its stop-word list because it appeared frequently while contributing very little to distinguishing one document from another. Our pipeline also lowercased tokens during normalisation to keep lookups uniform.
Both decisions were reasonable, and nothing about the corpus gave us an obvious reason to reconsider them.
Then, in 2009, Google released a programming language called Go.
The corpus starts filling with documentation, tutorials, conference talks, and Stack Overflow questions about the new language. The indexing rules do not change, because nothing in the system tells us that this old decision needs to be reconsidered.
Go is normalised to go. And go is discarded as a stop word.
Nothing is broken. There is no backlog, no failed job, and no replica waiting to catch up. If we wiped the cluster tonight and re-indexed the entire corpus from scratch, we would wake up to the same failure tomorrow. Every document processed, faithfully, through the same 2005 assumption about which words matter, rebuilding the same problem instead of fixing it. The corpus is completely current, but we're still interpreting it using a decision we made in 2005.
Go is not special here. A rule that drops single-character tokens might have been perfectly sensible when a corpus was mostly prose. Then that corpus starts accumulating technical documentation, and people need to search for programming languages like C and R. The rule hasn't changed but the corpus around it has.
That is the difference between mechanical staleness and assumption staleness. Mechanical staleness eventually heals when the pipeline catches up. An aged assumption does not heal, because running newer data through an older interpretation simply reproduces it.
Unfortunately, we can't turn to our monitoring dashboards.
When an indexing pipeline lags, we know what to measure. We track consumer offsets, replication delays, and document ingestion timestamps. We know how to put a number on indexing lag. Assumption staleness is harder to put a number on. Documents are ingested on schedule and the queries continue returning quickly. Every indexing SLA is met. None of those numbers tells us whether the assumptions underneath the index still fit the corpus.
The closest hints we get that an assumption may have aged badly are secondhand signals: a query that keeps coming back empty, users rephrasing their searches in different ways, a slow drift in what people click on. But those tell us something is wrong with the interaction between what people want and what the index can give them. They don't tell us which assumption is the problem, or when it became a problem.
At some point, someone has to go back and question the assumption itself.
When the Rules Become Representations
If you are like me, you are probably thinking that switching to modern semantic search might save us.
Stop-word lists and stemmers are fairly old-fashioned examples. Modern search systems can represent meaning using embeddings instead of relying entirely on hand-written rules about which words matter or which forms should be treated as equivalent. Perhaps the problem isn't that an index has to make bets before the query arrives, maybe we were simply making particularly crude ones, and a learned representation escapes the problem entirely.
With semantic search, we stop specifying many of those relationships by hand and use a model that has learned a representation from data instead. We no longer need a stemming rule to tell us that run and running are related. A search for car can retrieve a document that talks about automobiles without either word having to be transformed into the same token first.
That is a meaningful improvement. Some of the distinctions lexical search forces us to define explicitly no longer need explicit rules at all.
We are doing it differently now, but we are still building a representation before the query arrives.
Suppose we want to make a collection of technical documentation searchable using embeddings. We probably won't embed an entire manual as one vector. We'll break it into smaller chunks and embed those instead. Immediately, an old question returns in a different form: where does one retrievable unit end and another begin? A chunk can be a sentence, a paragraph, a fixed number of tokens, or something determined by the structure of the document. Make the chunks too small and information that belongs together ends up represented separately. Make them too large and a single vector has to represent several ideas at once.
Before, we just had to figure out what counted as a word. Now, we’re deciding how much text belongs together before we turn it into a vector. So, we haven’t exactly escaped the boundary problem.
And chunking is one of the decisions we can actually see. The embedding model contains another set of choices that are harder to inspect directly. Those relationships came from particular training data and a particular training objective. So the representation ends up being better at preserving some distinctions than others. Some things get compressed along the way. A model can capture relationships nobody sat down and explicitly enumerated. But when two pieces of text end up close together, there may be no equivalent of a stop-word file we can open to find the rule responsible.
We may be able to observe where retrieval is failing without being able to point to the particular representational decision responsible.
It can also go stale in a familiar way, mechanically current, every vector written, queries returning neighbours in milliseconds, while the representation underneath quietly stops fitting what we're now asking it to distinguish. This should sound familiar.
Semantic search can remove some of lexical search's more brittle assumptions. It also introduces new ones of its own, some explicit, like how we divide documents before embedding them, and some learned from data rather than written into configuration, which makes them harder to find and harder to question.
The Broader Systems Pattern
The index isn't all that unusual. If we look around, we'll see that software systems make bets about the future all the time. We cache values because we expect them to stay useful for a while. We design DB schemas around the shape we expect our data to take. We choose cryptographic primitives based on what we believe they can actually protect us against.
The real difference is what happens when those assumptions stop holding.
Take a cache. A cache does not just store values; it stores them under keys. The key decides which requests the system is willing to treat as equivalent. If we cache a response under (user_id, product_id), we are assuming that two requests from the same user for the same product can safely receive the same cached response.
Then we add localisation, and the response starts depending on the user's requested locale.
The cache can still be perfectly healthy. Entries expire when their TTL says they should, new values replace old ones and hit rates look normal. But when an expired entry is repopulated, it is written back under the same incomplete key. The TTL forces us to reconsider the value but it does not force us to reconsider the assumption we used to identify that value in the first place.
MD5 is a slightly different story.
MD5 was published in 1992 as a cryptographic hash function. Over the years, people kept finding weaknesses and increasingly practical ways of generating collisions. Eventually, relying on MD5 where collision resistance mattered was no longer considered safe.
The code still runs fine. If we feed it some bytes, it faithfully produces a 128-bit hash. What changed was what we could reasonably assume that hash guaranteed.
Even recognising that an assumption has aged does not make it disappear from deployed systems. In 2012, the Flame malware exploited an MD5 collision as part of an attack that produced a fraudulent certificate chaining to a Microsoft certificate authority, years after collision attacks had already given the industry reason to move away from MD5.
A cache key and a cryptographic hash function don't really have anything to do with tokenisation or stop words. But here’s what I keep coming back to: we make these choices based on what we happen to know at the time, and the system can carry them forward for years. Sometimes the world changes. Sometimes we just learn something new that makes our old choices look completely different. And the system can keep running through all of it, even when the assumption underneath it is no longer true.
That brings us back to the onboarding conversation, and finally, back to the search box.
Some of the decisions underneath that seemingly instantaneous response may have been made in the last sprint. Others may have survived several migrations, teams and generations of the system that originally introduced them.
The longer they survive, the easier it becomes to forget that they were decisions at all.
We experience the result as search: type a question and get an answer. What we don't see are all the earlier judgements about which distinctions were worth preserving, which representations to build, and what future users might eventually need from them.
By the time you arrive at the search box, those bets have already been made.
