Elasticsearch — Part 2: Internals & The Near-Real-Time Write Path

maang.io System Design Series

What is this?

Back to the library. In Part 1 we saw the master index that answers "who mentions X?" Now zoom into one branch and watch the librarian actually file a new book. A careless librarian would tear apart the branch's entire index every time a book arrives — reopen it, insert the new terms in the right alphabetical slots, re-stitch it. Unworkable. A clever librarian does something else: they jot the arrival in a daybook (so nothing is ever lost), keep the new book on a small incoming cart that's itself indexed, and periodically bind that cart's worth of arrivals into a small, sealed booklet that gets added to the shelf of booklets. Searches just consult all the booklets. Old booklets are never edited — occasionally several small ones get re-bound into one big tidy one.

That is almost exactly how a single Elasticsearch shard stores data, and it explains its two headline properties: writes are cheap, and a document becomes searchable about a second after you index it — "near real time," not instant, and there's a precise reason why. The sealed booklets have a name — Lucene segments — and the daybook is the translog. Let's build the whole picture from the inverted index up.

The one-line idea: A shard never edits its search index in place. New documents land in an in-memory buffer, get periodically refreshed into a brand-new immutable segment (that's when they become searchable), and are protected in the meantime by an append-only translog. Deletes and updates don't erase anything — they mark old data dead, and background merges reclaim it later. Immutability is the whole trick.