Designing incremental document updates
Make the update as small as the change. Explore hashes, chunk boundaries, and safe publication.
The system we are looking at
Every technique in this piece depends on the deployment it is applied to, so it is worth being specific about the one under examination. A retrieval assistant sits in front of roughly five million documents: policies, manuals, contracts, wiki pages, exported PDFs. None of them are read at answer time. Each has been parsed into text, cut into chunks, passed through an embedding model, and written into two search indexes, one matching on meaning and one on keywords, with permission metadata stored alongside.
The corpus has a shape worth naming, because it is the shape that decides everything later. Documents are long-lived and edited in place rather than replaced: a 200-page policy at roughly four chunks per page is about 800 chunks, and over a year it collects corrections a few at a time. Sources are mixed. Some systems emit change events, some are crawled on a schedule, and several deliver a freshly exported file every night whether or not a word of it moved.
Ingestion today runs at the document level. A change event or a crawl marks a document as dirty and the pipeline reprocesses it end to end: parse, chunk, embed, write. This is a sound design and the one most teams arrive at first. It is also trivially correct: when the job finishes, every chunk of that document reflects the current source, with no bookkeeping required to prove it.
The figures used throughout are illustrative teaching numbers rather than measurements. The ratios between them are what matter.
Where that design stops scaling
The strain comes from a mismatch of units. The unit of ingestion is a document; the unit of change is a paragraph. While those two are comparable in size the design holds comfortably. As documents grow long and edits stay small, the distance between them becomes the operating cost.
Concretely: a one-page correction to the 800-chunk policy sends all 800 chunks back through the embedding model. Twelve of them, the edited page and the two neighbours whose overlap crosses the change, carry new content. The other 788 are identical text, re-embedded because the pipeline has no way to say which twelve mattered. Work is proportional to document size rather than to edit size, and that ratio degrades as the library matures.
The consequences are not only financial. Large jobs occupy the pipeline, so an urgent correction queues behind a bulk re-export that changed nothing. Freshness is bounded by how long a whole document takes to process rather than by how small the edit was. And because one document becomes a sequence of writes across two indexes, there is a window during which a query can retrieve the new page alongside a summary derived from the old one.
Seen from outside the system, this is the complaint that usually starts the project: the assistant is still quoting last week’s rule. The cause is structural rather than operational. The pipeline cannot describe a change, so it replaces a document.
Try it yourself.
For a better experience, use full screen
Five million documents, rebuilt a document at a time.
The assistant never reads the originals. Every document is parsed, cut into chunks, embedded and written into two indexes. When anything marks a document dirty the pipeline reprocesses it end to end: 800 chunks for this policy, whichever page was touched. Send it an event and watch what that costs.
SEND AN EVENTOne page rewritten in place. Twelve chunks hold new text, counting the two neighbours the overlap window reaches. Chunks are cut with a little overlap so a sentence spanning a boundary is still findable.
Every mark is one chunk; every group of four is a page. All 200 pages of the policy are here: 800 chunks.
The system today reads a “correct page 42” as a document changed, and sends 800 chunks to the embedding model: 12 carrying new text, 788 identical to what is already indexed.
Try “Re-export, nothing changed”. The pipeline cannot tell it apart from a real edit, and pays the full price anyway.
- 12 chunks of genuinely new text
- 788 chunks of identical text, embedded again
The pipeline cannot say which part moved, so it redoes the document.
Illustrative model: 200 pages at four chunks per page, one edited page affecting its two overlapping neighbours, and an insertion at page 42 re-cutting the 159 pages behind it. Counts are teaching figures, not measurements.
Four things
to get right.
Compare normalized content hashes to locate the change. Parsing the full file may still be necessary.
Rebuild changed chunks and any overlapping neighbors. Refresh dependent summaries separately.
Stage vector and keyword records, verify visibility, then publish an active version manifest.
Validate retrieved revisions and cached answers. Retire old records after in-flight queries finish.