Designing incremental document updates

The system we are looking at

Every technique in this piece depends on the deployment it is applied to, so it is worth being specific about the one under examination. A retrieval assistant sits in front of roughly five million documents: policies, manuals, contracts, wiki pages, exported PDFs. None of them are read at answer time. Each has been parsed into text, cut into chunks, passed through an embedding model, and written into two search indexes, one matching on meaning and one on keywords, with permission metadata stored alongside.

The corpus has a shape worth naming, because it is the shape that decides everything later. Documents are long-lived and edited in place rather than replaced: a 200-page policy at roughly four chunks per page is about 800 chunks, and over a year it collects corrections a few at a time. Sources are mixed. Some systems emit change events, some are crawled on a schedule, and several deliver a freshly exported file every night whether or not a word of it moved.

Ingestion today runs at the document level. A change event or a crawl marks a document as dirty and the pipeline reprocesses it end to end: parse, chunk, embed, write. This is a sound design and the one most teams arrive at first. It is also trivially correct: when the job finishes, every chunk of that document reflects the current source, with no bookkeeping required to prove it.

The figures used throughout are illustrative teaching numbers rather than measurements. The ratios between them are what matter.

Where that design stops scaling

The strain comes from a mismatch of units. The unit of ingestion is a document; the unit of change is a paragraph. While those two are comparable in size the design holds comfortably. As documents grow long and edits stay small, the distance between them becomes the operating cost.

Concretely: a one-page correction to the 800-chunk policy sends all 800 chunks back through the embedding model. Twelve of them, the edited page and the two neighbours whose overlap crosses the change, carry new content. The other 788 are identical text, re-embedded because the pipeline has no way to say which twelve mattered. Work is proportional to document size rather than to edit size, and that ratio degrades as the library matures.

The consequences are not only financial. Large jobs occupy the pipeline, so an urgent correction queues behind a bulk re-export that changed nothing. Freshness is bounded by how long a whole document takes to process rather than by how small the edit was. And because one document becomes a sequence of writes across two indexes, there is a window during which a query can retrieve the new page alongside a summary derived from the old one.

Seen from outside the system, this is the complaint that usually starts the project: the assistant is still quoting last week’s rule. The cause is structural rather than operational. The pipeline cannot describe a change, so it replaces a document.

Five ways to know what changed

Replacement is what a system falls back on when it cannot name what changed. Every design that follows is an attempt to acquire that name, and they differ in how precise the name is, and in what the source has to give you before you can have it.

Precision is the governing variable. An approach that can only report “this document is different” will re-embed all 800 chunks, however small the correction was. An approach that can identify the paragraph re-embeds twelve. Cost, freshness, pipeline contention and how much of the library sits undisturbed are all downstream of that one difference in what the system is able to know.

• Trust the source’s revision stamp: the document says it changed, so process the document.
• Hash the file: confirm the bytes really differ before doing anything at all.
• Hash the chunks: compare the new revision against the active one and re-embed the difference.
• Diff the structure: use the source’s own block identities so an inserted page is an insert, not a rewrite.
• Let the source tell you: consume change events that name what was edited, and skip detection entirely.

Approach 1: Trust the source’s revision stamp

The simplest incremental update watches a watermark. Poll the source for documents whose revision number or modified timestamp is newer than the one you last processed, and re-ingest those documents. Nothing else in the library is touched.

Compared with a nightly rebuild this is already a large step: the unit of work is one document rather than five million. It also needs almost nothing from the source beyond a field that moves when content does, which is why it survives in environments where you control none of the upstream systems.

The limitation is in what a watermark can say. It reports that a document changed, never which part, so the whole document is reprocessed. It is also only as honest as the field itself. A re-export, a permissions edit or a metadata fix can move a timestamp without a single word of content changing, and you pay for the document anyway.

• Fits a source that exposes a reliable revision field and little else.
• Feels strained where documents are large or get re-exported routinely.

Approach 2: Hash the file before you believe the stamp

Keep a hash of each file you have processed. When a change event or a watermark says a document moved, hash the new copy first and compare. If the hashes match, the work stops there: no parsing, no chunking, no embedding.

This is a small addition that removes an entire category of waste: the nightly export that rewrites every file with a new timestamp, the sync job that re-uploads unchanged documents, the pipeline that replays its own backlog. In libraries fed by automated exports, a surprising share of change events are no-ops, and this is what catches them.

What it cannot do is look inside. A hash is a single yes-or-no for the whole file, so once the answer is yes the work is identical to approach 1: the full document is reprocessed for a one-word correction.

• Fits any pipeline fed by exports or syncs that touch files without changing them.
• Feels strained as the sole strategy: it filters noise, it does not reduce the cost of a real edit.

Approach 3: Hash the chunks and re-embed the difference

Now the comparison moves inside the document. Give every chunk an identity derived from its normalised content, keep those hashes with the active revision, and when a new revision arrives compare chunk by chunk. Only the chunks whose hashes differ are sent to the embedding model.

Normalisation is where this is won or lost. Collapse insignificant whitespace, but do not erase meaningful punctuation, numeric signs or table ordering: a changed minus sign is a changed meaning. Compare the exact text that will be embedded, including any heading or context prepended to the chunk, and record the parser, chunker and model versions alongside the hash, because a chunker upgrade invalidates every comparison you have stored.

Two things travel with the edit. A chunk whose overlap window crosses the change is affected even though its own text held still, and anything derived from the document, such as a section summary or extracted entities, depends on text you did not touch. The unit of work is the dependency closure, not the chunk.

A question that comes up immediately: if someone inserts a page, does everything below it mismatch? Not if the comparison is done properly. Identity is derived from content, not from position, so the comparison is a set difference between the old hashes and the new ones. The chunks after an insertion hash to exactly the same values they did before. They have moved, not changed. What needs updating for them is ordering metadata, a sequence number or a page reference, which is a write rather than an embedding call.

Two things do produce exactly that failure. Comparing chunk seven against chunk seven, position against position, reports the whole tail as changed, and that is simply the wrong comparison. The real hazard is boundary drift: a chunker that slices a continuous token stream into fixed-size pieces re-cuts every boundary after an insertion, so the tail chunks genuinely hold different text and genuinely need re-embedding. Nothing about hashing can rescue that, which is why chunk boundaries want to be anchored to something local, a paragraph, a section or a page, rather than to an offset in a stream.

Finally, be precise about the saving. Incremental embedding is not incremental parsing: if the source hands you a fresh PDF you still parse all 200 pages to find out what moved. What collapses is the model work: twelve chunks instead of eight hundred.

• Fits a living document that is corrected in small pieces while people keep reading it.
• Depends on: stable document identities, careful normalisation, and an explicit dependency graph.
• Feels strained when chunk boundaries are positional rather than anchored to structure (see approach 4).

Approach 4: Diff the structure, not the text

Hashing tells you whether two pieces of content match. It cannot tell you which two pieces are supposed to be compared, and that is the gap insertion exposes. Where chunk boundaries are derived from position in a stream, a new paragraph re-cuts every boundary behind it: the words are the same words, but they now fall into different chunks, so the hashes legitimately differ and the whole remainder of the document is re-embedded for an edit that added one paragraph.

The fix is to stop inferring structure and start using it. Content management systems, structured documents and well-formed exports all carry their own identities: block ids, section anchors, element keys. Align the two revisions on those identities first, then compare content within each matched block. Boundaries now belong to the content rather than to an offset, so an insertion reads as one new block, and the blocks around it are untouched because they were never re-cut.

Where the source gives you nothing to align on, you can approximate: match on headings, use a rolling hash over the content stream, or anchor on stable landmarks and diff between them. The point is the same either way: identify the unit of content before you compare it, so that moving text does not look like new text.

A PDF that gains a page is the case worth working through, because page numbers are both the citation coordinate and the thing that moves. Four decisions keep an insertion local. Keep the page out of the identity: a chunk identified as page 43, chunk 2 has its identity rewritten by any insertion above it, where a chunk identified by the hash of its own text does not. The page becomes a field on the record, patched with a metadata write rather than an embedding call. Strip page furniture before hashing, because extraction tends to bring running headers, footers and a printed “page 43 of 200” along with the body, and if that reaches the hash then every shifted page really has changed, by your own hand.

The third decision is to align the sequence rather than difference a set. Boilerplate repeats, with standard clauses and blank-page markers, so identical hashes collide and a set comparison cannot tell which copy is which. Running a longest-common-subsequence alignment over the ordered list of chunk hashes sorts the revision into three buckets instead: matched but shifted, which need their page metadata patched; inserted, which need parsing, embedding and indexing; and absent, which need tombstones.

The fourth decides whether any of that holds. Chunk on paragraphs or sections rather than on pages. A PDF re-exported from a word processor or a CMS after an insertion re-flows: text that sat on page 44 is now split across 44 and 45, so page-granular chunks genuinely differ and genuinely need re-embedding, while paragraph-granular chunks are untouched and have merely landed somewhere new. Record the page as a coordinate resolved from the active revision at citation time, and a cached answer can never quote a page number from a version that no longer exists.

• Fits sources with real structure: a CMS, a wiki, structured XML or Markdown in version control.
• Depends on: block identities that survive editing, and a matching step before any content comparison.
• Feels strained against flat scans and OCR output, where headings and landmarks are the only structure available to align on.

Approach 5: Let the source tell you what changed

The most precise answer to “what changed?” is the one the author already gave when they pressed save. Many sources can tell you directly: webhooks from a CMS, change feeds from a document store, change-data-capture from the database underneath. The event names the document and often the block, so there is nothing to detect.

Besides precision, this buys immediacy. There is no polling interval to wait out, so the edit becomes searchable in the time it takes to process one block, and the pipeline does no work at all while the library is quiet.

The catch is delivery. Events arrive twice, arrive out of order, or quietly do not arrive at all: a webhook times out, a consumer restarts mid-batch, a bulk import bypasses the event path entirely. Treat the event as a hint rather than the truth: make processing idempotent, order work by source revision, and keep a slower reconciliation pass that compares the source against what you published and repairs the difference.

• Fits a source you control, or one with a dependable change feed.
• Depends on: idempotent processing, revision ordering, and a reconciliation pass behind the events.
• Feels strained when bulk imports or migrations move content without emitting anything.

Whichever one you choose: publishing without a seam

All five approaches answer the same question and leave the same one open. However precisely you identified the change, you are now writing a handful of records into live indexes, and between the first write and the last, a reader can retrieve the new page alongside the old summary.

The protocol that closes this is the same in every case. Write the new records as a revision alongside the live ones rather than over them. When every write has landed, in the vector index, in the keyword index and in the metadata store, flip a manifest to name the new revision as active. Readers move from one complete version to the next, queries already in flight keep answering from the previous one, caches are invalidated by version rather than on a timer, and deletions are explicit tombstones so a removed section cannot linger as an active chunk.

This is the step that turns a cost optimisation into a correct system, and the one most often skipped, because nothing looks wrong without it until the day a half-updated answer reaches someone who acts on it.

• Applies to every approach above; none of them make it unnecessary.
• Depends on: a version manifest, idempotent writes addressed by content-derived identity, and tombstones for deletions.
• Adds: two live revisions side by side, and a publication protocol to operate and watch.

The five, side by side

Read the table on one axis: how precisely can this approach name what changed, and therefore how much work does an edit cause. The teaching numbers from the scenario sit behind it: a 200-page document at roughly four chunks per page, where one edited page and its overlapping neighbours come to twelve chunks.

Conclusion

Five ways to run an incremental update, separated by a single question: how precisely can this approach name what changed. Everything downstream (the size of the job, how quickly a correction surfaces, how much of the library sits undisturbed) is decided there.

They also stack more often than they compete. A pipeline can take events from the source, hash the file to discard the no-ops, align on block identities, and fall back to chunk hashing where the structure runs out. The question is usually how far along that chain a particular source lets you go, not which single technique is best.

A few shapes, and where each tends to land:

• A vendor system you cannot change, exposing a modified date and nothing else: approach 1 might fit, with approach 2 in front of it to absorb the re-exports.
• A nightly export that rewrites every file whether or not anything happened: approach 2 might fit, as the filter that makes everything after it affordable.
• A long policy document corrected in small pieces while people are reading it: approach 3 might fit.
• A CMS or a repository where authors insert and move sections: approach 4 might fit, because stable boundaries are what insertion threatens.
• A source you own, with a dependable change feed and someone to keep a reconciliation pass honest: approach 5 might fit.

Questions this raises