LLM document processing with a human in the loop

Document intake went from manual keying to reviewer-approved extraction, and every decision stays auditable.

AI & data / Software architecture / Hands-on engineering

Problem

Staff re-keyed information from incoming documents by hand. The process was slow and error-prone, and it was hard to audit why a document was handled a certain way.

Constraints

Wrong answers are expensive, so the model cannot have the last word
Sensitive documents: data stays in the client’s cloud account
Every automated decision must be explainable after the fact

Architecture & approach

An event-driven pipeline: documents land in storage, get OCR’d and chunked, and an LLM extracts fields using retrieval over the client’s policy corpus. A confidence score decides whether a result flows straight through or goes to a reviewer queue. Reviewer corrections become evaluation data.

Fig. 304-2Document pipeline with confidence routing Critical path

Key decisions

Fine-tune a model or ground it with retrieval? 304-3.1

RejectedFine-tune on historical documentsBakes today’s policy into weights, so every policy change means retraining.
ChosenRetrieval over the live policy corpusPolicy updates take effect immediately, and citations make answers auditable.

How much should run without a person? 304-3.2

RejectedFully automatedFastest, until the first confident mistake.
ChosenConfidence-routed human reviewAutomation where it is safe, people where it is not. Corrections feed evaluation.
RejectedSuggestions onlySafe, but saves little time.

Outcome

Manual keying replaced by reviewer-approved extraction
Every automated decision traceable to its source passages
Reviewer corrections captured as evaluation data

Technology

LLM APIs, Retrieval-augmented generation, PostgreSQL + vector search, AWS S3 · Lambda, Python, React review UI

Lessons & tradeoffs

The confidence threshold is a product decision, not a model setting.
Evaluation data is the moat. Design the review UI to produce it.