What an internal document assistant actually is, how the pieces fit together, and where it is allowed to run.
The One Correction
— the word "train" sends this project down an expensive road it does not need
Fine-tuning — "training the model"
Changes how the model writes — tone, format, structure
Does not reliably make it remember facts
Every document change means doing it again
Expensive, slow, needs ML expertise
Cannot tell you which document an answer came from
Right tool for teaching a style. Wrong tool for teaching facts.
Retrieval (RAG) — what you actually want
Documents are indexed and searched at the moment of the question
The model reads the retrieved passages and answers from them
Update a document → re-index that one file → done
Cheap to change, no ML training involved
Every answer can cite its source document
In insurance, the citation is not a nice-to-have. It is the difference between usable and not.
Almost everyone who says "train it on our documents" means the right-hand column. Worth knowing before anyone quotes for fine-tuning.
so the thing being built looks like this
How It Works
— five steps; the first two happen once, the last three happen on every question
1
PrepareOnce, per document
Collect & parse
Policy wordings, circulars, procedures, past reports — text pulled out of each file.
OCR if scanned
Scanned PDFs are pictures, not text. Without OCR they are invisible to the system.
Split into chunks
Broken into passages of a few hundred words, so retrieval returns a page, not a book.
This unglamorous step decides whether any of it works.
2
IndexOnce, then incremental
Each chunk is converted into a numeric vector that represents its meaning
Vectors are stored in a vector database
Similar meaning ends up close together, even with different wording
This is the semantic search part. "Water damage" finds "burst pipe" — no keyword match required.
3
AskTwo different inputs — worth not confusing them
The knowledge base
Your internal documents. Indexed once, searched on every question by everyone.
The attachment
A photo or PDF the user adds to this one question. Read in the moment, not indexed.
A shelf of reference books, versus the letter in your hand. The system needs both, differently.
4
Retrieve
The question becomes a vector too
Nearest passages are pulled from the index
Top handful are passed forward — not the whole library
Permissions applied here: a user only retrieves what they may see
Most of these projects fail here — on retrieval quality, not on which vendor was chosen.
5
AnswerThe only generative step
The model writes an answer using only the retrieved passages
Each answer carries its source documents
If nothing relevant was found, it should say so rather than invent
The model supplies the language. Your documents supply the facts.
Note what is absent: no training, no model of your own, no ML team. Steps 1 and 2 are data plumbing; steps 4 and 5 are a search and a paragraph.
and all of it has to physically run somewhere
Where It Runs
— the platform is the last decision, and it falls out of sensitivity and volume together
Two things decide this — how sensitive the content is, and how much of it you push through the model
Not sensitive · low volume
Consumer cloud — the pilot
A Custom GPT, a Claude Project, or NotebookLM. Upload files, no code, working in an afternoon.
Fastest way to find out if answers are good enough
Flat monthly seat cost — fine at small scale
Caps on volume; retrieval quality is a black box
Check the data terms on your specific tier
Use it to test the idea, not to hold the real corpus.
Sensitive · volume varies
Enterprise cloud — the usual answer
Azure OpenAI, AWS Bedrock, or an enterprise tier from OpenAI, Anthropic or Google.
Contractual guarantee your data is not trained on
You choose the region the data sits in
Pay per token — no capital outlay, but the bill scales with every question asked
Logging and access control you can show an auditor
Right answer while volume is unknown. You are renting, not buying.
Sensitive, or high steady volume
Your own hardware
An open-weights model — Llama, Qwen, Mistral — on GPUs you own. Chosen for privacy, economics, or both.
Data never leaves the building. Simplest answer to a client's question
Cost flips from per-token to fixed — GPU, power, cooling, maintenance. Above a certain steady volume this is the cheaper shape, and it stops scaling with usage
Still no training — you run someone else's model, with retrieval on top
Answering from retrieved passages is a gentle workload — far easier for a smaller model than tool-using agents
Capable, generally a step behind the frontier
It pays off on utilisation, not ownership. An idle GPU is pure cost.
These are not a ladder from casual to paranoid. Renting per token wins while volume is low or unpredictable; owning capacity wins once usage is high and steady, or once the content simply cannot leave. Both are ordinary engineering choices — the mistake is picking either one before you know your volume. And note that self-hosted still does not mean custom-built: that confusion is what turns a weeks-long project into a months-long one.
Decide In This Order
What content goes in — and is it text, or scans?
How sensitive is it, at its worst single document?
How much of it — a few questions a day, or constant heavy use?
Where is it allowed to live — contractually and under PDPA?
Then pick the platform. It is a consequence, not a starting point.
Step 3 is the one people skip, and it is what decides whether renting or owning is cheaper.
The One-Sentence Version
You are not training a model — you are building a searchable index of your own documents and letting a language model answer from what it retrieves, which means a new document costs a re-index rather than a retrain, the answers can cite where they came from, and the only genuinely hard decisions are how sensitive the content is and how good the retrieval turns out to be.
Days, not months
Cites sources
Re-index, not retrain
You choose where
Watch For These
Scanned PDFs need OCR, or they simply do not exist to the system
Insist on citations from day one; an answer without a source is unusable here
Permissions must mirror who may already read each document
Pilot on non-sensitive files before choosing any platform
Measure real usage for a month before buying hardware — the crossover is a number, not an opinion
"Agent" usually means multi-step actions — this is a knowledge assistant, and that is cheaper