RAG Chatbots: Building AI That Knows Your Business

Most people who ask for a "chatbot trained on our documents" do not want a trained model. They want a system that reads the right page of the right document and answers from it. That is retrieval-augmented generation, usually shortened to RAG, and it is a much cheaper and more controllable thing than fine-tuning a model on your files.

The language model is the easy part. The hard part is the pipeline in front of it: getting clean text out of your documents, cutting it into pieces that make sense, finding the right pieces for a question, and knowing when the honest answer is "I can't find that." Most disappointing RAG projects fail in that front half and then get blamed on the model.

Start with the documents, not the model

Open ten of your real documents before you choose any tooling. Product manuals exported from Word with headers and footers repeated on every page. Scanned PDFs that are actually images. Spreadsheets pasted into a wiki as screenshots. Policy documents where the exception to a rule lives three pages away from the rule.

Whatever extraction you pick will mangle some of this, and the chatbot will quietly inherit every mistake. A repeated footer that says "Confidential, version 3.2" will show up in a lot of retrieved chunks and push out useful text. Strip it. If a table matters, convert it to something a model can read, such as one row per line with the column names repeated, instead of letting it collapse into a stream of numbers.

Scanned files need OCR first, and OCR quality sets a ceiling on everything after it. It is worth reading a sample of the OCR output by eye. Bad character recognition in part numbers or amounts is the sort of error nobody notices until a customer does.

Chunking is where quality is decided

A chunk is the unit you store and retrieve. Too small and a chunk loses the context that gives it meaning. "Refunds are processed within 14 days" is useless if the sentence that says which refunds was in the previous chunk. Too large and each chunk contains several topics, so the embedding becomes a blur and retrieval gets vague.

A reasonable starting point is a few hundred tokens per chunk with a little overlap, but the better rule is to split on the structure the author already gave you. Headings, numbered clauses and FAQ entries are natural boundaries. Cut on those first, and only fall back to a fixed size when a section is too long.

def split_by_heading(sections, max_chars=1800):
    """sections: list of (heading, body) pairs from the parsed document."""
    chunks = []
    for heading, body in sections:
        text = body.strip()
        if len(text) <= max_chars:
            chunks.append(f"{heading}\n{text}")
            continue
        # long section: split on paragraphs, keep the heading on every piece
        buf = ""
        for para in text.split("\n\n"):
            if len(buf) + len(para) > max_chars and buf:
                chunks.append(f"{heading}\n{buf.strip()}")
                buf = ""
            buf += para + "\n\n"
        if buf.strip():
            chunks.append(f"{heading}\n{buf.strip()}")
    return chunks

The detail that matters there is repeating the heading on every piece. A chunk that starts with "Returns policy, international orders" is far easier to retrieve and far easier for the model to interpret than the same paragraph on its own.

Retrieval: vectors alone are not enough

Embedding search is good at meaning and surprisingly bad at exact strings. Ask for "error E-4021" or a part number and a pure vector search may return chunks about errors in general. Keyword search (BM25) has the opposite strengths. Running both and merging the results, usually called hybrid search, fixes a large share of the "it can't find something I know is in there" complaints for very little effort.

The next improvement is a reranker. You retrieve, say, thirty candidates cheaply, then let a slower model that reads the question and each candidate together pick the best five. It costs some latency and tends to be worth it.

For storage, you do not need a dedicated vector database to start. If you already run PostgreSQL, the pgvector extension handles a few hundred thousand chunks without trouble. FAISS is fine for a single-machine prototype. Choose the boring option that your team can operate.

Permissions are part of retrieval

This one gets skipped in demos and causes real trouble in production. If the HR folder and the sales folder are indexed together, a question from a sales person can pull up HR text. The model cannot be trusted to hide what it was handed, so the filter has to happen before the model sees anything. Store the access group with each chunk and filter on it in the retrieval query, using the identity of the person asking.

Make it say "I don't know"

A system that always answers will eventually answer wrongly and sound sure about it. Two habits help. First, tell the model to answer only from the supplied passages and to say so when they do not contain the answer. Second, look at the retrieval scores: when the best match is weak, skip the model entirely and return a plain message that points the person to a human or a search page.

Always show the source. A link or page number next to each answer lets people check it in ten seconds, and it changes how they use the tool. They stop treating it as an oracle and start treating it as a fast route into the documents, which is what it is.

Measure it, or you are guessing

The most useful hour in the whole project is the one spent writing down fifty to a hundred real questions along with the document that answers each. This is your test set. It turns arguments about quality into numbers.

Measure retrieval and answers separately. Retrieval first: for each question, did the correct source appear in the top five results?

def recall_at_k(test_cases, search, k=5):
    hits = 0
    for case in test_cases:
        results = search(case["question"], top_k=k)
        if any(r.doc_id == case["expected_doc"] for r in results):
            hits += 1
    return hits / len(test_cases)

If recall is low, no amount of prompt tuning will rescue the answers, so fix chunking and search first. Once recall is decent, read the answers themselves and mark each as correct, partly correct, wrong, or correctly declined. Rerun the whole set whenever you change chunk size, the embedding model or the prompt. Changes that feel like improvements have a habit of breaking questions that used to work.

Keeping it current

Documents change. Plan the update path on day one: re-index a file when it changes, remove chunks when it is deleted, and record the document version with each chunk so answers can say which edition they came from. A chatbot quoting last year's price list is worse than no chatbot.

Also assume that some documents contain text written to manipulate the model, such as a web page that says "ignore previous instructions". If you index content from outside your organisation, treat it as untrusted input and keep the model's available actions narrow.

When RAG is the wrong tool

If the answers live in structured data, such as order status, stock levels or account balances, a database query is the right tool and a language model is only there to turn the question into that query. If your documents number in the dozens and rarely change, a well-organised search page may serve people better than a chat interface. And if nobody can agree on what the correct answer is, no retrieval system will fix that. The cleanest RAG projects start where the knowledge already exists in writing and people are tired of hunting through it.