Building a Knowledge Base from Scratch, EP02

Most RAG tutorials skip straight to the answer. You upload documents, ask a question, and the chat window streams back a confident paragraph. The retrieval step, the actual mechanism that decides which passages reach the model, stays a black box. When the answer is wrong, you have no idea whether the model misread good context or the retrieval fed it garbage.

This episode of my knowledge-base series does the opposite: I build the smallest possible library, then stop at retrieval and examine exactly what comes back. No answer generation, no prompt engineering. Just: store, retrieve, inspect.

The experiment design

The library: three Markdown support-FAQ documents for a fictional small store: shipping policy, returns & warranty, payments & invoices. Each file is 1-2KB. Built in the Model Studio console (standard edition, file connector), ten minutes of clicking. Notably, there is no CLI command for this part; the bl CLI's knowledge group exposes exactly one command, retrieve. Creation stays in the console, retrieval goes to the terminal. That split turns out to be a reasonable design boundary: creation is low-frequency and benefits from human judgment; retrieval is the high-frequency operation you actually want to script.

The documents contain one deliberate structural property. Cash-on-delivery is mentioned in two of them: the shipping policy defines which regions support COD (major eastern and southern cities only; remote regions like Xinjiang and Tibet are excluded), while the payments document defines COD as a payment method carrying a 3 RMB collection fee. Neither file alone answers "does COD cost extra, and can I even use it?" completely.

This mirrors every real organization I've worked with. Ops writes the shipping rules. Finance writes the payment rules. Nobody owns the seams. The question I wanted answered: does retrieval handle the seams, or does the user still have to know the file layout?

What retrieval actually returns

The command:

bl knowledge retrieve --index-id zj0knmrbye --query "新疆可以货到付款吗"

(Does COD work for Xinjiang? The query is Chinese because the documents are Chinese; the CLI doesn't care.)

The response is JSON, and its structure tells you more than any demo screenshot:

{
  "data": {
    "total": 3,
    "nodes": [
      {
        "score": 0.8299149870872498,
        "metadata": {
          "doc_name": "01-发货与物流政策",
          "content": "……货到付款仅支持华东、华南主要城市。偏远地区（新疆、西藏等）不支持货到付款，请选择在线支付。货到付款订单需额外支付 3 元代收服务费。……"
        }
      },
      {
        "score": 0.48945897817611694,
        "metadata": { "doc_name": "03-支付与发票说明" }
      }
    ]
  }
}

The metadata is a builder's field day. doc_name gives you provenance, which file the passage came from, which means any downstream system can display sources, group results by document, or audit what the model was fed. content is the verbatim passage, not a paraphrase. And the raw JSON also carries sub-scores (_rc_v_score, _q_score, _rc_t_score) that hint at multi-route retrieval underneath: vector scores and query-relevance scores computed separately and fused. That's the next-but-one episode's material.

Five queries, three lessons

I ran five queries across the typical shapes of support questions. The full table:

Query	Hits	What it proved
Does COD work for Xinjiang?	Shipping 0.83 + Payments 0.49	Cross-file recall
Free shipping threshold?	Shipping 0.68	Factual hit
My humidifier arrived broken	Returns 0.75	Semantic match, zero keyword overlap
Can coupons stack?	Payments 0.54 (only hit)	Below-threshold filtering
Does COD cost extra?	Shipping 0.82 + Payments 0.77	Both files, both high

Lesson one: semantic retrieval is real and load-bearing. The humidifier query shares no keywords with any document heading. It hit at 0.75, returning 「电器类商品（如加湿器、小夜灯）提供 1 年保修，非人为损坏免费维修或换新」: "electrical products (humidifiers, night lights) carry a 1-year warranty, free repair or replacement for non-artificial damage." The query says "broken"; the passage says "non-artificial damage." If you're building a support system, this means your users' colloquial phrasing is not a failure mode you need to engineer around with synonym lists.

Lesson two: the system refuses to pad. The coupon query returned exactly one result out of a three-document library: 「新人券：首单满 59 减 10，仅限新注册会员，不可与其他优惠叠加」, "the newcomer coupon cannot be combined with other offers," which is precisely what "can coupons stack?" was asking about. Results below the relevance threshold were dropped rather than included with a shrug. For downstream integration this has a concrete implication: the nodes array's length is not constant. Code that assumes "always top-3" will eventually meet a query that returns top-1.

Lesson three: the seams dissolve. The COD-fee query, the one whose answer was deliberately split across two files, returned both files at 0.82 and 0.77 simultaneously: shipping with 「货到付款仅支持华东、华南主要城市（上海、杭州、南京、广州、深圳、厦门等）」, payments with 「需额外支付 3 元代收服务费」. The user never needed to know the fee lived in the payments doc and the region rules in the shipping doc. This is, in one response, the entire argument for runtime retrieval over manual document assembly. It's also the honest counterpoint to the "RAG is dead, just compile a wiki" discourse of the past months: a personal folder of Markdown files is a fine system when one person owns all the seams. The moment knowledge is produced by different people in different documents, something has to close the gaps at question time.

What I'm deliberately not showing

No answer generation. The retrieved passages are raw material: verbatim policy text with no tone, no greeting, no phrasing. Pasting them into a customer-facing reply would be a mistake, and I'd rather show the boundary explicitly than imply the pipeline ends here. The "phrase it as a human answer" step is one more command, but the interesting questions come before it: how to make retrieval sharper (multi-route search and reranking, EP04), and how to let the model decide what to look up (Agentic RAG, EP05). Foundations first.

Also worth flagging honestly: three kilobyte-scale documents never trigger chunking; each returns as a single whole slice. That's the best case and it won't survive contact with a fifty-page PDF. Chunking strategy is the entire next episode.

One measurement caveat for anyone reproducing this: score is a within-query ranking signal, nothing more. Across my five queries, 0.54 was the sole correct answer to one question while 0.49 was a bystander in another. Anyone building a filter on absolute score thresholds will ship a bug that only manifests in production.

Reproduce it

npm install -g bailian-cli
bl auth login --api-key sk-xxxxx

Write three Markdown files from your own domain: policies, docs, FAQs. Build the library in the console (ten minutes, no code), copy the index ID, run one retrieval. Then ask the question I designed this whole episode around: one whose answer lives in two different files.

---

All five retrievals ran for real on the Bailian CLI against a Model Studio knowledge base. Commands and full JSON responses are kept in the project repo. CLI install: Bailian CLI docs. API key: get one free.