Embeddings and retrieval
嵌入向量与检索
Anything that turns on meaning rather than characters runs on embeddings: related pages, cross-lingual search, composer references and the visitor question box. One vector per block, computed on publish, stored in Firestore.
Retrieval
The bigram search index handles keyword queries within one language. Everything else that involves meaning rather than characters runs on embeddings: related pages, cross-lingual search, composer references and the visitor question box. This page describes the arrangement.
A block is the chunk
Retrieval systems spend effort splitting documents into chunks. A block stream is already split. Each block is a semantic unit with a stable id, so a block is the unit that gets a vector, a hit can cite a page and a block, and a link can deep-link to the block on the page. A site of a few hundred pages is a few thousand vectors.
Computed on publish
Vectors are computed when a page is published, for the blocks the change summary says have changed, alongside the search projection. Drafts are never embedded. A page-level vector summarises the whole document for related-page queries.
Storage
There is no separate vector database. Each vector lives on its own Firestore document with a declared vector index and is queried with the store's nearest-neighbour search. That covers tens of thousands of vectors. If a client ever needs more, a dedicated store becomes another adapter behind the same interface.
Provider
The default adapter calls a small multilingual embedding model. Cost is negligible next to the composer. A fake adapter with deterministic vectors exists for tests and offline work, and the site runs without any embeddings service at all, on the bigram index alone.
What it powers
- related pages: page-level nearest neighbours at the foot of an article, replacing the guess of "adjacent in the list"
- cross-lingual search: an English query finds a body written in Chinese; the find panel runs the bigram index first and fuses in semantic hits when keywords return little
- composer references: pages that mention the same programme, people or places are handed to the composer as references for terminology and links, never as sources of fact
- the question box: a visitor's question is answered from published excerpts with block-level citations, behind the cost guardrails described on the next page
Rebuilding
After enabling embeddings on an existing site, or changing the embedding model, one admin call recomputes every projection and vector from the published revisions. Both are derived data and can always be rebuilt.