Skip to main content
An ingestion plugin decides how each Version’s text is cut into passages (segments) and turned into vectors, and encodes search queries with the same model. Quivr keeps the index, the registry of vector spaces, rebuilds, access control and withdrawals. This page starts from the SDK’s sample, which embeds by hashing words, so you can replace its model with yours.

Prerequisites

  • Go 1.26 or later, a quivr checkout in $QUIVR_REPO, and the quivr command on your PATH (Build your first plugin shows how).
  • To move existing Corpora onto the plugin, an API key with the projections:rebuild and operations:read actions.

Steps

1

Start from the sample

2

Declare your vector spaces

A vector space is one model with its dimensions and distance. Your plugin owns every space it declares, so each space id is the plugin id or starts with it. Edit quivr-plugin.yaml:
quivr-plugin.yaml
Bump a space’s version whenever its vectors change: a new model, new weights, a new input template. Quivr treats acme.embedder.base@2 as a new space and never mixes its vectors with those of version 1. Declare several spaces to compare models: a deployment serves search from one and indexes the others for evaluation.
3

Implement the two operations

main.go registers one type with two methods:
  • SegmentAndEmbed receives the Version’s text Parts and the spaces to fill. It returns segments in reading order, each with the Part key, start and end offsets in Unicode code points, and one vector per requested space. It may add a LexicalText, indexed for keyword search while excerpts keep the source text, and a Provenance map.
  • EmbedQuery returns the vector of one search query in one space. A search waits for it, within query_timeout_ms.
Replace hashWords with a call to your model. Both methods must be deterministic: rebuilds reuse stored segments and vectors, and Quivr refuses a different answer for the same Version. When a Version arrives, Quivr first asks for the segments alone, with no spaces, so the article is searchable by keyword even while your model is down; it asks for the vectors afterwards. Return the same segments both times.Return quivrplugin.RetryableIngestError when your model is unavailable: the Version waits and is retried. Return TerminalIngestError for content you can never handle: the Version is blocked with ingestion_refused, and its diagnostic shows your error’s code and message, so say in plain words why, such as no_indexable_text.A call that runs past timeout_ms (at most 5 minutes) is cancelled and retried, and after three such calls the Version’s enrichment stops. If your model is slow on long Versions, keep the vectors you already computed and return a retryable error before the deadline: the next call resumes instead of starting over.
4

Certify it

The Contract Runner sends the normative article and your fixtures in fixtures/, checks every segment and vector as the engine will, replays each fixture, compares the segments-only answer with the full one, and encodes the fixture queries in every space. Set a fixture’s expect.segments to pin your segmentation.
5

Pin it

Several ingestion plugins can run beside core.ingest; source media routing chooses one per Version. To use yours alone, pin it in the configuration of every api, worker and migrate process, and say which space serves search:
QUIVR_CONFIG
At startup Quivr registers each space with its owner. It refuses a space another plugin already registered (space_owner_conflict), and a registered space whose model, dimensions or metric changed without a new version (space_changed).
6

Move a Corpus onto it

New Corpora use the plugin at once. An existing Corpus keeps the spaces it was built with, and stays searchable, until you rebuild it:
The rebuild asks each Version’s routed plugin for every Version without vectors in the new spaces, then switches search over at once. Follow it with GET /v0/operations/{operation_id}.

Check it worked

The response lists the Corpus’s spaces with their owner, their role (served or evaluation) and how many segments hold a vector in each. Search hits name the space that answered in vector_space_id.

Troubleshooting