Write an ingestion plugin

An ingestion plugin decides how Quivr cuts each article into segments and turns them into vectors, and encodes search queries the same way. The core keeps what must stay safe and uniform: the index, the vector space registry, rebuilds, authorization and withdrawal. The contract is the ingestion Contribution; this page takes a plugin from its manifest to searchable Records.

Declare its spaces

A vector space is one embedding model with its dimensions and distance. The plugin owns every space it declares, so each space id is the plugin id or starts with it:

id: acme.embedder
version: 0.1.0
compatibility: {engine: ">=0.1.0 <0.2.0", plugin_api: ">=0.6.0 <0.7.0"}
contributions:
  ingestion:
    spaces:
      acme.embedder.base:
        version: "1"
        model: acme/text-embed-base
        dimensions: 768
        metric: cosine
        indexes: [text]
        query_modalities: [text]
      acme.embedder.large:
        version: "1"
        model: acme/text-embed-large
        dimensions: 1024
        metric: cosine
        indexes: [text]
        query_modalities: [text]

Bump a space’s version whenever its vectors change: a new model, new weights or a new input template. Quivr treats acme.embedder.base@2 as a new space, never mixing its vectors with those of version 1.

Implement it

With the Go SDK, one type answers both operations:

SegmentAndEmbed returns the segments of the Version’s text Parts, in reading order, each with Unicode code point offsets into its Part and one vector per requested space. It may add a LexicalText (folded or stemmed words, indexed in a separate keyword field while excerpts keep the source text) and a Provenance map.
EmbedQuery returns the vector of one query in one space.

Both must be deterministic: rebuilds reuse the stored segments and vectors, and a different answer for the same Version is refused. Return a retryable error when a backend is down, a terminal one for content you can never handle. sdks/go/examples/hash-embedder is a complete example with two spaces.

Certify it

quivr plugin test --report report.json .

The Contract Runner sends the normative article and your own ingestion fixtures (fixtures/*.json with a top-level ingestion), checks every segment and vector as the engine will, replays each fixture, encodes the fixture queries in every space, and checks that invalid requests are refused and that no declared secret appears in an answer or your plugin’s output. Set the fixture’s expect.segments to pin your segmentation.

Pin it

Add it to plugins in the QUIVR_CONFIG file of every api, worker and migrate process, and say which space answers search:

{"plugins": [
  {"manifest": "/etc/quivr/plugins/acme-embedder/quivr-plugin.yaml",
   "endpoint": "http://127.0.0.1:9910",
   "spaces": {"acme.embedder.base": "served", "acme.embedder.large": "evaluation"}}
]}

Served and evaluation. Exactly one space is served. Evaluation spaces are embedded on the same segments and indexed, so you can compare them, but search never uses them. With one declared space, spaces may be omitted.
One plugin. A deployment pins one ingestion plugin (ingestion_conflict); compare embedders as several spaces of it.
Startup. The spaces are registered with their owner. Startup refuses a space another plugin already registered (space_owner_conflict) and a registered space whose model, dimensions or metric changed without a new version (space_changed). An unreachable plugin does not prevent startup.

Move a Corpus onto the plugin

Each Corpus keeps the spaces it was built with. A Corpus created before the pin stays on its current space, searchable as before, until you rebuild it (POST /v0/corpora/{corpus_id}/rebuilds, see the HTTP API reference). The rebuild asks the plugin for every Version that has no stored vectors in the new spaces, then switches search over at once. From then on new Records go through the plugin, and search encodes queries with it. GET /v0/corpora/{corpus_id}/vector-spaces shows the Corpus’s spaces, their owner and role, and how many segments hold a vector in each.

When something fails

What happens	Effect
The plugin is down or answers a retryable error	The Version waits and is retried with backoff; search on a plugin space answers 503
The plugin answers a terminal error, or segments or vectors the checks refuse	The Version is blocked as `ingestion_refused`; a rebuild fails with `ingestion_refused`
`embed_query` answers a terminal error	The search is refused with 422 `unsupported_search`
The plugin is unpinned while Corpora use its spaces	Their new Versions wait, and semantic search on them answers 422, until it is pinned again or they are rebuilt

Using Quivr

Writing plugins

Contributing to Quivr

Write an ingestion plugin

Declare its spaces

Implement it

Certify it

Pin it

Move a Corpus onto the plugin

When something fails

​Declare its spaces

​Implement it

​Certify it

​Pin it

​Move a Corpus onto the plugin

​When something fails

Declare its spaces

Implement it

Certify it

Pin it

Move a Corpus onto the plugin

When something fails