API walkthrough

​

Corpora

​

Ingesting text

• Reusing a source_revision with different content keeps history and reports a conflict.
• source_position is an optional decimal string of 1 to 1000 digits; leading zeros are normalized.
• A single command is limited to 1 MiB.
• Structured Manifests (kind: "manifest"), extensions and relations are accepted and preserved; see ingestion contracts (docs/dated/design/quivr-v2-ingestion-contracts.md).
• POST /v0/records/withdrawals withdraws a Record; withdrawn Records are fenced so a late or stale submission cannot resurrect them.

​

Batches

​

Uploads

​

External normalizers

"plugin": {
  "manifest": "/etc/quivr/plugins/markdown/quivr-plugin.yaml",
  "endpoint": "http://127.0.0.1:9900",
  "configuration": {"max_sections": 32},
  "routes": [{"media_type": "text/markdown", "mode": "required"}]
}

• Startup checks. quivr api and quivr worker validate the pin before starting and refuse to start with the list of issues if any check fails. They check:
  • the manifest schema and its engine and Plugin API ranges;
  • configuration against the manifest’s configuration schema;
  • that the endpoint is an http(s) URL;
  • that every route names a media type the normalizer declares, and that no media type is routed twice;
  • that an optional route names a text/* media type.
  A route’s mode is required (the default) or optional; see failure handling below. The plugin itself is never contacted at startup, so an unreachable plugin does not affect /healthz or /readyz.
• Acceptance. A routed Blob is accepted by reference without being read. The Version identity is the digest of the submitted input: the verified Blob, its checksum and media type, plus the submitted extensions. Replaying the same input converges on the same Version.
• Invocation. In the worker, before publication, the engine:
  • checks that the plugin’s discovery document matches the pinned manifest digest;
  • sends the invocation context with a short-lived signed GET URL to the input Blob, never its bytes;
  • waits at most the declared timeout, capped at 2 minutes;
  • reads at most the declared max_response_bytes, capped at 16 MiB.
• Output checks. The output goes through the same validation as a kind: "manifest" submission, plus the declared max_parts. The stored Manifest must fit in 2 MiB, like any canonical object. Blob Parts may reference only the input Blob. Extensions, top-level and on Parts, must use a namespace and schema version the plugin declares, with valid data (undeclared_namespace, undeclared_schema_version, invalid_extension); otherwise nothing is recorded and the Receipt is blocked with normalizer_invalid_output.
• Plugin-owned namespaces. At startup the pinned plugin’s declared namespaces are registered beside the built-in ones; a namespace not prefixed by the plugin id or clashing with a built-in one refuses startup. Valid top-level extensions are published on the Version beside the submitted ones (Part extensions stay in the Manifest), and retrieval mappings may point at /extensions/{namespace}/.... A client submission, single or batch, that writes a plugin-owned namespace returns 422 extension_namespace_owned.
• Publication. The validated Manifest is stored once per Version, so re-running the step converges on it. It is then published and searchable like any Manifest. Rebuilds and retrieval generations read the stored Manifest and never call the plugin again.
• Provenance. GET /v0/records/{id}/versions/{version_id} shows provenance.normalization: plugin_id, plugin_version, plugin_api, contribution, invocation_id, idempotency_key and input_sha256. producer keeps naming the acquirer, and source_blob_ids keeps the input Blob. Clients cannot submit normalization.

• Unavailable plugin (connection failure, a 5xx answer without an error envelope, or a discovery document that does not match the pinned manifest): the step retries with backoff, without limit, and never quarantines. The Receipt stays pending with the retryable diagnostic plugin_unavailable, and the Version is published once the plugin is back.
• Retryable plugin error or timeout: retried until the manifest’s retry.max_attempts (capped at 5) invocations have failed this way, then quarantined with normalizer_retries_exhausted or normalizer_timeout.
• Terminal plugin error or invalid output (schema, malformed or duplicate Parts, a Blob Part that is not the input Blob, an undeclared extension namespace, too many Parts, a response over the size bound): quarantined at once with normalizer_failed or normalizer_invalid_output.
• Quarantine publishes the Version with only its submitted input Blob Part, never the plugin’s output. Its availability is quarantined, and it is announced by a record.quarantined change event. The Version read lists the reason in diagnostics:
  {"code": "normalizer_invalid_output", "message": "invalid normalizer output: …",
   "retryable": false, "plugin": "acme.markdown", "contribution": "normalizer",
   "invocation_id": "inv_…"}
  
  The input reference, the reason and the invocation are kept for a later reprocessing, which is not available yet.
• Optional routes ("mode": "optional", text/* only): instead of quarantining, the Version is published through the built-in text path and stays searchable. Its provenance.normalization names the failed invocation and carries fallback: {code, message}, and the failure is listed in diagnostics. An unavailable plugin is still retried rather than bypassed.
• Divergent output for the same idempotency key (normalizer_conflict): the first recorded output is kept and published. The conflict is listed in the Version’s diagnostics with the divergent invocation.
• Skipped normalization. A withdrawn Record, or a Version superseded by a newer accepted revision before it was normalized, never calls the plugin. A withdrawal resolves the Receipt as conflict, and a superseded Version is quarantined with normalization_superseded. If a route is removed while work is pending, a text/* Blob takes the built-in text path; any other media type is quarantined with normalizer_unrouted.

​

PDF documents

• The published Manifest has one body Part page-<n> per page with text, and a source Blob Part that references the PDF. A search hit names its page through part_key, for example page-2.
• The Version shows provenance.normalization.plugin_id: "pdf-text" with its version, and extensions["pdf-text.document"] holds page_count and text_pages.
• Pages without text, such as scans, are skipped and listed in an empty_pages warning. There is no OCR.
• An encrypted or damaged PDF is quarantined with a normalizer_failed diagnostic that names pdf-text.

​

Processing: segmentation and embeddings

​

Search

• Modes are lexical, semantic and hybrid. Defaults: hybrid, profile default (balanced is its deprecated name until engine 0.2.0), 10 results; 50 maximum. GET /v0/search/profiles lists profiles; others return 422 unsupported_profile. A pinned retrieval plugin ranks under its own profiles.
• The built-in default profile (version balanced.e5-token-windows.v1) accepts non-empty queries of at most 256 tokens (8,192 code points on the wire), without truncation; a longer query is 422 query_too_long, whose message names the limit. CRLF/CR become LF and surrounding whitespace is trimmed; case, accents and language are kept.
• Hybrid uses alpha 0.5, relative score fusion and title/body weights of 2/1.
• Every requested Corpus must be authorized. PostgreSQL selects the logical generation and physical routing; rehydration re-reads the S3 bytes, validates excerpts and rechecks access, current Version, quarantine and withdrawal.
• Excerpt coordinates are Unicode code points. No raw score, physical collection name or vector is exposed. A dependency outage returns 503, never an empty success.
• filter.source_namespaces narrows to those Source Namespaces before ranking; a Corpus projected before it existed returns 422 source_filter_unavailable until rebuilt.

​

From the command line

export QUIVR_API_URL=http://127.0.0.1:<api_port>
export QUIVR_API_KEY=<key with content:read and search:query>
quivr search --corpus <corpus_id> --mode lexical --limit 5 "eclipse"

1 hit (profile default, version balanced.e5-token-windows.v1)

1. record <record_id>  version <version_id>  part body  [0,18)
   Éclipse 🌞 à Paris.

​

Changes, catalog and monitoring

• GET /v0/changes (polling) and GET /v0/changes/stream (resumable SSE) expose committed changes per Organization behind opaque cursors.
• GET /v0/records?corpus_id=… traverses the authorized Record catalog; when a change cursor expires, use it as the resync_url to resynchronize. A Version read (GET /v0/records/{id}/versions/{version_id}) gives accepted_at, when Quivr accepted it, so a client rebuilt from the catalog can still date what arrived.
• /v0/saved-queries and /v0/subscriptions create pinned, versioned Saved Queries and activate, disable or re-enable Subscriptions (re-enabling does not backfill the pause).
• Edit either by committing a new Version (POST …/{id}/versions). Subscriptions keep the Saved Query Version they pin until their own new Version, which judges only later changes; earlier Matches keep their Versions, readable through …/versions/{version_id}. POST …/{id}/rename changes the name without a Version. POST /v0/subscriptions/{id}/delete stops evaluation and deliveries for good, history kept; POST /v0/saved-queries/{id}/delete works once no Subscription uses the Saved Query.
• Give a Subscription an optional owner at creation, an opaque reference to one of your application’s end users (for example "owner": "user-123"); without it the Subscription is global to the Organization. The owner never changes and is echoed on the Subscription, its Versions, its Matches, its webhooks (references.owner) and its change-feed events (monitoring.owner), so you can route each alert to its user. GET /v0/subscriptions?owner=user-123 (or ?owner=none for global ones) pages the active Subscriptions your key can see; per-user limits or quotas belong in your application, which can count with this listing.
• Enabled Subscriptions evaluate Record Versions that become searchable or enriched after their activation boundary. Each positive evaluation creates one unique Match with a pending Delivery, an immutable match.created notice and a change-feed event. Read them through GET /v0/matches, GET /v0/matches/{match_id} and GET /v0/deliveries/{delivery_id}. The evaluator is the subscription Contribution of a plugin pinned at startup (plugins in the configuration), named by plugin_id and version; a Saved Query expression or evaluator configuration its schemas refuse is 422 invalid_expression or invalid_subscription_configuration. A plugin outage delays alerts, never skips them.
• Preview a Subscription before creating it: POST /v0/subscription-previews (an inline definition or an existing Saved Query Version, the evaluator, limit 1 to 50) runs the evaluator on the newest eligible Record Versions and returns what it would match, with evidence. It writes nothing; each Record Version costs one plugin call. The first-party alerts plugin ({"plugin_id": "alerts", "version": "0.2.0"}, pinned in the local stack) takes an expression such as {"kind": "keywords", "match": {"all": [{"term": "Airbus"}, {"not": {"term": "sport"}}]}}. Its Match evidence names the matched terms and their Parts (Keyword alerts). With a TypeSafe key it also takes {"kind": "described", "description": "Labour strikes at ports and harbours"}, which a classifier judges and whose evidence carries its score (Described alerts). A plugin pin’s optional kinds lists the alert kinds the installation accepts, for example ["keywords"]; a Saved Query of another kind is 422 invalid_expression.
• The worker POSTs each notice to the Subscription’s destination, signed with Standard Webhooks headers (webhook-id is the notice event_id). A 2xx response marks the Delivery delivered. Network errors, timeouts, 408, 429 and 5xx are retried with jittered exponential backoff (1 s doubling to 5 min, or a valid Retry-After on 429/503) within a 24-hour window; other responses, or a failure after the window, mark it exhausted. Retries resend the same event_id and body bytes, so receivers deduplicate on webhook-id. GET /v0/deliveries/{delivery_id} shows state, admission, last_error and, while a retry is scheduled, next_attempt_at; GET /v0/deliveries/{delivery_id}/attempts pages the append-only attempt history. Disabling the Subscription stops further attempts.
• When an alerted Record changes, the Subscription receives a linked follow-up notice (reference-only, like match.created, with the same feed and webhook identity):
  • a correction that still matches creates a new Match with previous_match_id and a match.corrected notice carrying every match.created reference plus previous_match_id, so it can be acted on alone;
  • a correction that no longer matches sends match.no_longer_matches, whose match_id is the earlier Match and record_version_id the correction; no Match is created. An evaluator failure never sends it;
  • a withdrawal sends match.withdrawn for the latest Match. Search stops returning the Record at once; the notice follows asynchronously and is delivered even though the Record is withdrawn. A Subscription disabled at that point gets the notice too. It stays pending with no attempt until POST /v0/subscriptions/{id}/enable, then it is delivered: its delivery window starts at the re-enable.
  A match.created or match.corrected not yet delivered when a later correction notice exists for the same Subscription and Record is not sent any more, nor is a match.no_longer_matches once a later match.corrected shows the Record matches again: its Delivery stays pending with admission superseded. The earlier Match stays readable through GET /v0/matches/{match_id}.

​

Connectors

POST /v0/connectors
Authorization: Bearer <key with connectors:write>
Content-Type: application/json

{"idempotency_key":"x-watchlist-1","corpus_id":"<corpus_id>","source_namespace":"x-watchlist","kind":"x_list","config":{"list_id":"1234567890123456789"},"credential":{"secret":{"bearer_token":"<X app bearer token>"}}}

PUT /v0/connectors/{connector_id}/schedule
Authorization: Bearer <key with connectors:write>
Content-Type: application/json

{"interval_seconds":600}

​

Verification

​

Retrieval measurement