Skip to main content
License: MIT An open-source engine that turns continuous content streams into search and monitoring. Quivr V2 ingests content durably, makes it searchable within seconds, enriches it in the background and lets you follow a topic over time. The core stays generic: formats, AI models and business rules belong in plugins, so you can adapt Quivr to your domain without forking the platform.
Status: evaluation stage. The API is v0 and may change without notice. Do not run it in production yet.

​
Why Quivr V2

  • Durable before fast. A write is acknowledged only once it is committed; outages delay processing, they never lose or silently fail content.
  • Idempotent everywhere. Every write carries an idempotency key. Replays return the same result, changed replays are conflicts, and corrections create new immutable Versions instead of overwriting history.
  • Useful early, richer later. Text is lexically searchable as soon as it is segmented; embeddings and other enrichments arrive afterwards without blocking it.
  • Rebuildable indexes. PostgreSQL and S3 hold the canonical data; the search index is a projection that can be rebuilt from durable artifacts.
  • Honest search. Every hit is rehydrated from canonical storage and re-authorized. A dependency outage returns an error, never an empty “success”.
  • Generic core, extensible edges. Connectors, normalizers, enrichers, retrievers and delivery channels are plugin capabilities, not core code.

​
Architecture at a glance

A single quivr binary provides the api, worker and migrate commands.

​
Quickstart

Requirements (Linux; the local harness targets Linux hosts): Go 1.27.1, Docker with Compose v2, Python 3 with venv, Node.js 22+ and jq. The first run downloads pinned images and the E5 model (~1 GB).
make verify runs every feature’s acceptance suite and one assembled monitoring journey on an isolated stack. It removes only its own project, even after a failure or Ctrl+C. It then prints the path of a report.md that names any failed step, the pinned versions and the dependency inventory. Linux x86_64 is the only supported platform; see the remaining limits. make dev prints the API address and the path of a generated config.json holding throwaway local keys. Then follow Your first search: create a Corpus, add a text Record and search it, with commands that make verify replays against a real stack. For a browser UI over the same API, run make demo and open http://127.0.0.1:5183 (see quivr-search/).

​
What works today

  • Corpora with scoped API keys per Organization, action and Corpus.
  • Durable, idempotent ingestion: inline text, bounded batches with per-entry outcomes, verified uploads (presigned PUT + checksum confirm), structured Manifests, extensions and relations.
  • Corrections and withdrawals with immutable Versions and fenced withdrawn Records.
  • Search: lexical, semantic and hybrid, with canonical rehydration and access rechecks on every hit, optionally within chosen Source Namespaces (filtered before ranking).
  • Change feed through polling and resumable SSE, plus catalog resync after cursor expiry.
  • Saved Queries and Subscriptions, pinned and versioned; enabled Subscriptions turn newly searchable Versions into unique Matches (/v0/matches), each with a Delivery. Matching is decided by a pinned alert-rule plugin (the subscription Contribution), batched per article. Both can be renamed without a new Version.
  • Keyword alerts through the first-party plugin plugins/alerts, pinned by default in the local stack:
    • queries such as "Airbus" AND (grève OR strike) NOT sport, with exact phrases, “any of”, “none of” and grouping;
    • case, accents and punctuation are ignored, and words match whole;
    • metadata filters such as source:wire or author:"Jane Doe" (names mapped in the plugin configuration), and a filter alone is a valid alert;
    • each Match’s evidence names the matched terms and the Parts where they matched (guide).
    • the browser demo’s Alertes tab writes these alerts and shows what each one caught, live (quivr-search/).
  • Described alerts through the same plugin: a plain-language description such as “Labour strikes at ports and harbours”, judged by TypeSafe’s Jev classifier, so rephrased and translated articles alert too:
    • one classifier call per article covers all described alerts;
    • an alert can be limited to chosen sources, whose other articles are never sent;
    • the Match evidence carries the classifier’s score;
    • they are off without a TypeSafe key, because article text is sent to TypeSafe (guide);
    • the browser demo’s Alertes tab offers them next to keyword alerts when the deployment has a classifier, and shows each caught article’s score (quivr-search/).
  • Subscription previews (POST /v0/subscription-previews): before saving an alert, see which of the most recent articles it would have caught, judged by the same plugin with the same rules. A preview saves nothing and sends nothing, and it judges at most 50 articles; the demo’s alert form shows it as you type.
  • Subscription owners: an application can create a Subscription for one of its end users (an opaque owner such as user-123) or a global one, see the owner on the Subscription, its Matches, webhooks and change feed to route each alert, and list a user’s active Subscriptions with GET /v0/subscriptions?owner=….
  • Correction and withdrawal notices for alerted Records: a correction that still matches gets a linked successor Match (match.corrected), one that no longer matches gets match.no_longer_matches without a new Match, and a withdrawal gets match.withdrawn. Earlier Matches stay readable.
  • Signed webhook delivery (Standard Webhooks) to deployment-configured destinations, with append-only attempt history, jittered exponential retries within a bounded delivery window, exhaustion, and no new attempt once a Subscription is disabled. The worker exposes delivery metrics on its probe listener (/metrics).
  • Projection rebuilds from durable artifacts as recoverable Operations, with cancel and rerun.
  • Typed retrieval mappings per Corpus (PUT /v0/corpora/{id}/retrieval): logical fields pointing into source data take effect only when their rebuilt generation is validated and activated.
  • Connector Instances: scheduled pull acquisition into a Corpus, with write-only deposited credentials and health, through the same ingestion path as pushed content. Delivered kinds: rss (RSS and Atom feeds), m365_mail (Microsoft 365 mailboxes) and x_list (first-party plugin plugins/x-list), which polls an X list: edits become corrections, deleted or protected posts are withdrawn, and health shows daily reads (guide). In webhook mode, X posts arrive in near real time through Filtered Stream webhooks relayed by the core to the plugin, with polling as the fallback (guide). The deployment credential_key is optional. Without it, credential deposits are refused with 503 credentials_unavailable, and everything else works. GET /v0/connector-kinds publishes each enabled kind’s config and credential JSON Schemas, PUT /v0/connectors/{id}/schedule changes the polling interval, POST /v0/connectors/{id}/runs checks a source again now, and validation errors name the offending field as a JSON Pointer.
  • Sources page in the web app (quivr-search, Sources tab): paste a site or feed address and the web app finds its RSS or Atom feed (refusing private addresses), or add a suggested feed in one click from DEMO_FEED_SUGGESTIONS. Each source shows its health and last article, and can be paused, resumed or removed; a failing one can be checked again at once (Réessayer). Other kinds keep forms generated from their schemas, so new kinds need no UI change (guide).
  • Live feed page in the web app (Veille tab): everything entering the demo Corpus, newest first, with source, time, title and excerpt. New items arrive over SSE, which the app’s server relays from the change feed, and can be filtered by source (guide).
  • Operational metrics and correlated logs on each process’s private probe listener (/metrics, Prometheus text, bounded labels):
    • API: accepted commands and the pending-ingestion backlog;
    • worker: processing outcomes, time from acceptance to searchable, and delivery attempts and durations.
    JSON logs link request, Receipt, Record and Version IDs (harness).
  • Retrieval measurement with a frozen workload (make measure), and search quality on public French and English evaluation sets or a private set, nightly (make eval, guide).
  • Plugin Protocol v0 contract (contracts/plugins/v0/) and quivr plugin inspect, which validates a quivr-plugin.yaml and reports its compatibility, Contributions, schemas, secrets and limits.
  • Plugin registry: the plugins pinned at startup are recorded in the database on first start, with the active Pipeline Plan saying which plugin serves each role (a media type, an alert rule, a connector kind). An operator key with plugins:admin, which no Organization key gets, reads them with GET /v0/admin/plugins and GET /v0/admin/plugins/plan; a later start with other pins only logs a warning. Quivr never starts a plugin process.
  • External normalizer: the startup configuration pins one plugin and routes Blob media types to its normalizer. A Blob of a routed type, ingested by reference, becomes searchable through the plugin’s Parts, and its Version shows provenance.normalization. An unavailable plugin is retried and never blocks the API or other ingestion. A plugin error, invalid output or exhausted retries quarantine the Version with a structured diagnostic, and an optional text route falls back to the built-in text path (walkthrough).
  • Plugin-owned extension namespaces: the pinned plugin’s declared namespaces are registered at startup beside the built-in ones. Its normalizer’s extensions are validated against their schemas and published on the Version, clients cannot write them (422 extension_namespace_owned), and retrieval mappings can map them into search.
  • Python Plugin SDK (sdks/python/) with quivr plugin init, which scaffolds a Markdown normalizer, and quivr plugin dev, which runs it locally, checks its discovery digest and replays a fixture through the engine’s Manifest validation, without a Quivr stack (SDK guide).
  • Plugin Contract Runner (quivr plugin test), which certifies a normalizer over the public protocol, launched from its manifest or at --endpoint <url>. It runs health, discovery, normative and plugin fixtures, deterministic replay, the declared deadline, terminal errors for invalid requests, and compatibility ranges. It judges output with the engine’s own validation: Manifest rules, response size, input-Blob-only Blob Parts and declared namespaces. It writes a JSON report with --report, and CI publishes one for the quivr plugin init template.
  • Searchable PDFs through the reference plugin plugins/pdf-text (pypdf, BSD-3-Clause). An application/pdf Blob becomes one body Part per page with text, and a phrase is found on its page’s Part. Blank or scanned pages give warnings; encrypted or damaged PDFs are quarantined with a diagnostic naming the plugin. make dev pins it by default; there is no OCR (walkthrough).
  • quivr search from the command line: set QUIVR_API_URL and QUIVR_API_KEY, then quivr search --corpus <corpus_id> "query" prints ranked hits with their excerpt and Record / Version / Part provenance, or the unchanged API response with --json. Failures exit with one code per class (rejected key or scope, invalid request, unreachable server). It uses a Go client generated from the contract (package client) and never touches the stack’s storage (walkthrough).
  • AI agents search and cite Quivr over MCP: quivr mcp --profile read serves an agent on stdio with three read-only tools. The agent can list the Corpora its key reaches, search them, and read a hit’s Record Version and Manifest, keeping Record, Version, Part and exact excerpt offsets to cite. --profile ingest adds text to a Corpus and follows its Ingestion Receipt until it is searchable; retries never duplicate, and no tool deletes. The API key alone decides access (Connect an AI agent).
  • A guide to writing a normalizer: scaffold, run, certify, pin, ingest and observe your own plugin (Write a normalizer).
  • Source collectors as plugins, in Go (Plugin API 0.3): the connector contract (fetch a page after an opaque checkpoint, check_credential, classified errors), a Go Plugin SDK (sdks/go) that redacts credentials, and quivr plugin test checks that pages resume from their checkpoint and that no credential leaks. The core does not call connector plugins yet.
  • Segmentation and embedding as a plugin (Plugin API 0.6, the ingestion Contribution): a pinned plugin declares the vector spaces it owns, one served and others for evaluation, cuts each article into segments with a vector per space and an optional keyword text, and encodes queries for its spaces. A Corpus moves onto it with a rebuild; GET /v0/corpora/{id}/vector-spaces shows each space’s owner, role and coverage (Write an ingestion plugin). The built-in E5 path stays the default.

​
What comes next

  • Alerts that catch rephrased or translated articles with Quivr’s own vectors, without an external classifier.
  • Filtering on typed field mappings (filter roles are validated and stored today).
  • Reprocessing quarantined Versions.
The contract already describes some of these routes; the ones not implemented yet are listed here, not in “What works today”.

​
Documentation

New to Quivr? What Quivr can do explains it in plain words. Then start from the page for what you want to do: These start pages are generated from docs/inventory.toml, so every living page appears on the one for its reader. The authoritative request and response shapes are in the OpenAPI contract.

​
Repository layout

​
Contributing

  • Read AGENTS.md and CONTEXT.md first; use the domain vocabulary in code and docs.
  • Change the contract in contracts/http/v0/openapi.yaml, then run make generate.
  • Run make check before pushing and keep make verify green; add tests with every behaviour change.
  • Declare every new living doc page in docs/inventory.toml with one line giving its audience and kind (the file’s header explains both), then run make start-pages; make docs fails on an undeclared page, a stale start page, a broken relative link or a missing repository path, and names the fix.
  • Show API requests in guides as runnable blocks, which make verify replays.
  • Never edit an accepted ADR or a dated document under docs/dated/: supersede it with a new one (ADR 0004 (docs/adr/0004-documentation-rules-are-enforced-by-ci-only.md)); make docs compares them with where your branch forked from origin/main.
  • Pull request titles follow Commitizen conventions, for example feat(ingestion): accept record versions.
  • Keep customer-specific formats and rules out of the core; they belong in plugins.

​
License

MIT — see LICENSE.