How it works
You keep documents (Records, in the API) in a collection (a Corpus). Each accepted text of a document is a Version, which never changes; the document points to its current one. The change feed lists, in order, a short notice (a change event) for each change: what happened and to which resource, but not the new content. Each read of the feed returns a cursor, a string marking your place in one collection. You send it on the next read. Quivr keeps notices for a retention window, 7 days by default. If the changes after your cursor are older than that, Quivr refuses the cursor, and you rebuild your copy from the list of all documents in the collection, the catalog. Your application reads changes after its cursor, updates its copy, then saves the new cursor: In words:- Your app asks to start now and gets cursor A, with no past notices.
- It reads the changes after A, gets notices and cursor B, reads the changed documents again, then saves B.
- After a disconnect, it reads the changes after B. If B is within the retention window, it gets notices, some possibly seen before, and cursor C.
- If B is too old, Quivr answers that it expired, and the app rebuilds its copy from the catalog.
Prerequisites
- A running Quivr, and
QUIVR_API_URLandQUIVR_API_KEYin your shell. With the local stack,eval "$(make -s env)"sets them (Quickstart). - To follow an existing collection, a key with
changes:read,content:readand access to that collection. Running this page also needscorpora:write,content:writeand access to all collections, to create a collection and an article. curl,jqandsed:command -v curl jq sedprints a path for each.
Start following a collection
Create a collection for this page:Read what changed
Add an article to the collection:record.enrichment_available:
jq filter above prints only its type and the documents to read again. One item looks like this:
- Save
next_cursoronce you have handled the whole page, and send it ascursoron the next read. It can move forward even when a read returns no items, and stays the same when nothing new was committed. has_more: truemeans more changes are waiting: read again at once. When it isfalse, you are caught up; wait as long as suits your app before the next read.limitsets the page size, from 1 to 100 (the default).- Each item also carries its own
cursor, to resume right after that item.
Update your copy
A notice tells you that a document changed, not how. For each document in a page of notices, read it once, however many notices name it, and act on what you read:- If the read answers
404, your key can no longer reach the document: remove it from your copy. - If
withdrawnistrue, remove its content from your copy, keeping only a marker if your app shows withdrawals. Withdrawn documents stay readable, so they do not answer404. - If
current_version_idis absent, the document has no searchable text yet: keep what you have and wait for a later notice. - Otherwise, read that Version and store its text.
next_cursor, once these updates have succeeded. Read the article now:
namespace and record_key. Send one:
record.retrieval_ready arrives, the document points to the corrected Version; read until record.enrichment_available to see them all:
Save your place and handle repeats
Each journal notice has one durableevent_id and appears exactly once while moving forward through cursors within retention. Its order stays stable, and notices for the same document keep their order. Concurrent writes to different documents can appear in a different order from their commits. Reconnecting from an older cursor can deliver a notice again. To lose nothing:
- Read a page of notices.
- Update your copy.
- Save the updates, the
event_idof each notice you handled, andnext_cursorin one transaction. - After a crash or a disconnect, read again from the saved cursor.
event_ids for at least one retention window, and skip a notice you have already handled. Some notices, such as record.enrichment_available, can come back with the same event_id even later, after their first copy was deleted.
An action outside your store, such as sending an email, cannot be made exactly-once by a stored event_id alone. If you crash after sending and before saving, the email goes twice; if you save first and crash before sending, it is lost. Instead, add the action to a durable queue in the same transaction as the cursor, and have the sender pass the event_id as an idempotency key to a service that accepts one. With a service that accepts none, a crash right after a send can still repeat it; recording each send as soon as it succeeds keeps that window small.
A cursor stays valid while the first change after it is within the retention window: 7 days by default, set by the operator with change_retention (configuration); ask them if yours differs. An app that reads until has_more is false at least once per window keeps a valid cursor. An operator can also delete notices sooner (change_prune), which expires the cursors behind them.
Build or rebuild your copy
You build a copy from the catalog when you start following an existing collection, and rebuild it when Quivr refuses your cursor rather than skip changes silently:
Neither is retryable, and both carry
resync_url, where to start rebuilding. On the stream, both arrive as an HTTP error before the response starts; cursor_expired can also arrive as a stream_error frame once it has started. For any other error, check its retryable field: when it is true, as for 503 changes_unavailable, read again later from the same cursor; when it is false, fix the request (the key, the collection or the cursor) before you retry. This answer was captured by hand on a local stack whose change_retention was set to 1 second:
resync_url is relative to your API address; it is the catalog of the collection, /v0/records?corpus_id=…. To build or rebuild your copy:
- Start now: read
GET /v0/changes?corpus_id=…without a cursor and keep itsnext_cursor. Take it before listing, so changes made during the listing reach you through the feed. - List the catalog into a fresh copy, page by page: send each page’s
next_page_cursoraspage_cursoruntil it is absent. Replace your old copy only when the listing is complete. - Read the changes after the cursor from step 1, and update your copy from them.
- Carry on as before. If a cursor fails again during these steps, start over at step 1.
GET /v0/matches?subscription_id=… lists first matches and corrections that still match (keyword alerts); match.no_longer_matches and match.withdrawn store no Match, so those missed notices are gone. The catalog reference lists every parameter.
List by acceptance time
For a date-based feed, useGET /v0/records with order=accepted_at_desc.
It lists Records by the time Quivr accepted their current Version, newest first;
a correction moves a Record when the replacement Version becomes current.
Set accepted_after for an inclusive start and accepted_before for an exclusive end.
Both take RFC 3339 timestamps with an offset and at most 9 fractional-second digits.
Convert days in your own time zone
into these bounds; Quivr applies no time-zone rules. Use local midnight at the
start of the day and local midnight at the start of the next day, computing each
offset separately if daylight-saving time changes.
Repeat the order and bounds when following next_page_cursor. Changing them
returns 409 cursor_scope_changed. Newer arrivals do not shift later pages;
pages remain independent reads, so reconcile concurrent corrections and readiness
changes through the change feed. Reread the current Version on an invalidation
and reapply your bounds. Withdrawn Records remain included and marked.
Records without a current Version appear last in an unbounded date listing;
either time bound excludes them.
Use Count records, GET /v0/records/count,
with the same Corpus and at least one time bound to get an exact count without listing pages.
It uses the listing’s inclusion rules for withdrawn and undated Records, and
needs the same content:read permission and Corpus scope. The count is an
independent read, so it can change between listing requests. Exact queries have a
two-second budget; 422 record_count_too_broad is nonretryable: narrow the time window.
An unbounded count of more than 10,000 Records returns the stored catalog total
with approximate: true, without scanning the corpus. Unbounded counts up to 10,000 are exact.
Responses include observed_at and complete; complete: false means historical
initialization has not finished, so a stored total is partial.
The CLI exposes these reads as quivr records:
--corpus selects the Corpus, --order accepted_at_desc selects newest first,
--accepted-after and --accepted-before set the bounds, and --page-cursor
continues one page at a time. It prints the API JSON; list pages include the next cursor when another page exists.
Use --limit to select 1 to 100 Records per page. Use --count with the Corpus
and optional bounds to count; omit --order, --page-cursor and --limit.
Count responses contain count, observed_at, approximate and complete, without a continuation cursor.
ID order remains the default for the rebuild procedure above.
Read document totals
GET /v0/corpora/{corpus_id}/stats returns running total and catalog_total
with an observed_at snapshot time, under the same content:read permission and Corpus grant.
total counts current Versions whose baseline is ready for word search, excluding quarantined,
withdrawn Records and Records with a permanent withdrawal fence (a tombstone). catalog_total counts Record identities,
including pending and withdrawn Records, matching the catalog listing. An archived
Corpus is refused. undated_total and catalog_undated_total account for Records
without a current acceptance time; they do not appear in the histogram.
Add include=histogram,sources for both breakdowns, or choose either.
Histogram entries use start, count and catalog_count; source entries also
name namespace and connector_id. The statistics reference defines the full response. Histograms
use current-Version acceptance time: a correction moves the document to its new
bucket when it becomes current. resolution=day (default) groups UTC days;
resolution=hour returns UTC hours. For local days, request hours and sum between
local midnights converted to UTC. Only whole-hour timezone offsets are supported;
half-hour and quarter-hour offsets cannot be reconstructed exactly. Compute each
midnight separately when daylight-saving time changes. Bounds must fall on UTC
hour boundaries; accepted_after is inclusive and accepted_before exclusive.
History defaults to the full occupied date range, including documents older than
a year. Each page covers at most 10,000 intervals. Missing buckets within the
returned from/to interval mean zero as of that response; they say nothing about
other intervals. Source pages list lifetime totals by Source Namespace and connector
instance, including an explicit unknown connector for historical documents or
public submissions without trusted connector attribution. source_limit defaults
to 100, maximum 1,000. Bounds affect history only, not totals or sources.
Follow histogram.next_page_cursor using histogram_cursor, or
sources.next_page_cursor using sources_cursor, repeating all original options. A series omits its cursor on
the last page. Changing scope, Corpus or
options returns 409 cursor_scope_changed; malformed cursors return
422 invalid_cursor. Each response is internally consistent, but pages have
independent observed_at times and can change under writes. Check complete
before treating missing history as final: complete: false and approximate: true
mark partial historical initialization. Operators enable it after upgrading all
writers, following Initialize document totals.
What the notices mean
Thetype says what happened, and resource.kind and resource.id say to what. Every change to a document produces at least one notice with resource.kind record and that document’s ID.
A resend of the current text under a new
idempotency_key produces record.accepted but no new Version.
Alert notices are match.created, match.corrected, match.no_longer_matches and match.withdrawn (how alerts work). They carry a monitoring object with the references of the catch: match_id, record_id, the alert’s subscription_id, the owner your app gave the alert, and delivery_id. Their event_id is the same as the event_id in the body of the webhook Quivr sends for that catch, which is also its webhook-id header. An app can follow alerts in the feed instead of, or as well as, through webhooks, and skip one copy by its event_id.
Other notices concern alerts and collectors: saved_query.*, subscription.*, connector.*, operation.updated. A Saved Query (an alert’s search) or a Subscription (the alert that uses it) that watches several collections gets a notice in each of them. Skip types your app does not use, including types added later.
Stream changes live
GET /v0/changes/stream sends the same notices as they happen, over one long HTTP response in the Server-Sent Events format. It replaces polling, not the rest of this page. To see a notice arrive, withdraw the article:
The stream waits between reads with no notices for your collection, even while other collections change. The default wait is 250 ms, set by the operator with change_stream_poll (configuration). Pages with more notices for your collection are sent without that wait.
stream.txt holds one frame per notice, trimmed here:
Only
change and checkpoint frames carry an id, the cursor after them. Save the last one you handled; comments and errors leave it unchanged. When the connection drops, or after a stream_error whose retryable is true, reconnect with that cursor in the Last-Event-ID header; it wins over a cursor query parameter. Without either, the stream starts now with a checkpoint frame. The stream reference has the details.
Open the stream from your server. A browser’s EventSource cannot send the Authorization header, and an API key must never go in a URL.
Following the feed creates nothing in Quivr. The collection and the withdrawn article of this page can stay; each run of the page adds one of each.
Next
Keyword alerts
Get notified when a new article matches a query, by webhook or through the feed.
Core concepts
What the stages behind each notice mean: acceptance, word search, meaning search.