Skip to main content
Read a collection’s change feed to learn which documents or alerts changed. Handle each notice by reading the changed document again, then save your place so you can resume after a disconnect.

How it works

You keep documents (Records, in the API) in a collection (a Corpus). Each accepted text of a document is a Version, which never changes; the document points to its current one. The change feed lists, in order, a short notice (a change event) for each change: what happened and to which resource, but not the new content. Each read of the feed returns a cursor, a string marking your place in one collection. You send it on the next read. Quivr keeps notices for a retention window, 7 days by default. If the changes after your cursor are older than that, Quivr refuses the cursor, and you rebuild your copy from the list of all documents in the collection, the catalog. Your application reads changes after its cursor, updates its copy, then saves the new cursor: In words:
  1. Your app asks to start now and gets cursor A, with no past notices.
  2. It reads the changes after A, gets notices and cursor B, reads the changed documents again, then saves B.
  3. After a disconnect, it reads the changes after B. If B is within the retention window, it gets notices, some possibly seen before, and cursor C.
  4. If B is too old, Quivr answers that it expired, and the app rebuilds its copy from the catalog.

Prerequisites

  • A running Quivr, and QUIVR_API_URL and QUIVR_API_KEY in your shell. With the local stack, eval "$(make -s env)" sets them (Quickstart).
  • To follow an existing collection, a key with changes:read, content:read and access to that collection. Running this page also needs corpora:write, content:write and access to all collections, to create a collection and an article.
  • curl, jq and sed: command -v curl jq sed prints a path for each.
The requests below use idempotency keys: a request sent again with the same key returns its first result and adds no notice. To run the page again, give each run its own label, which the keys include:

Start following a collection

Create a collection for this page:
Read the feed without a cursor. Quivr returns no past notices and a cursor at the current end of the feed:
Save this cursor before anything else. It excludes changes already committed when you started, even when publication is lagging, and keeps writes that were still in flight. Starting now skips the documents already in the collection: to follow an existing collection, take this cursor first, build your initial copy from the catalog as in Build or rebuild your copy, then follow changes from this cursor. A cursor belongs to one collection, so an app that follows several keeps one cursor for each. Parallel imports publish a complete safe prefix. On a healthy database, publication takes at most the longest configured database transaction limit (30 seconds by default) plus two 100 ms publisher polls. Outside long-running or prepared transactions, or database unavailability, can delay it. A reader waits at its cursor until publication is safe; it never skips a late lower position. Operators can inspect publication status.

Read what changed

Add an article to the collection:
Then read the changes after your cursor. Quivr processes the article in the background, so its notices arrive over a few seconds; read again until you see record.enrichment_available:
Each item is a full change event; the jq filter above prints only its type and the documents to read again. One item looks like this:
  • Save next_cursor once you have handled the whole page, and send it as cursor on the next read. It can move forward even when a read returns no items, and stays the same when nothing new was committed.
  • has_more: true means more changes are waiting: read again at once. When it is false, you are caught up; wait as long as suits your app before the next read.
  • limit sets the page size, from 1 to 100 (the default).
  • Each item also carries its own cursor, to resume right after that item.
Each read looks up this collection’s notices directly. Changes in other collections do not add extra pages to read. The poll reference lists every parameter.

Update your copy

A notice tells you that a document changed, not how. For each document in a page of notices, read it once, however many notices name it, and act on what you read:
  1. If the read answers 404, your key can no longer reach the document: remove it from your copy.
  2. If withdrawn is true, remove its content from your copy, keeping only a marker if your app shows withdrawals. Withdrawn documents stay readable, so they do not answer 404.
  3. If current_version_id is absent, the document has no searchable text yet: keep what you have and wait for a later notice.
  4. Otherwise, read that Version and store its text.
Then save the page’s next_cursor, once these updates have succeeded. Read the article now:
The text is in the Version’s Parts, its named pieces such as a title or a body:
A correction is new text for the same document: the same collection, namespace and record_key. Send one:
It produces the same notices as the first text, for the same document. Once record.retrieval_ready arrives, the document points to the corrected Version; read until record.enrichment_available to see them all:
While a correction is still processing, or on hold (quarantined), the earlier Version stays current, so a read in between returns the earlier text. The Version reference lists every field.

Save your place and handle repeats

Each journal notice has one durable event_id and appears exactly once while moving forward through cursors within retention. Its order stays stable, and notices for the same document keep their order. Concurrent writes to different documents can appear in a different order from their commits. Reconnecting from an older cursor can deliver a notice again. To lose nothing:
  1. Read a page of notices.
  2. Update your copy.
  3. Save the updates, the event_id of each notice you handled, and next_cursor in one transaction.
  4. After a crash or a disconnect, read again from the saved cursor.
A crash before step 3 means you read that page again; reading documents again is harmless. Keep the handled event_ids for at least one retention window, and skip a notice you have already handled. Some notices, such as record.enrichment_available, can come back with the same event_id even later, after their first copy was deleted. An action outside your store, such as sending an email, cannot be made exactly-once by a stored event_id alone. If you crash after sending and before saving, the email goes twice; if you save first and crash before sending, it is lost. Instead, add the action to a durable queue in the same transaction as the cursor, and have the sender pass the event_id as an idempotency key to a service that accepts one. With a service that accepts none, a crash right after a send can still repeat it; recording each send as soon as it succeeds keeps that window small. A cursor stays valid while the first change after it is within the retention window: 7 days by default, set by the operator with change_retention (configuration); ask them if yours differs. An app that reads until has_more is false at least once per window keeps a valid cursor. An operator can also delete notices sooner (change_prune), which expires the cursors behind them.

Build or rebuild your copy

You build a copy from the catalog when you start following an existing collection, and rebuild it when Quivr refuses your cursor rather than skip changes silently: Neither is retryable, and both carry resync_url, where to start rebuilding. On the stream, both arrive as an HTTP error before the response starts; cursor_expired can also arrive as a stream_error frame once it has started. For any other error, check its retryable field: when it is true, as for 503 changes_unavailable, read again later from the same cursor; when it is false, fix the request (the key, the collection or the cursor) before you retry. This answer was captured by hand on a local stack whose change_retention was set to 1 second:
resync_url is relative to your API address; it is the catalog of the collection, /v0/records?corpus_id=…. To build or rebuild your copy:
  1. Start now: read GET /v0/changes?corpus_id=… without a cursor and keep its next_cursor. Take it before listing, so changes made during the listing reach you through the feed.
  2. List the catalog into a fresh copy, page by page: send each page’s next_page_cursor as page_cursor until it is absent. Replace your old copy only when the listing is complete.
  3. Read the changes after the cursor from step 1, and update your copy from them.
  4. Carry on as before. If a cursor fails again during these steps, start over at step 1.
The catalog lists every document of the collection your key can read, withdrawn ones included, up to 100 per page. This loop reads every page:
Then read the current Version of each document, as in Update your copy. A catalog page cursor is not a change cursor: each is refused where the other is expected. The listing is not a snapshot taken at one instant; step 3 brings the copy up to date. A rebuild restores the current state of your documents, not the notices you missed. For alerts, GET /v0/matches?subscription_id=… lists first matches and corrections that still match (keyword alerts); match.no_longer_matches and match.withdrawn store no Match, so those missed notices are gone. The catalog reference lists every parameter.

List by acceptance time

For a date-based feed, use GET /v0/records with order=accepted_at_desc. It lists Records by the time Quivr accepted their current Version, newest first; a correction moves a Record when the replacement Version becomes current. Set accepted_after for an inclusive start and accepted_before for an exclusive end. Both take RFC 3339 timestamps with an offset and at most 9 fractional-second digits. Convert days in your own time zone into these bounds; Quivr applies no time-zone rules. Use local midnight at the start of the day and local midnight at the start of the next day, computing each offset separately if daylight-saving time changes. Repeat the order and bounds when following next_page_cursor. Changing them returns 409 cursor_scope_changed. Newer arrivals do not shift later pages; pages remain independent reads, so reconcile concurrent corrections and readiness changes through the change feed. Reread the current Version on an invalidation and reapply your bounds. Withdrawn Records remain included and marked. Records without a current Version appear last in an unbounded date listing; either time bound excludes them. Use Count records, GET /v0/records/count, with the same Corpus and at least one time bound to get an exact count without listing pages. It uses the listing’s inclusion rules for withdrawn and undated Records, and needs the same content:read permission and Corpus scope. The count is an independent read, so it can change between listing requests. Exact queries have a two-second budget; 422 record_count_too_broad is nonretryable: narrow the time window. An unbounded count of more than 10,000 Records returns the stored catalog total with approximate: true, without scanning the corpus. Unbounded counts up to 10,000 are exact. Responses include observed_at and complete; complete: false means historical initialization has not finished, so a stored total is partial. The CLI exposes these reads as quivr records: --corpus selects the Corpus, --order accepted_at_desc selects newest first, --accepted-after and --accepted-before set the bounds, and --page-cursor continues one page at a time. It prints the API JSON; list pages include the next cursor when another page exists. Use --limit to select 1 to 100 Records per page. Use --count with the Corpus and optional bounds to count; omit --order, --page-cursor and --limit. Count responses contain count, observed_at, approximate and complete, without a continuation cursor. ID order remains the default for the rebuild procedure above.

Read document totals

GET /v0/corpora/{corpus_id}/stats returns running total and catalog_total with an observed_at snapshot time, under the same content:read permission and Corpus grant. total counts current Versions whose baseline is ready for word search, excluding quarantined, withdrawn Records and Records with a permanent withdrawal fence (a tombstone). catalog_total counts Record identities, including pending and withdrawn Records, matching the catalog listing. An archived Corpus is refused. undated_total and catalog_undated_total account for Records without a current acceptance time; they do not appear in the histogram. Add include=histogram,sources for both breakdowns, or choose either. Histogram entries use start, count and catalog_count; source entries also name namespace and connector_id. The statistics reference defines the full response. Histograms use current-Version acceptance time: a correction moves the document to its new bucket when it becomes current. resolution=day (default) groups UTC days; resolution=hour returns UTC hours. For local days, request hours and sum between local midnights converted to UTC. Only whole-hour timezone offsets are supported; half-hour and quarter-hour offsets cannot be reconstructed exactly. Compute each midnight separately when daylight-saving time changes. Bounds must fall on UTC hour boundaries; accepted_after is inclusive and accepted_before exclusive. History defaults to the full occupied date range, including documents older than a year. Each page covers at most 10,000 intervals. Missing buckets within the returned from/to interval mean zero as of that response; they say nothing about other intervals. Source pages list lifetime totals by Source Namespace and connector instance, including an explicit unknown connector for historical documents or public submissions without trusted connector attribution. source_limit defaults to 100, maximum 1,000. Bounds affect history only, not totals or sources. Follow histogram.next_page_cursor using histogram_cursor, or sources.next_page_cursor using sources_cursor, repeating all original options. A series omits its cursor on the last page. Changing scope, Corpus or options returns 409 cursor_scope_changed; malformed cursors return 422 invalid_cursor. Each response is internally consistent, but pages have independent observed_at times and can change under writes. Check complete before treating missing history as final: complete: false and approximate: true mark partial historical initialization. Operators enable it after upgrading all writers, following Initialize document totals.

What the notices mean

The type says what happened, and resource.kind and resource.id say to what. Every change to a document produces at least one notice with resource.kind record and that document’s ID. A resend of the current text under a new idempotency_key produces record.accepted but no new Version. Alert notices are match.created, match.corrected, match.no_longer_matches and match.withdrawn (how alerts work). They carry a monitoring object with the references of the catch: match_id, record_id, the alert’s subscription_id, the owner your app gave the alert, and delivery_id. Their event_id is the same as the event_id in the body of the webhook Quivr sends for that catch, which is also its webhook-id header. An app can follow alerts in the feed instead of, or as well as, through webhooks, and skip one copy by its event_id. Other notices concern alerts and collectors: saved_query.*, subscription.*, connector.*, operation.updated. A Saved Query (an alert’s search) or a Subscription (the alert that uses it) that watches several collections gets a notice in each of them. Skip types your app does not use, including types added later.

Stream changes live

GET /v0/changes/stream sends the same notices as they happen, over one long HTTP response in the Server-Sent Events format. It replaces polling, not the rest of this page. To see a notice arrive, withdraw the article: The stream waits between reads with no notices for your collection, even while other collections change. The default wait is 250 ms, set by the operator with change_stream_poll (configuration). Pages with more notices for your collection are sent without that wait.
Then open the stream from your saved cursor. The stream stays open, so this command stops after 3 seconds; curl reports that stop as exit code 28. Run it again if the notice is not there yet:
The raw stream in stream.txt holds one frame per notice, trimmed here:
Only change and checkpoint frames carry an id, the cursor after them. Save the last one you handled; comments and errors leave it unchanged. When the connection drops, or after a stream_error whose retryable is true, reconnect with that cursor in the Last-Event-ID header; it wins over a cursor query parameter. Without either, the stream starts now with a checkpoint frame. The stream reference has the details. Open the stream from your server. A browser’s EventSource cannot send the Authorization header, and an API key must never go in a URL. Following the feed creates nothing in Quivr. The collection and the withdrawn article of this page can stay; each run of the page adds one of each.

Next

Keyword alerts

Get notified when a new article matches a query, by webhook or through the feed.

Core concepts

What the stages behind each notice mean: acceptance, word search, meaning search.