object_storage_archive connector to import individual members of
.tar.gz and .zip objects under an S3-compatible prefix. Each matching member
becomes a raw source Blob, the immutable bytes Quivr normalizes into a document.
Before you start
You need a running Quivr with theconnector.object_storage_archive plugin
pinned. Version 1.1 requires Plugin API 0.19; earlier engines should retain
the earlier plugin version. External processes require engine call signing;
local and packaged launchers provision the signing ring automatically. Local stacks and the packaged worker include it. For an external
plugin process, pin its manifest and endpoint.
Create a Corpus, the collection that will hold the
imported documents. Your API key needs connectors:write, connectors:read
and access to that Corpus. A storage reader needs ListBucket on the prefix
and GetObject on its archives. Temporary credentials also need their session
token. A normalizer must support the configured media type: for XML, pin a
normalizer with the matching XML route before importing.
Keep the source objects immutable. Archives are visited in lexicographic
object-key order. New keys must sort after the highest archive key already selected by the connector, including an in-progress archive. Use a
separate Connector Instance, a scheduled source reader, to import earlier
keys or a different historical range.
Choose the member identity
A Record Key groups revisions of the same document. A Source Position is its non-negative numeric revision number; Quivr keeps the highest position current, even if an older version arrives later. By default, the key is the URL-decoded basename and the position is the matching-member ordinal across archives. The ordinal default is correct only when archive order equals revision order. A basename that includes a revision suffix also needs a key capture to group those revisions into one document. For example, these synthetic paths carry version 12 before version 3:record_key_pattern: "urn:example:([A-Z0-9]+)-" and
source_position_pattern: "-([0-9]+)\\.xml$". The connector URL-decodes the
path first, so the first capture sees urn:example:ITEM7-12.xml and yields
ITEM7. The second yields 12 or 3. Both revisions remain available;
revision 12 stays current after importing the later folder.
Each pattern is a Go regular expression with exactly one capture group.
Every matched member must produce a nonempty key and a position containing
only decimal digits. Encode a literal percent sign as %25; decoding is strict so identity captures cannot silently switch to a different path representation. A malformed URL escape or invalid capture stops the page
before it can advance its checkpoint. Instance configuration is fixed: disable
the failed instance and create one with a corrected mapping or member filter.
Keep the same Corpus and source_namespace when the Record Key mapping remains
the same; a new instance rereads the prefix. Changing keys creates different
Records, so withdraw any previously accepted incorrect identities separately.
Revisions of one Record may share a page and are submitted in archive order,
with at most one revision of each key in flight. Only different keys run
concurrently; a later older position still cannot replace a higher current position.
Configure and start the source
Use the connector creation and credential workflow with kindobject_storage_archive. Set work_queue: "bulk" at the creation
request’s top level, alongside kind, config and schedule, so this historical
import uses bulk workers. For example, these are request fields, outside config:
config.
This is an example for your own storage; replace the bucket, prefix and region:
endpoint to an HTTP or HTTPS origin for an S3-compatible store.
Omit it for AWS S3. Globs match the raw object key or member path; * stays
within one directory, ** spans directories, and ? matches one character.
Archive patterns default to both supported archive types; the member pattern
defaults to **. Directories and links are excluded.
Deposit access_key_id, secret_access_key and, when needed, session_token
in the credential’s secret object. These fields are write-only and never
returned by the API. The plugin uses deposited credentials exclusively.
Keep these values out of source control and shell history.
batch_size accepts 1–1000 matching members, default 100. The page cache also
caps the batch at 64 MiB. concurrency accepts 1–32 distinct Record submissions,
default 1. Increase it only after measuring database and storage headroom on
your installation; pages remain sequential. Each member must contain 1 byte to 25 MiB of uncompressed data.
An empty or larger matched member stops the page with a source error.
Read progress and resume
Read the Connector Instance’s health as shown in Check collection. Its diagnostics report:
For a page from the current archive,
page_cut is batch_size, page_bytes,
or archive_end. These mean the requested item count, the 64 MiB
page byte bound, or the end of that archive. Repeated Record Keys do not cut a page.
The ZIP reader’s 1 MiB read window bounds buffering; it does not limit page bytes.
acquisition.items counts page members. fetch_ms is elapsed fetch time,
including the engine’s lease/archive admission and its release. grant_ms, upload_ms, verify_ms, accept_ms,
receipt_ms and describe_ms sum time across workers for upload grants,
plugin uploads, stored-byte verification, durable submission, replay checks
and attachment descriptions. These sums overlap when submissions run in
parallel; compare them with page_ms (elapsed fetch and submission time),
rather than adding them to estimate elapsed time. Upload time includes the
plugin’s invocation, source/cache access, spooling and PUT. run_ms is elapsed
time since the run began acquiring pages. Times are milliseconds.
more reports whether acquisition can continue. stop_reason is empty for
an intermediate page, or source_drained, page_limit, soft_limit or
attachment_budget for the last page. Failed or interrupted page logs use
incomplete; intentionally skipped or inactive runs emit no page diagnostic. Runs remain bounded to ten pages, two
minutes before starting another page, and a 200 MiB attachment budget.
acquisition.stop_reason: page_limit means the run reached ten pages;
use page_cut to explain the size of an individual archive page.
continuation: true means the engine requests another durable leased run
immediately after a successful bounded run. The final page must report more: true and advance the checkpoint from both this page’s input and the run’s starting
value.
Replayed items and pages containing only filtered entries can advance this cursor
without creating new versions. Unchanged checkpoints, source exhaustion, errors, notices and permanent
item rejections keep the configured interval; RetryAfter and healthy push
can make that wait longer. Each continuation has a new run sequence and remains
in the same work queue. Dispatch and processing capacity still govern its pace.
Before each bulk fetch, the engine checks the ingestion-only waiting count in
the shared queue snapshot. At 1,000 waiting documents, acquisition defers to
its interval until processing catches up. This count deduplicates pending
receipts and documents awaiting baseline processing; active ingestion attempts,
post-baseline enrichment, rebuilds and other operations do not contribute.
Missing or stale observations also defer bulk acquisition. Live connectors
continue polling independently. Snapshot refresh lag and concurrent pages can
overshoot this watermark; it bounds prefetching rather than setting an import
size limit.
acquisition.continuation_reason explains a false continuation: run_in_progress,
source_drained, checkpoint_stalled, checkpoint_cycled, source_notice,
item_rejected, store_unsupported or incomplete. An empty value accompanies a requested
continuation. A deferral before the next fetch logs connector run deferred
with continuation_reason: ingestion_backpressure or queue_unavailable;
it keeps the last committed page diagnostic and checkpoint. A run that defers
before fetching its first page does not count as a successful source poll.
interval_ms reports the configured wait used when continuation is false.
For those runs, estimate scheduled idle share as
interval_ms / (run_ms + interval_ms); a continuation has no configured wait.
Logs and successive committed pages show any additional dispatch delay.
Diagnostics capture timing before checkpoint persistence. The connector acquisition page log includes that persistence in elapsed page/run time and
reports a soft limit crossed during the commit. That crossing can also change
continuation from false in the stored snapshot to true in the committed log;
use the log’s post-commit decision when classifying how the run ended. An incomplete page is logged
without advancing its checkpoint. If kind diagnostics are omitted or null,
already use acquisition, or adding timing would exceed the 16 KiB diagnostics
bound, the engine preserves their existing behavior and reports timings in the
log only.
For a known total across the prefix, set optional members_total. Tar archives
are never scanned once just to count members. ZIP’s directory supplies its
matching-member count. ZIP directories are bounded to 4 MiB and 100,000 entries before parsing; use tar.gz for larger member sets. ZIP member reads retain one 1 MiB range window. A percentage measures source reading, while
members_done measures committed ingestion progress, including permanent
rejections reported as item_rejected in health. Accepted members can still
be waiting for normalization or search publication; acceptance does not mean
searchability.
Use pause and resume to stop
collection and continue from its last saved checkpoint. Pause also stops
continuing imports. A source request already in progress may finish, but cannot
save a later checkpoint after pause; already accepted documents keep processing.
Resume makes collection due immediately unless the source’s Retry-After delays it.
A manual run while paused is refused; permanent disable cannot be resumed.
The checkpoint records the archive ETag and member
offset. After a worker or plugin restart, the connector scans the compressed
tar stream to that offset and continues. ZIP uses ranged member reads.
Escape-heavy archive identities retain compact refs to fit the 1,024-byte
protocol bound; recovering out-of-order uploads of those refs can require
more than one gzip prefix scan.
A small page cache accelerates upload grants; recovery also works when that
cache is gone. Repeated submissions converge on idempotent receipts.
Replacing an in-progress source archive reports archive_changed; deleting
it reports archive_missing. Restore the original object with its original ETag
and request another run, or disable it
and create another for the replacement immutable source. Identical bytes alone
may produce a different ETag. Credential failures show as access errors.
Invalid identity captures and malformed archives show as source errors;
retryable storage failures retain the last committed checkpoint.
Measure throughput
On a running local demo or development stack, runmake measure-archive
with its stack name, as described in the plugin’s
package README.
The measurement seeds synthetic source objects and reports accepted members
per second against the 50 members/second acquisition target. It also reports
searchable and enriched unique Records per second, stage rollups, worker service
durations and hardware metadata. It waits for enrichment before finishing.
Organization-wide rollups need an otherwise idle Organization; document timeline
intervals include queue waits. Synthetic text uses built-in normalization.
It is outside CI.
To measure hosted embedding without calling a paid provider, use the loopback
fake on the same local stack:
hosted.embed, then restores the stack configuration
and processes. It records provider batch sizes and latency alongside the archive
measurement under .scratch/demo/. Compare with --batch-wait-ms 0 to disable
cross-Version batching, keeping hardware, source settings and fake latency equal.
The default 25 ms collection window groups document inputs from the same
Organization within provider item and token limits; query embedding bypasses it.
The ingestion worker admits four processing batch activities with up to 16
receipts each, or 64 receipts per process. Each receipt keeps its pinned plan,
stage deadlines and replay guards. Provider concurrency remains separately
bounded by max_concurrent_requests. Add worker replicas only within your
database and provider capacity.
See Run a large import for storage sizing,
provider capacity and queue monitoring, and hosted embedding settings.