Skip to main content
The rss kind polls one feed URL and turns each feed item into a Record in the instance’s Corpus and Source Namespace. It reads RSS 0.9x/2.0, RSS 1.0 (RDF) and Atom. JSON Feed documents are also accepted (feed.format: "json"). Read the common guide first: it covers credentials, health, events and disabling for every kind.

​
Install the plugin

The kind comes from the first-party plugin connector.rss, not from the core: build and pin it as its README shows. make dev, make verify, the demo and the Railway image pin it by default; QUIVR_RSS=off make dev leaves it out. Without the pin, rss instances keep their checkpoints but their runs fail with unsupported_connector_kind, and pinning it again resumes them. Instances created when the kind was built into the core continue without being recreated.

​
Create an instance

Use one instance per feed. Collection starts right after creation and ingests every item currently in the feed.

​
Protected feeds

Some feeds need HTTP authentication. Deposit one of these as the credential secret:
This is sent as HTTP Basic authentication.
This is sent as Authorization: Bearer …. The credential is sent to the feed URL. On a redirect it follows Go’s rule: it is kept only for the same host or its subdomains and dropped for any other domain. A redirect from https to http is refused (insecure_redirect), so a credential never travels in clear text. Rotate it with PUT /v0/connectors/{id}/credential, and set expires_at if the provider’s credential expires. Never collect paid content, and never use this to bypass the publisher’s access controls.

​
Polling

  • Conditional requests. Each poll sends If-None-Match / If-Modified-Since from the previous response. A 304 Not Modified is a successful poll with nothing new.
  • Feed ttl. When honor_ttl is true and the feed declares <ttl> (minutes), polls are skipped until that time has passed since the last fetch. The ttl counts for at most 60 minutes. The effective wait is roughly max(interval, min(ttl, 60 min)), rounded up to the next scheduled run. A skipped run does not touch the feed and does not update last_success_at. Set honor_ttl: false to poll at the interval regardless.
  • Bounds.
    • Each request times out after 20 seconds.
    • Responses larger than 10 MiB (after decompression) are refused.
    • At most 5 redirects are followed.
    • At most the first 1,000 items of a feed are considered, and at most 250 new or changed items are submitted per page (fewer when their text passes 12 MiB; the rest follows on the next page).
  • Network policy. Unspecified, loopback, private, ULA, link-local (including cloud metadata), multicast, carrier-grade NAT, documentation, benchmarking, reserved and other special-purpose ranges, and IPv6 prefixes that embed an IPv4 address (IPv4-mapped, IPv4-compatible, NAT64, 6to4, Teredo), are refused (address_not_allowed). The check runs on the address actually dialed, after DNS resolution, for every connection and redirect hop, with the same rule as webhook delivery. Only a local test deployment should set allow_private_addresses: true in the pin’s configuration.

​
What is collected

Only the feed document is fetched. Linked article pages and enclosures are never downloaded; enrichment belongs to plugins. Each item becomes one Record Version whose Manifest has these Parts: HTML is converted to text. Scripts, styles and embedded objects are dropped, block elements become line breaks, and entities are decoded. Only title and body are indexed for search. The other Parts are kept but not searched. The connector.rss extension (schema version 1) carries the rest:
  • item: guid, link, other links, authors (name, email), published and updated (RFC 3339), categories (up to 50) and enclosures (url, type, length; references only, up to 20).
  • feed: format (rss, atom, json), version, title, link and language. Volatile values such as the feed’s build date are left out so that a re-fetched item stays byte-identical.
  • item.truncated: true when a bound below was applied.
Bounds per item:
  • Metadata strings longer than 2 KiB are truncated.
  • The body text is kept up to 384 KiB, the summary up to 64 KiB, and the whole item’s text up to 768 KiB. Original markup is dropped first when an item would exceed that.
  • Metadata lists are halved until the extension fits in 48 KiB.
Items with neither a title nor any text are skipped.

​
Identity, corrections and disappearance

  • Record Key. The key is the item’s guid (RSS) or id (Atom). Without one, it is the item link, and without a link, a hash of the title and text. Keys longer than 1 KiB are replaced by their hash.
  • Revision. The revision is a hash of the item’s own fields: title, content, summary, links, authors, dates, categories and enclosures. Changes to feed metadata (such as a renamed feed) do not change it.
  • Edited items. A republished or edited item gets a new revision and becomes a correction: a new Version of the same Record. Items without guid or link cannot be followed across edits; an edit creates a new Record.
  • Unchanged items. They are not submitted again and create no Version. A feed that keeps returning the same items therefore counts toward silent.
  • Crash window. If the worker stops after submitting new items but before recording the poll, the retry replays those items. They keep a single Version, but they no longer refresh last_item_at. The next new item does.
  • Items dropping out. An item that drops out of the feed (feeds only list their latest items) is not withdrawn. Its Record stays searchable.

​
Health and error codes

A malformed feed is rejected as a whole, so a broken document never creates partial Records.

​
Troubleshooting

  • access_error with unauthorized or forbidden. Check the feed in a browser or with curl -I. Deposit or rotate the credential if the feed is protected.
  • access_error with gone. The publisher retired the feed. Disable the instance and create a new one for the replacement URL, reusing the Source Namespace only if the new feed keeps the same item guids.
  • malformed_feed. The URL may point at an HTML page instead of the feed. Look for a <link rel="alternate" type="application/rss+xml"> on the page.
  • silent but the site publishes. The feed may lag the site, or the publisher may reuse guids. Compare the feed’s newest item with the site.
  • Items appear late. Check schedule.interval_seconds and the feed’s <ttl>. Set honor_ttl: false if the ttl is too conservative.

​
Checking a real feed (optional)

CI uses only a local fake feed server. To try a real public feed, create an instance on a scratch Corpus pointing at a feed you are allowed to read. Wait for last_success_at, then list the Corpus Records. This check is manual and never part of make verify.