> ## Documentation Index
> Fetch the complete documentation index at: https://docs.quivr.thevibecompany.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade a plugin with no downtime

> Replace the plugin version in service with a new one, watch the switch, roll back if it misbehaves, and fill a new vector space, while Quivr keeps ingesting and answering searches.

You run the new version next to the old one, switch to it in one call, watch the old one drain, and roll back in one call if needed. If the new version brings an embedding model, you then fill its vector space for past articles. No step restarts the api or the worker. Each step links to the page that details it.

## Prerequisites

* An operator key in `QUIVR_OPERATOR_KEY`, with the actions `plugins:admin`, `observability:read`, `operations:read` and `operations:write` on every Corpus of its Organization. Registering, activating and rolling back apply to the whole deployment; the plugin stats, the quarantine list and backfills cover only the key's Organization, so watch each Organization with its own key.
* The new version certified by `quivr plugin test`, deployed at its own address and reachable from the api and the worker. The old version keeps running.
* For a new embedding model: the new version's manifest declares the new vector space, and it cuts articles into the same segments as the old version.

## Steps

<Steps>
  <Step title="Note the plan in service">
    ```bash theme={null}
    curl -s "$QUIVR_API_URL/v0/admin/plugins/plan" -H "Authorization: Bearer $QUIVR_OPERATOR_KEY" \
      | jq '{plan_id, roles: [.roles[] | {role, plugin_id, version}]}'
    ```

    This is the plan a rollback returns to.
  </Step>

  <Step title="Register the new version and wait for its check">
    Follow the "Register it" and "Wait for the check" steps of [Switch plugins without restarting](/plugins/switch-plugins-without-restarting#steps). Keep the vector space in service as `served`. Register a new embedding model's space as `evaluation`, so search does not use it before it is filled:

    ```json theme={null}
    {"spaces": {"acme.embedder.base": "served", "acme.embedder.large": "evaluation"}}
    ```

    Go on once the registration is `validated`, and keep its id:

    ```bash theme={null}
    export REGISTRATION_ID=<the registration_id of the new version>
    ```
  </Step>

  <Step title="Activate it">
    ```bash theme={null}
    export SWITCHED_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
    curl -s -X POST "$QUIVR_API_URL/v0/admin/plugins/$REGISTRATION_ID/activate" -H "Authorization: Bearer $QUIVR_OPERATOR_KEY" \
      | jq '{plan_id, roles: [.roles[] | {role, plugin_id, version}]}'
    ```

    Every api and worker process follows within `plugin_plan_poll` (2 seconds by default). From then on, new work and every search use the new version.
  </Step>

  <Step title="Watch the old version drain">
    Work that started before the switch finishes on the old version. The old version reads `draining` until that work is done, then `inactive`:

    ```bash theme={null}
    curl -s "$QUIVR_API_URL/v0/admin/plugins" -H "Authorization: Bearer $QUIVR_OPERATOR_KEY" \
      | jq '[.items[] | select(.state == "active" or .state == "draining") | {plugin_id, version, state, pinned_work}]'
    ```

    While it drains, compare the two versions' calls and errors, and list the articles quarantined since the switch, which should stay empty:

    ```bash theme={null}
    curl -s "$QUIVR_API_URL/v0/admin/stats/plugins?window=1h" -H "Authorization: Bearer $QUIVR_OPERATOR_KEY" \
      | jq '[.items[] | {plugin_id, plugin_version, operation, calls: .summary.count, errors: .summary.errors}]'
    curl -s "$QUIVR_API_URL/v0/admin/quarantine?quarantined_after=$SWITCHED_AT" -H "Authorization: Bearer $QUIVR_OPERATOR_KEY" \
      | jq '[.items[] | {version_id, code: .reason.code, plugin: .reason.plugin}]'
    ```

    If the new version's errors climb or articles are quarantined, [roll back](#roll-back).
  </Step>

  <Step title="Stop the old version">
    Stop its process once it reads `inactive`. Stopping it earlier makes its pinned work retry, then stop with `pinned_plugin_unavailable`; the work never moves to the new version (see [Work finishes on the version it started with](/plugins/switch-plugins-without-restarting#work-finishes-on-the-version-it-started-with)).
  </Step>
</Steps>

## Roll back

One call returns to the plan you noted in the first step. Choose what happens to the work already pinned to the new version:

| Situation | Call |
| - | - |
| The new version works but you prefer the old one, for example it is slower | `{"pinned_work": "drain"}`: that work finishes on the new version, which drains as the old one did. Keep it running until it reads `inactive` |
| The new version produces bad output or hangs | `{"pinned_work": "stop"}`: that work stops with `pinned_plan_stopped` and its articles are quarantined |

The request, its answer and its errors are in [Roll back](/plugins/switch-plugins-without-restarting#roll-back). After a `stop`, [reprocess the quarantined articles](/plugins/reprocess-quarantined-versions) through the plan now active.

## Fill a new vector space

When the new version brings an embedding model, a Corpus created after the switch carries its evaluation space from the start. An existing Corpus gets vectors in it only once a backfill of that Corpus starts: from then on live ingestion fills it for new articles, and the backfill fills the ones before. So [fill it](/plugins/backfill-a-vector-space) for each existing Corpus without `accepted_before`: run the dry run, start the backfill, follow it to `succeeded`, then promote the space so search uses it. Promoting the former space goes back to the previous model at once.

## Check it worked

* `GET /v0/admin/plugins/plan` names the new version for its roles, and the old version reads `inactive`.
* No article was quarantined since the switch.
* A semantic search's hits name the vector space you promoted, in `vector_space_id`.

## Restarts during an upgrade

Every step survives a restart of the api or the worker:

* an activation or a rollback is kept in the database, and the configuration applies at startup only what changed in it;
* pinned work keeps its plan across worker restarts;
* a backfill resumes after its checkpoint.

A single api process does not answer while it restarts. Run two or more behind a load balancer if you restart the api during an upgrade.

## Troubleshooting

| Symptom | Cause | Fix |
| - | - | - |
| The old version stays `draining` | Work is still pinned to it, such as a long connector run | Keep it running. Do not terminate that work by hand in Temporal: its count would never be released |
| `409 registration_not_validated` on activation | The check failed | Read `check.checks` on the registration, fix the build and register it again with a new `idempotency_key` |
| `409 plugin_unreachable` on rollback | The version you return to is stopped or answers with another build | Start it again at its address, then retry |
| Articles are quarantined after the switch | The new version refuses them | Roll back, then reprocess the quarantined articles |