# Database operations

> **For AI agents:** the complete documentation index is at [llms.txt](https://questdb.com/docs/llms.txt). Every page is available as markdown by appending `.md` to its URL, or by sending an `Accept: text/markdown` request header.

Operate, scale, resize, maintain, and delete Operator-managed QuestDB clusters.

# Database operations

Examples use `<name>` and `<namespace>`. Status is trustworthy only after the
controller has observed the current spec.

## Check database health

Check generation freshness first, then conditions. `status.phase` is only a
human-readable summary.

```sh
kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='generation={.metadata.generation}{" observedGeneration="}{.status.observedGeneration}{" phase="}{.status.phase}{"\n"}'
kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" reason="}{.reason}{" observed="}{.observedGeneration}{" message="}{.message}{"\n"}{end}'
```

Do not act on old conditions while `.metadata.generation` differs from
`.status.observedGeneration`. For an ordinary writable cluster, writer readiness
requires all four signals: current generation, `Available=True/PrimaryReady`,
`Progressing=False/Settled`, and `WriteHealthy=True/Healthy`. `Available=True`
alone can mean only that a replica or a read-serving primary is available.
`WriteHealthy=False/PrimarySuspended` names the impaired WAL table or tables in
its message; reads and writes to other tables can remain available.
`WriteHealthy=True` reports the latest `wal_tables()` observation. It does not
perform a synthetic write and does not guarantee free disk capacity, so retain
application write probes and storage monitoring where those guarantees are
required.

An intentional replica-only follower is the exception: no primary is correct, so
`WriteHealthy` is omitted rather than reported healthy. Require current
generation, `phase=Following`, `status.replication.following=true`, the expected
`readyInstances`, and an appropriate `ReplicationHealthy` follower result.
`True/FollowingExternalSource` reports observed progress; a quiet source may
report `Unknown/StreamNotDetermined`, which is acceptable only after confirming
the source identity and roots. Treat `ReplicationHealthy=False` as unhealthy.

```sh
kubectl get questdbcluster <name> -n <namespace> -o wide
kubectl get pods -n <namespace> -l questdb.io/cluster=<name> \
  -L questdb.io/instance,questdb.io/role -o wide
kubectl get pvc -n <namespace> -l questdb.io/cluster=<name> \
  -L questdb.io/role,questdb.io/bootstrap -o wide
```

## Services and ports

| Service     | Purpose                                                                                                                                                              |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `<name>-rw` | Current primary. Use for writes and administration.                                                                                                                  |
| `<name>-ro` | Created for store-backed replication. Routes to qualified replicas, but falls back to the primary when no replica qualifies. It is **not** strict replica isolation. |
| `<name>`    | Headless, internal identity/DNS service. It publishes unready addresses and must not be used as a client availability endpoint.                                      |

| Port     | Scope                                                                   |
| -------- | ----------------------------------------------------------------------- |
| 9000/TCP | Published on database Services                                          |
| 8812/TCP | Published on database Services; optional TLS chosen at cluster creation |
| 9009/TCP | Published on database Services                                          |
| 9003/TCP | Always on Pods for kubelet/operator health and metrics; not on Services |
| 9007/UDP | Published on headless and `<name>-rw` only when enabled                 |

QWP over WebSocket has no port of its own. Ingestion (`/write/v4`) and streaming
query results (`/read/v1`) are served by the HTTP server on 9000 and share its
network settings, so they are available wherever 9000 is.

Existing Service-based database metrics scrapers must move to Pod discovery or a
`PodMonitor` that targets the named `metrics` port on QuestDB Pods. A Service
port-forward to 9003 no longer works.

The QWP UDP receiver is off by default. Enable it with
[`spec.protocols.qwp.udp.enabled`](/docs/enterprise-kubernetes-operator/configuration/#qwp-udp);
until then, port 9007 is neither opened on the pod nor published on any Service.
It is not published on `<name>-ro`.

The operator owns these Services and reconciles them as `ClusterIP` (the
identity Service is headless). For temporary access, use port-forwarding. For
durable external access, create a separate customer-managed Ingress, Gateway, or
LoadBalancer that targets the operator Service. Do not mutate the operator-owned
Service type.

## Connect

Find the admin Secret without printing its value:

```sh
ADMIN_SECRET="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.status.adminSecretName}')"
kubectl get secret "$ADMIN_SECRET" -n <namespace> \
  -o custom-columns='NAME:.metadata.name,CREATED:.metadata.creationTimestamp'
```

When an administrator explicitly needs the credential, load it into the
environment without echoing it and clear it after use:

```sh
ADMIN_PASSWORD="$(kubectl get secret "$ADMIN_SECRET" -n <namespace> \
  -o jsonpath='{.data.password}' | base64 -d)"

kubectl port-forward -n <namespace> service/<name>-rw 8812:8812 \
  >/tmp/questdb-pgwire-port-forward.log 2>&1 &
PF_PID=$!
trap 'kill "$PF_PID" 2>/dev/null; unset ADMIN_PASSWORD' EXIT
sleep 2
PGPASSWORD="$ADMIN_PASSWORD" psql -h 127.0.0.1 -p 8812 -U admin -d qdb
kill "$PF_PID"
unset ADMIN_PASSWORD
trap - EXIT
```

For a TLS-enabled cluster whose certificate uses the default Service identity,
keep the same port-forward but give `psql` both the certificate identity and the
local address:

```bash
PGPASSWORD="$ADMIN_PASSWORD" psql \
  "host=<name>-rw.<namespace>.svc hostaddr=127.0.0.1 port=8812 user=admin dbname=qdb sslmode=verify-full sslrootcert=/secure/path/ca.crt"
```

`host` supplies the certificate identity and SNI; `hostaddr` directs the
connection to the local tunnel.

Use `service/<name>-ro` only for read-only traffic that can tolerate primary
fallback. To access the Web Console temporarily, forward `service/<name>-rw`
from local port 9000 and stop the process after the session.

## Make mutable changes safely

Changes to `spec.image`, `spec.imagePullSecrets`, `spec.resources`,
`spec.config`, replication tuning, protocol fields within their allowed mode, or
scheduling recreate affected pods. Ordinary rollout is serialized: replicas roll
in serial order before the primary, and at most one ordinary delete is attempted
per instanceset reconcile. The operator may hold a rollout for exact topology,
usable PVCs, healthy peers, node readiness, or read-route safety. A rolling
change remains single-writer-safe, but recreating the primary can briefly
interrupt writes; a singleton has read/write downtime while its only Pod
restarts. Before deleting the only `ro-ready` replica, the operator first adds
the primary as an overlapping Service-selector candidate. That overlap is not an
EndpointSlice acknowledgement, connection drain, or zero-gap guarantee.

### Before you start

1. For an ordinary writable cluster, confirm current generation,
   `Available=True/PrimaryReady`, `Progressing=False/Settled`, and
   `WriteHealthy=True/Healthy`. `Available=True` alone is not writer readiness.
2. For an intentional follower, instead confirm current generation,
   `phase=Following`, `status.replication.following=true`, the expected ready
   count, and an appropriate non-failing follower `ReplicationHealthy` result as
   described above.
3. Save the current spec.
4. Make one logical change at a time.

```sh
kubectl get questdbcluster <name> -n <namespace> -o yaml \
  > /secure/path/<name>-before.yaml
```

### Change and watch

Apply a reviewed manifest or a narrow patch. This example changes an engine
setting:

```sh
kubectl patch questdbcluster <name> -n <namespace> --type merge \
  -p '{"spec":{"config":{"cairo.max.uncommitted.rows":"500000"}}}'

GEN="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.metadata.generation}')"
for _ in $(seq 1 60); do
  OBS="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.observedGeneration}')"
  [ "$OBS" = "$GEN" ] && break
  sleep 5
done
[ "$OBS" = "$GEN" ]
kubectl wait questdbcluster/<name> -n <namespace> \
  --for=condition=Available --timeout=5m
```

The `kubectl wait` above is only an availability gate. Before declaring the
writer ready, re-read the current-generation conditions and require
`Available=True/PrimaryReady`, `Progressing=False/Settled`, and
`WriteHealthy=True/Healthy`. Then verify pod UIDs/restarts, connectivity, and
the intended setting. `Progressing=True/RollingUpdate` can mean the controller
is safely waiting on topology, PVC, peer, Node, or read-routing conditions. If
the change fails, inspect `ConfigRejected`, events, and pod logs. Recover by
reverting the mutable spec to the saved value and repeat the bounded generation
and full writer-health checks. Do not try to revert immutable fields; create a
new cluster when required by the
[API Reference](/docs/enterprise-kubernetes-operator/reference/api/).

### Rotate static object-store credentials safely

1. Make old and new provider credentials valid concurrently.
2. Update the same-namespace Secret referenced by `QuestDBObjectStore`.
3. Allow the manager's roughly two-minute resync plus serialized Pod restart to
   converge.
4. Verify expected Pod UID changes and `BackupHealthy`/`ReplicationHealthy`.
5. Revoke the old credentials only after convergence.

Immediate provider-side revocation can interrupt in-flight QuestDB object-store
I/O.

## Scale out

A replica is born from a completed backup and then consumes object-store WAL.
Before increasing `spec.instances`, the cluster needs:

- `spec.objectStoreRef`;
- enabled scheduled backup; and
- a completed seed in `.status.replication.seed`.

An explicit `spec.replication` block is optional; add it only for tuning.

```sh
BACKUP_STATUS="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.status.backup.lastBackup.status}')"
SEED="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.status.replication.seed.backupInstanceName}')"
printf 'backup=%s seed=%s\n' "$BACKUP_STATUS" "$SEED"
[ "$BACKUP_STATUS" = "completed" ] && [ -n "$SEED" ]

kubectl patch questdbcluster <name> -n <namespace> --type merge \
  -p '{"spec":{"instances":3}}'
```

Watch with bounded checks:

```sh
for _ in $(seq 1 90); do
  READY="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.readyInstances}')"
  [ "$READY" = "3" ] && break
  sleep 10
done
[ "$READY" = "3" ]
kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{range .status.conditions[?(@.type=="ReplicationHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{range .status.replication.replicas[*]}{.instance}{" caughtUp="}{.caughtUp}{" caughtUpNow="}{.caughtUpNow}{" lagTxns="}{.lagTxns}{"\n"}{end}'
```

Verify `readyInstances` equals the requested count, `ReplicationHealthy=True`,
and each current replica is healthy. `caughtUp` means it has caught up at least
once; it is a latch. `caughtUpNow` is the live freshness reading. An absent
`caughtUpNow` means freshness was not determined and must fail closed.

## Scale in

Lowering `spec.instances` removes replicas. Instance serials are monotonic and
are never reused, so names may not remain contiguous after scale or promotion.

| `spec.pvcRetentionPolicy` | Removed replica PVC                                                                 |
| ------------------------- | ----------------------------------------------------------------------------------- |
| `Retain` (default)        | Kept for deliberate inspection or cleanup.                                          |
| `Delete`                  | Deleted during replica scale-in. A later replica gets a new serial and seeds again. |

The policy applies only to replicas removed by scale-in. It never deletes the
current primary PVC and does not govern whole-cluster deletion. On every managed
PVC, `questdb.io/role` is the instance's **current** role and changes across
promotion; `questdb.io/bootstrap` records how the volume was born and remains
ancestry/state. Do not infer the current primary from `bootstrap`.

Before scaling in, identify the primary and replicas, choose the retention
policy, and inventory PVCs:

```sh
kubectl get questdbcluster <name> -n <namespace> -o wide
kubectl get pvc -n <namespace> -l questdb.io/cluster=<name>
kubectl patch questdbcluster <name> -n <namespace> --type merge \
  -p '{"spec":{"instances":1,"pvcRetentionPolicy":"Retain"}}'
GENERATION="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.metadata.generation}')"
OBSERVED=""
AVAILABLE_OBSERVED=""
for _ in $(seq 1 120); do
  OBSERVED="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.observedGeneration}')"
  AVAILABLE_OBSERVED="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.conditions[?(@.type=="Available")].observedGeneration}')"
  [ "$OBSERVED" = "$GENERATION" ] && \
    [ "$AVAILABLE_OBSERVED" = "$GENERATION" ] && break
  sleep 5
done
if [ "$OBSERVED" = "$GENERATION" ] && \
   [ "$AVAILABLE_OBSERVED" = "$GENERATION" ]; then
  kubectl wait questdbcluster/<name> -n <namespace> \
    --for=condition=Available=True --timeout=10m &&
  kubectl get questdbcluster <name> -n <namespace> -o wide &&
  kubectl get pods,pvc -n <namespace> -l questdb.io/cluster=<name> -o wide
else
  printf 'Timed out waiting for generation %s (status %s, Available %s)\n' \
    "$GENERATION" "$OBSERVED" "$AVAILABLE_OBSERVED" >&2
  false
fi
```

Verify the managed and ready counts, current primary, the full ordinary
writer-health or separate follower contract, and the expected retained or
deleted replica PVCs. Confirm each current primary/replica PVC's
`questdb.io/role` label matches that role.

## Move the cold-storage manager

With
[`spec.coldStorage`](/docs/enterprise-kubernetes-operator/configuration/#cold-storage),
the cold-storage [manager role](/docs/concepts/cold-storage/#roles) is moved
by changing `spec.coldStorage.manager`, never by running
[`SWITCH COLD STORAGE ROLE`](/docs/query/sql/switch-cold-storage-role/)
against instances. Run the manager on a replica: that moves upload, manifest,
and garbage-collection work off the primary, and a `Planned` promotion
requires the manager settled away from the departing primary.

Before the change, require current generation,
`ColdStorageHealthy=True/ManagerReady`, and an empty `handoffSource`:

```sh
kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{range .status.conditions[?(@.type=="ColdStorageHealthy")]}{.type}{"="}{.status}{" reason="}{.reason}{" observed="}{.observedGeneration}{"\n"}{end}manager={.spec.coldStorage.manager}{" current="}{.status.coldStorage.currentManager}{" term="}{.status.coldStorage.managerTerm}{" handoffSource="}{.status.coldStorage.handoffSource}{"\n"}'
```

Choose the serial of a ready replica (`<name>-3` has serial `3`) and patch:

```sh
kubectl patch questdbcluster <name> -n <namespace> --type merge \
  -p '{"spec":{"coldStorage":{"manager":3}}}'
```

The operator demotes the old manager, verifies it settled as a refresher,
then promotes the replacement — never two managers at once. While the handoff
is in flight, `status.coldStorage.handoffSource` names the demoted instance.
Uploads and remote garbage collection pause during the short managerless
interval; reads continue everywhere. Wait for `True/ManagerReady` with
`currentManager` naming the target and a positive `managerTerm`.

`False/ManagerHandoffBlocked` fails closed rather than risking two managers —
for example while the current manager is unreachable. Recover the named
instance; do not work around the block with direct SQL. Two interactions to
plan around:

- An active promotion pauses new cold-manager handoffs. Finish or resolve the
  cutover first.
- `manager` must not exceed `instances`, so scale in only after moving the
  manager to a remaining serial.

## Grow storage

Storage is expand-only, and `spec.storage.storageClassName` is immutable.

### Before you start

```sh
STORAGE_CLASS="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.spec.storage.storageClassName}')"
kubectl get storageclass "$STORAGE_CLASS" \
  -o jsonpath='allowVolumeExpansion={.allowVolumeExpansion}{"\n"}'
kubectl get pvc -n <namespace> -l questdb.io/cluster=<name>
```

Proceed only when `allowVolumeExpansion=true` and the CSI driver supports the
requested expansion.

### Change and verify

```sh
kubectl patch questdbcluster <name> -n <namespace> --type merge \
  -p '{"spec":{"storage":{"size":"200Gi"}}}'
GENERATION="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.metadata.generation}')"
OBSERVED=""
AVAILABLE_OBSERVED=""
for _ in $(seq 1 120); do
  OBSERVED="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.observedGeneration}')"
  AVAILABLE_OBSERVED="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.conditions[?(@.type=="Available")].observedGeneration}')"
  [ "$OBSERVED" = "$GENERATION" ] && \
    [ "$AVAILABLE_OBSERVED" = "$GENERATION" ] && break
  sleep 5
done
if [ "$OBSERVED" = "$GENERATION" ] && \
   [ "$AVAILABLE_OBSERVED" = "$GENERATION" ]; then
  kubectl wait questdbcluster/<name> -n <namespace> \
    --for=condition=Available=True --timeout=10m &&
  kubectl get questdbcluster <name> -n <namespace> -o wide &&
  kubectl get pvc -n <namespace> -l questdb.io/cluster=<name> \
    -o custom-columns='NAME:.metadata.name,REQUESTED:.spec.resources.requests.storage,CAPACITY:.status.capacity.storage,PHASE:.status.phase'
else
  printf 'Timed out waiting for generation %s (status %s, Available %s)\n' \
    "$GENERATION" "$OBSERVED" "$AVAILABLE_OBSERVED" >&2
  false
fi
```

The availability wait is not proof of writer health. Require the
current-generation `Available=True/PrimaryReady`, `Progressing=False/Settled`,
and `WriteHealthy=True/Healthy` conditions before closing the change. If
`StorageResizeBlocked=True`, read its reason/message and fix the StorageClass or
CSI limitation. A size reduction is rejected. To change StorageClass, restore
into a new cluster.

## Node maintenance and disruption budgets

The default PodDisruptionBudget (PDB) uses `minAvailable: 1` for a singleton and
`instances-1` for a replicated cluster. A singleton therefore blocks voluntary
eviction and can make `kubectl drain` wait indefinitely. A replicated cluster's
default permits one voluntary disruption at a time.

### Rotate a node pool without reducing replica capacity

Use a temporary surge replica for planned node-pool replacement. Kubernetes node
maintenance remains external to the operator: your infrastructure tooling
cordons, drains, powers off, and deletes Nodes; the operator manages only the
QuestDB Pods and PVCs.

This procedure preserves each existing instance identity and PVC. After a Pod is
evicted, the operator recreates the same instance on an eligible node and
reattaches its PVC. The temporary highest-serial replica is removed when you
restore the original instance count.

Before starting:

- use a replicated cluster with a completed backup seed, as required for
  [scale out](#scale-out);
- require current generation, `Available=True/PrimaryReady`,
  `Progressing=False/Settled`, `WriteHealthy=True/Healthy`, and
  `ReplicationHealthy=True`;
- provide replacement capacity in every zone required by the existing PVCs,
  matching the cluster's node selectors, affinity, and tolerations;
- finish any ordinary rollout, promotion, or cold-manager handoff; and
- record the original instance count, Pod UIDs, roles, Nodes, and PVCs.

```sh
ORIGINAL_INSTANCES="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.spec.instances}')"
[ "$ORIGINAL_INSTANCES" -ge 2 ]

kubectl get questdbcluster <name> -n <namespace> -o wide
kubectl get pods -n <namespace> -l questdb.io/cluster=<name> \
  -L questdb.io/instance,questdb.io/role,questdb.io/ro-ready -o wide
kubectl get pvc -n <namespace> -l questdb.io/cluster=<name> -o wide
```

#### 1. Cordon retiring nodes

Cordon every retiring node before creating the surge replica. Cordon does not
move existing Pods, but it prevents the scheduler from placing the surge or any
replacement Pod on those nodes.

```sh
kubectl cordon <retiring-node-1> <retiring-node-2>
kubectl get nodes <retiring-node-1> <retiring-node-2>
```

By default, QuestDB Pods have preferred hostname anti-affinity with weight 100
and a soft zone spread with `maxSkew: 1`. These rules prefer different hosts and
balanced zones but do not guarantee either under capacity pressure. An explicit
`spec.scheduling.affinity` fully replaces the default hostname anti-affinity; an
explicit `topologySpreadConstraints` list replaces the default zone spread.
Configure any required hard placement rules before this maintenance and let the
resulting Pod rollout settle first.

#### 2. Add and verify one surge replica

Increase the instance count by one. One surge is sufficient when nodes are
drained sequentially, which the default PDB requires.

```sh
SURGE_INSTANCES=$((ORIGINAL_INSTANCES + 1))
kubectl patch questdbcluster <name> -n <namespace> --type merge \
  -p "{\"spec\":{\"instances\":${SURGE_INSTANCES}}}"

GENERATION="$(kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{.metadata.generation}')"
for _ in $(seq 1 120); do
  OBSERVED="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.observedGeneration}')"
  READY="$(kubectl get questdbcluster <name> -n <namespace> \
    -o jsonpath='{.status.readyInstances}')"
  [ "$OBSERVED" = "$GENERATION" ] && \
    [ "$READY" = "$SURGE_INSTANCES" ] && break
  sleep 10
done
[ "$OBSERVED" = "$GENERATION" ]
[ "$READY" = "$SURGE_INSTANCES" ]

kubectl get questdbcluster <name> -n <namespace> \
  -o jsonpath='{range .status.conditions[?(@.type=="ReplicationHealthy")]}{.status}{" "}{.reason}{" "}{.message}{"\n"}{end}{range .status.replication.replicas[*]}{.instance}{" caughtUp="}{.caughtUp}{" caughtUpNow="}{.caughtUpNow}{" lagTxns="}{.lagTxns}{"\n"}{end}'
kubectl get pods -n <namespace> -l questdb.io/cluster=<name> \
  -L questdb.io/instance,questdb.io/role,questdb.io/ro-ready -o wide
```

Proceed only when `ReplicationHealthy=True`, every expected instance is Ready,
and the new replica has a current `caughtUpNow=true` observation and
`questdb.io/ro-ready=true`. A latched `caughtUp=true` by itself is not current
freshness. Confirm that the surge Pod is not on a retiring node.

If the surge remains Pending, do not drain anything. Add compatible capacity or
restore `spec.instances` to `ORIGINAL_INSTANCES` and uncordon the nodes to
abort.

#### 3. Handle a primary on a retiring node

Draining the primary recreates the same primary identity elsewhere and briefly
interrupts writes. For a planned handoff instead, first re-home an original
replica onto a non-retiring node, then create a
[`QuestDBPromotion`](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover)
targeting that replica and wait for it to complete. Drain the primary's node
last. Do not promote the temporary highest-serial surge replica: keeping it a
replica makes the final scale-in remove the temporary instance rather than a
long-lived one. Structured cold storage may require moving its manager first;
follow [Move the cold-storage manager](#move-the-cold-storage-manager).

Do not use an Emergency promotion for routine maintenance. It explicitly accepts
loss of WAL that the old primary has not uploaded.

#### 4. Drain one node at a time

Drain one retiring node through your cloud's managed-node workflow or with
`kubectl drain`. The QuestDB eviction consumes the default PDB's one allowed
voluntary disruption, so wait for full recovery before draining the next node.

```sh
kubectl drain <retiring-node-1> --ignore-daemonsets
```

Do not bypass the PDB or force-delete a QuestDB Pod to accelerate a healthy
planned drain. After the eviction, the operator waits for the old Pod object to
be gone, then recreates that same instance and PVC. The cordon keeps it off
every retiring node.

Before proceeding to the next node, require all of the following again:

- `status.observedGeneration` equals `metadata.generation`;
- `status.readyInstances` equals `SURGE_INSTANCES`;
- `ReplicationHealthy=True` and every replica is currently healthy;
- the recreated Pod has a new UID and runs on a non-retiring node; and
- the ordinary writer-health contract is satisfied.

If recovery stalls, leave the surge in place and stop the maintenance workflow.
Inspect Pod scheduling events, PVC/PV topology, volume attachment, Node health,
and cluster conditions. Recover the node or add same-zone capacity; do not move
on to another drain while the cluster is below the surge count.

Repeat this step sequentially for each retiring node that hosts a QuestDB Pod.
Nodes with no QuestDB Pod do not require another database surge or recovery
cycle.

#### 5. Return to the steady instance count

After every QuestDB Pod runs on a non-retiring node and the cluster is healthy,
restore the original count. Scale-in removes the temporary highest-serial
replica.

```sh
kubectl patch questdbcluster <name> -n <namespace> --type merge \
  -p "{\"spec\":{\"instances\":${ORIGINAL_INSTANCES}}}"
```

Wait for current generation, `readyInstances=ORIGINAL_INSTANCES`, settled
conditions, and the full writer and replication health contracts. Inventory Pods
and PVCs afterward. With `pvcRetentionPolicy: Retain`, the temporary replica's
PVC remains for deliberate inspection or cleanup; with `Delete`, the operator
removes it during scale-in.

The retiring Nodes can now remain cordoned and be removed by the external
node-pool workflow. If maintenance is cancelled instead, uncordon only Nodes
that remain valid destinations.

It is safe to pause with the surge replica running. Restoring the steady count
is cleanup, not a prerequisite for database availability.

### Other maintenance choices

When a surge replica is not appropriate, lower
`spec.scheduling.podDisruptionBudget.minAvailable`, or set
`spec.scheduling.podDisruptionBudget.enabled: false` only after explicitly
accepting the resulting database downtime.

```yaml
spec:
  scheduling:
    podDisruptionBudget:
      enabled: true
      minAvailable: 0
```

Restore the normal PDB after maintenance and verify the current-generation
writer contract: `Available=True/PrimaryReady`, `Progressing=False/Settled`, and
`WriteHealthy=True/Healthy`.

A PDB protects only voluntary disruption. On node loss, the operator reports
`InstanceUnreachable` and does not automatically promote a replica. It avoids
recreating a pod while the old pod may still run on an unreachable node. Restore
the node/volume or follow the explicit
[promotion and failover](/docs/enterprise-kubernetes-operator/high-availability/#promotion-and-failover)
procedure.

## Delete a database cluster

Deleting a `QuestDBCluster` removes operator-owned pods, Services, ConfigMaps,
and related resources. Its data PVCs are deliberately unowned and remain, and
object-store backup/WAL data also remains. `pvcRetentionPolicy` does not change
this whole-cluster behavior.

### Before you start

Inventory and export the CR, PVCs, PV reclaim policies, store reference, and
effective object prefixes:

```sh
kubectl get questdbcluster <name> -n <namespace> -o yaml \
  > /secure/path/<name>-cluster.yaml
kubectl get pvc -n <namespace> -l questdb.io/cluster=<name> -o wide
kubectl get questdbobjectstore -n <namespace>
```

:::danger Deleting the custom resource stops the database and removes its
managed runtime objects. Confirm applications are stopped and that the CR export
and data-retention inventory are complete. :::

```sh
kubectl delete questdbcluster <name> -n <namespace> --timeout=5m
kubectl get pvc -n <namespace> -l questdb.io/cluster=<name>
```

Decide separately whether to retain or delete each PVC/PV according to its
StorageClass reclaim policy. Decide separately whether to retain or delete cloud
objects using customer-owned cloud tooling and policy. The operator never cleans
the object store.
