SYS.DOCS // DOCS

ClickHouse

Edka provisions ClickHouse on your cluster with the ClickHouse Operator. Each database is a ClickHouseCluster for the servers and a KeeperCluster for ClickHouse Keeper coordination, in a namespace you choose.

A ClickHouse database has Overview, Insights, Logs, Backups, and Settings tabs.

Prerequisites

Kubernetes v1.33 or later on the cluster. The operator requires it, and Edka rejects the create or update on an older version.
For high availability, at least three worker nodes.
For backups, an Object Storage integration with the Amazon S3 or S3-Compatible provider. Google Cloud Storage and Azure Blob Storage integrations are not ClickHouse backup targets.
For a Tailscale endpoint, the Tailscale add-on installed on the cluster. For a private load balancer, private networking enabled on the cluster.

Create a database

Open the cluster and select Databases under Workloads. Select Create Database, then pick ClickHouse.
On General, enter the Database name and the Namespace. Both are fixed after creation. Pick the ClickHouse version and set the Default user password.
On Topology, select Single instance or High availability, or set the counts yourself. See Topology and storage.
On Resources, pick a Resource profile and set requests and limits. See Resource profiles.
On Placement, pick a Node pool or keep the default scheduling. Keep Tolerate node-pool taints on when the pool is tainted.
On Access, set TLS and external access. See TLS and external access.
On Backup, turn on scheduled backups or restore a snapshot. See Backups.
Select Create ClickHouse.

While the database starts, the status message counts ready replicas, for example Waiting for Keeper: 2 of 3 replicas ready. Creation fails if Keeper, and then the servers, are not ready within 20 minutes.

The versions are 26.8.8.8 (LTS, the default), 26.4.5.143, and 25.12.12.1. Each is an exact image tag, so every replica runs the same build. New databases run Keeper 26.8.8.8.

Topology and storage

Field	Values	Default
Replicas per shard	`1` to `5`	`1`
Shards	`1` to `16`	`1`
Keeper replicas	`1`, `3`, or `5`	`1`
ClickHouse storage (Gi)	`1` to `5000`	`20`
Keeper storage (Gi)	`1` to `1000`	`10`

Replicas per shard times shards is the number of server pods, up to 16. More than one server pod requires 3 or 5 Keeper replicas.

Each server replica and each Keeper replica has its own volume. Both storage sizes can grow later. They cannot shrink. Shards, ClickHouse storage class, and Keeper storage class are fixed after creation. Replicas per shard and Keeper replicas can change later.

Resource profiles

The Resources tab sets CPU and memory for the server pods and the Keeper pods. CPU limits are optional and empty by default.

Pod	CPU request	Memory request	Memory limit
ClickHouse	`250` m	`512` Mi	`2048` Mi
Keeper	`100` m	`256` Mi	`512` Mi

Resource profile sizes ClickHouse memory settings from the server memory limit. New databases use Small instance. Small instance and Production need a memory limit of at least 2048 MiB. Both keep 25% of the limit outside the ClickHouse server memory budget and cap the total query memory of the default user at 50% of the limit. CPU and memory requests stay as entered.

Setting	Small instance	Production
Per-query memory	12.5% of the memory limit	25% of the memory limit
Sort and `GROUP BY` spill to disk	after 50% of per-query memory	after 50% of per-query memory
Query threads	1	1 per 2 GiB, up to 8, capped by the CPU limit when set
Concurrent queries	1 per 512 MiB, up to 4	1 per 512 MiB, up to 32
Mark cache	10% of memory, up to 512 MiB	10% of memory, up to 5 GiB
Uncompressed cache	off	off
Block size	8192 rows	65536 rows
Query, metric, asynchronous metric, and trace logs	kept 1 day	kept 7 days

For a 2048 MiB Small instance, the server budget is 1536 MiB, the default user can use 1024 MiB across all queries, each query is limited to 256 MiB, and sorts spill to disk after 128 MiB. The total budget can reject queries before the concurrency limit is reached.

Per-query memory, the user total, the spill thresholds, query threads, and block size are set on the default settings profile. Settings profiles you create in SQL need their own limits. The server memory budget, concurrent queries, caches, and log retention apply to the whole server. Spilled data goes to the ClickHouse data volume, so keep free space for it. Select Operator defaults to run without these overrides.

TLS and external access

The Access tab has two TLS switches for traffic inside the cluster:

Enable TLS issues certificates for ClickHouse and Keeper with cert-manager.
Require TLS points the in-cluster services at the secure ports and turns off the plain protocols. Without it, clients in the cluster use plain HTTP on 8123 and the plain native protocol on 9000.

Turn on External Access to reach ClickHouse from outside the cluster. It turns Enable TLS on. Pick a Network, then set the HTTP port and the Native port. The defaults are 8123 and 9000, and the two ports must differ. Outside clients connect to the database hostname on these ports, always over TLS. The HTTPS interface is on the HTTP port and the secure native protocol is on the Native port. Custom load balancer source ranges are not supported.

The networks, hostname, and certificate work the same for every engine. See External access.

Once the certificate is issued, Verified connection on Overview shows the client command:

clickhouse-client --secure --host <hostname> --port <native port> --user default --ask-password

clickhouse-client verifies the certificate and hostname with the system trust store. TablePlus needs the system CA bundle in its SSL path field.

High availability

For a database that survives a single worker failure, select High availability on the Topology tab. It sets one shard, 3 replicas per shard, and 3 Keeper replicas. Run it on at least three workers. With the default sizes, three servers at 20 GiB and three Keepers at 10 GiB provision 90 GiB. Placement in one datacenter protects against a worker failure, not loss of the datacenter.

Replicas share row data only for tables in the ReplicatedMergeTree family. The Replicated default database synchronizes table definitions across replicas. It does not convert plain MergeTree tables into replicated ones. The Health card on Overview flags tables whose rows are not replicated.

Writes acknowledge on one replica by default. Set insert_quorum=2 to have a write acknowledged on two of three replicas, which keeps inserts flowing while one server restarts. Clients must reconnect after a connection drop and retry with a bound. Retry an insert only with a stable insert_deduplication_token for that insert, so a retry after a lost acknowledgement does not duplicate rows. The Kubernetes Service does not replay failed queries.

Upgrade ClickHouse and Keeper

Open Settings, then General, and pick a newer ClickHouse version.
Check the Upgrade checks card. Save Changes stays disabled until it reads Health checks passed.
Save, and wait for the upgrade to complete.
To upgrade Keeper, set Keeper version in a separate save. Keep current version leaves Keeper untouched. Keeper 26.8.8.8 is the only upgrade target.

Downgrades are rejected. An upgrade that spans 12 months of releases or more, for example 25.12 to 26.12, needs an intermediate version. A database on a version that has left the catalog keeps it when you save other settings.

The upgrade checks block the save when:

a server or Keeper replica is not ready, or the servers run mixed versions
replicas per shard is above 1 and a table is outside the ReplicatedMergeTree family
a replica is inactive, read-only, missing a table, more than 60 seconds behind, or has a replication queue older than 5 minutes
a server has less than 5% free disk
the database has more than 1000 tables
backups are on and there is no completed backup, or the last one is older than twice the backup interval

The database shows Updating while a version or configuration change applies. The operator restarts one server replica at a time and waits for it to become ready before moving to the next. During each restart the server waits up to 60 seconds for running queries to finish, within a 90 second pod termination window. Longer queries are cancelled, and new connections can still land on a pod that is shutting down, so clients need to reconnect and retry during an upgrade.

Backups

ClickHouse backups are full snapshots of the application databases, written with native BACKUP ... ON CLUSTER to a bucket you own. System databases, server configuration, and users and roles created in SQL are not included. There is no point in time recovery.

To turn on scheduled backups:

Open the database, select Settings, then the Backup tab. The same tab is in the create form.
Turn on Scheduled backups.
Select the integration under Backup storage.
Set Backup interval (hours), from 1 to 168. The default is 24.
Set Retention (days), from 1 to 365. The default is 7.
Save. The first backup starts at the next full hour (UTC), and later ones follow the interval.

Snapshots are stored under clickhouse/<organization-id>/<database-id>/backup-<uuid>/ in the bucket. Each snapshot keeps the retention that was set when it was taken. Retention removes expired snapshots unless a restore still uses them.

The schedule runs as a Kubernetes CronJob in your cluster and talks to ClickHouse directly, so scheduled backups keep running while Edka is unreachable. The snapshot list, Backup now, and retention cleanup resume when the connection returns. Settings cannot be saved while a scheduled backup is running.

The Backups tab shows the last backup, schedule, retention, and storage target. Below that, Backup Snapshots lists each snapshot with its time and short id, its status and failure message, its duration, and its expiry. Backup now takes an extra full backup to the same destination without shifting the schedule. It is available when backups are on, the database is ready, and no other backup is running. When backups are off, the tab shows Off and a Backup settings button.

Restore a snapshot

On the Backups tab, select Restore on a completed snapshot.
Keep or change the Name, which defaults to <source>-restored, and the Namespace.
Select Restore. Edka opens the new database, which becomes Ready once ClickHouse confirms the restore.

Restore always creates a separate database and leaves the source unchanged. The new database uses the snapshot’s ClickHouse version, replica and shard counts, and storage size. It gets a new default user password, default placement, and scheduled backups off.

To start a new database from a snapshot, choose it under Restore from backup on the Backup tab of the create form. The list includes snapshots from deleted databases, so this is also how to recover a database after deleting it. The form copies the snapshot’s version, topology, storage, memory limit, and resource profile. The version and the replica and shard counts must match the snapshot, and the storage size cannot be smaller. Organization owners and administrators also see snapshots taken in other clusters of the organization. For those, pick storage classes that exist in the target cluster.

Monitoring and alerts

Insights charts queries, inserted rows, CPU, and memory for one server or all servers, and reports connections, active queries, running merges, and disk used. The counts include internal queries and system tables. Replication and operator details lists replication health for each table. History needs VictoriaMetrics installed on the cluster.

The Health card on Overview reads Replication healthy, Needs attention, Inspection incomplete, or No local data tables. Edka raises a notification when the database needs attention.

With VictoriaMetrics installed, open Observability, then Alerts, then Packs, and install ClickHouse. Its eight rules cover an unreachable metrics endpoint, read-only replicas, replication lag and backlog, query failures, low disk space, and missing health metrics. Installing packs needs the Standard plan or higher. See Alert packs.

Troubleshooting

A restore failed. Edka does not retry a failed restore in place. Delete the failed database and start a new restore. A restore that ClickHouse does not confirm within 24 hours is marked failed.
Backups fail after rotating the integration’s access keys. Edka copies the keys into the ch-backup-<database-id> Secret when you save the database settings. Save a settings change to copy the new keys. Until then the Status column shows the storage error from ClickHouse.
A backup is marked failed after 24 hours. It could not confirm completion. Its objects stay in the bucket for investigation and retention does not delete them.
A scheduled backup did not start. The ch-backup-<database-id> CronJob in the database namespace runs the schedule. Check its Jobs:
Terminal window
```
kubectl get cronjob,jobs -n <namespace>
```