Cluster Alerts
Use Alerts inside Clusters -> Observability to manage cluster-scoped metric alerts.
The Alerts workspace stores alert definitions in Edka and evaluates them from the cluster metrics store. Firing alerts feed into the same global Notifications system as built in cluster, Kubernetes, add-on, and storage checks.
Requirements
Section titled “Requirements”Custom alerts require VictoriaMetrics on the cluster. If VictoriaMetrics is not installed, you can still open the Alerts tab, but rules wait for the metrics store before they can evaluate.
Edka Agent evaluates saved rules inside the cluster and reports the latest series state back to Edka.
Some alert packs have extra metric requirements:
- workloads, cluster health, and storage packs use kube-state-metrics, cAdvisor, or kubelet metrics
- PostgreSQL packs use CloudNativePG metrics
- ClickHouse packs use ClickHouse exporter metrics
- Valkey packs use the built-in Valkey exporter, collected through the existing VictoriaMetrics annotated-pod scrape
- NAT Gateway packs use Edka NAT Gateway metrics
- host infrastructure packs use Node Exporter metrics
- the certificates pack uses cert-manager certificate metrics, collected through the VictoriaMetrics annotated-pod scrape
- the External Secrets pack uses External Secrets Operator metrics, collected through the VictoriaMetrics annotated-pod scrape
Alert rules
Section titled “Alert rules”Open Alerts -> Rules to create, inspect, edit, or delete custom alert rules.
Each rule includes:
- PromQL query
- comparator and threshold
forduration- keep firing for: how long a firing alert waits without a match before it resolves, 2 minutes by default
- evaluation interval
- severity:
critical,warning, orinfo - labels and annotations
- optional routing labels such as
teamorowner
Rules can be enabled or disabled without deleting them. Disabled rules remain saved but are not evaluated.
Test before saving
Section titled “Test before saving”Use Test rule from the editor to run the PromQL query and threshold against VictoriaMetrics before saving.
The test result shows:
- how many series were returned
- how many series matched the comparator and threshold
- the labels, values, and per-series match status for the returned series (up to the first several)
This is useful for checking metric names, label filters, and threshold units before the rule starts producing notifications.
Alert states
Section titled “Alert states”The Rules view summarizes current evaluation state:
- OK: enabled rules with no matching alert series
- Pending: rules with matching series that have not reached their
forduration yet - Firing: rules whose matching series reached the configured
forduration - Disabled: saved rules excluded from evaluation
- Not evaluated: rules that have not reported a state yet
Rule rows also show the last evaluation time, matching series count, and any evaluation error reported by the evaluator.
Delivery
Section titled “Delivery”Alerts reach the delivery channels whose cluster, severity, and label filters match. Channels are organization-wide and live in Settings -> Notifications -> Channels. See Platform Notifications.
Admins see a Delivery line at the top of Alerts -> Rules:
- the active channels that receive this cluster’s alerts, with Manage to edit them in Settings
- a warning for each channel whose severity or label filters match none of the enabled rules
- Add channel when no channel covers the cluster, which opens a new channel in Settings with this cluster selected
Alert packs
Section titled “Alert packs”Open Alerts -> Packs to install predefined rules as editable cluster alert rules.
Current packs include:
- Workloads: availability of Deployments, StatefulSets, and DaemonSets, stalled rollouts, and pod restarts
- Postgres: connection pressure, replication lag, blocked backends, and long transactions for CloudNativePG clusters
- ClickHouse: replica availability, replication health, query failures, and disk pressure
- Valkey: availability, memory and connection pressure, replication, persistence failures, and missing core metrics
- Storage: unbound claims and high or critical persistent volume claim usage
- Cluster Health: node readiness, pods that fail, cannot start, or stay pending, and container resource alerts
- NAT Gateway: conntrack table capacity on NAT gateway members. NAT gateway health and failovers notify without a pack.
- Gateway: Envoy Gateway proxy availability, 5xx rate, p95 latency, rate limiting, backend health per route, upstream timeouts, TLS handshake failures, and certificate expiry
- Host Infrastructure: host root filesystem usage, inode usage, sustained CPU, encrypted local storage usage, IO pressure and degraded software RAID alerts based on Node Exporter metrics
- Certificates: overdue renewals and upcoming expiry for every cert-manager certificate in the cluster
- External Secrets: synced secrets that fail to sync and secret stores that can’t connect to their provider
Pack rules are installed as normal custom rules. You can edit or delete them
after installation. They do not count toward your plan’s alert rule limit,
which applies only to rules you create yourself. Labels starting with
edka.io/ mark pack rules and are managed by Edka.
Installing packs from this tab requires the Standard plan or higher. Packs Edka installs on its own are included on every plan: Workloads, Cluster Health, and Storage once VictoriaMetrics is installed, Postgres when a database becomes ready, Host Infrastructure with Node Exporter, Certificates once VictoriaMetrics and cert-manager are both installed, and External Secrets once VictoriaMetrics and the External Secrets add-on are both installed.
A rule marked off by default is installed disabled. Turn it on from Alerts -> Rules.
If a pack is already partially installed, Edka can install the missing rules. Rules with name conflicts are skipped instead of overwriting unrelated custom rules.
Workloads alert pack
Section titled “Workloads alert pack”Edka installs Workloads once VictoriaMetrics is on the cluster. It adds nine editable rules based on kube-state-metrics. They evaluate once a minute, keeping each Deployment, StatefulSet, DaemonSet, pod, and Job separate.
| Rule | Condition | Duration | Severity |
|---|---|---|---|
| Deployment has no ready replicas | A Deployment that wants at least one replica has none ready | 3 minutes | Critical |
| Deployment has unavailable replicas | Some replicas are unavailable while at least one is ready | 5 minutes | Warning |
| Deployment rollout is stalled | The observed generation is behind the metadata generation | 10 minutes | Warning |
| StatefulSet has no ready replicas | A StatefulSet that wants at least one replica has none ready | 3 minutes | Critical |
| StatefulSet has unavailable replicas | Fewer replicas are ready than wanted while at least one is ready | 5 minutes | Warning |
| DaemonSet has no ready pods | A DaemonSet scheduled on at least one node has no ready pod | 3 minutes | Critical |
| DaemonSet has unavailable pods | Some pods are unavailable while at least one is ready | 5 minutes | Warning |
| Pod restart spike | A pod restarted more than 3 times in the last 15 minutes | 5 minutes | Warning |
| CronJob run failed | A Job created by a CronJob reached its Failed condition | 1 minute | Critical |
A workload that is fully down matches only its “no ready” rule, so one outage
produces one alert. Rollouts shorter than the for duration produce none.
The CronJob alert clears when the failed Job is removed, which happens once
newer runs replace it. Change a duration or disable a rule from
Alerts -> Rules.
Cluster Health alert pack
Section titled “Cluster Health alert pack”Edka installs Cluster Health once VictoriaMetrics is on the cluster. It adds six editable rules based on kube-state-metrics and cAdvisor. The two container resource rules are off by default because their thresholds are absolute and large workloads exceed them on purpose.
| Rule | Condition | Duration | Severity | Default |
|---|---|---|---|---|
| Node not ready | A node reports NotReady or Unknown | 5 minutes | Critical | On |
| Pod failed phase | A pod is in the Failed phase | 5 minutes | Warning | On |
| Pod stuck pending | A pod is in the Pending phase without a container error: unschedulable or a volume wait | 5 minutes | Warning | On |
| Pod cannot start | A container or init container waits on CrashLoopBackOff, an image pull, or a container error | 3 minutes | Critical | On |
| Container CPU usage high | A container uses more than 900 millicores | 10 minutes | Warning | Off |
| Container memory usage high | A container working set is above 1.5 GiB | 10 minutes | Warning | Off |
Storage alert pack
Section titled “Storage alert pack”Edka installs Storage once VictoriaMetrics is on the cluster. It adds three editable rules based on kube-state-metrics and kubelet volume statistics. The two usage tiers do not overlap, so a claim fires one of them.
| Rule | Condition | Duration | Severity |
|---|---|---|---|
| Persistent volume claim unbound | A claim is Pending or Lost | 5 minutes | Warning |
| Persistent volume usage high | A claim is between 85% and 95% usage | 10 minutes | Warning |
| Persistent volume usage critical | A claim is above 95% usage | 5 minutes | Critical |
Gateway alert pack
Section titled “Gateway alert pack”Install Gateway from Alerts → Packs to add ten editable rules for Envoy Gateway entry points. They use the Envoy proxy metrics VictoriaMetrics already collects and evaluate once a minute, keeping each gateway class separate. Only customer-facing listeners count as traffic; readiness and stats probes do not.
| Rule | Condition | Duration | Severity |
|---|---|---|---|
| Gateway proxy is not live | An Envoy proxy reports itself as not live | 3 minutes | Critical |
| Gateway 5xx rate is high | More than 5% of requests return 5xx, with at least 60 requests in 5 minutes | 5 minutes | Critical |
| Gateway p95 latency is high | p95 latency above 2 seconds, with at least 60 requests in 5 minutes | 10 minutes | Warning |
| Gateway is rate limiting clients | More than 10% of requests get 429 from the class rate limit, with at least 60 requests in 5 minutes | 10 minutes | Warning |
| Route has no healthy backends | Every backend behind an HTTPRoute fails health checks | 3 minutes | Critical |
| Route has unhealthy backends | At least one backend behind an HTTPRoute fails health checks | 10 minutes | Warning |
| Gateway upstream timeouts | Backend timeouts above 1 per second | 5 minutes | Warning |
| Gateway TLS handshake failures | TLS handshake failures above 1 per second | 5 minutes | Warning |
| Gateway certificate expires within 14 days | Soonest-expiring certificate served by the class is under 14 days from expiry | 1 hour | Warning |
| Gateway certificate expires within 3 days | Soonest-expiring certificate served by the class is under 3 days from expiry | 1 hour | Critical |
The 5xx and latency rules read 0 for a quiet gateway class instead of firing on a handful of requests.
Certificates alert pack
Section titled “Certificates alert pack”Edka installs Certificates once VictoriaMetrics and cert-manager are both on the cluster. It adds two editable rules that cover every cert-manager certificate, whichever issuer and validation method it uses. They evaluate every 5 minutes, keeping each certificate separate.
| Rule | Condition | Duration | Severity |
|---|---|---|---|
| Certificate renewal overdue | cert-manager was due to renew the certificate over 24 hours ago | None | Warning |
| Certificate expires within 7 days | A certificate whose renewal is overdue expires within 7 days | None | Critical |
cert-manager renews a certificate at its renewal time, by default a third of its lifetime before it expires, and sets a new renewal time once the renewal succeeds. A renewal time in the past means renewals are failing, so the warning arrives weeks before the certificate expires. The certificate keeps working until then. The Events tab of the Certificate and its Challenges in Explorer show why the renewal fails.
External Secrets alert pack
Section titled “External Secrets alert pack”Edka installs External Secrets once VictoriaMetrics and the External Secrets add-on are both on the cluster. It adds two editable rules that evaluate once a minute, keeping each synced secret and each store separate. They cover stores and synced secrets created outside Edka too.
| Rule | Condition | Duration | Severity |
|---|---|---|---|
| Secret sync failing | A synced secret fails to sync | 10 minutes | Warning |
| Secret store not connected | A secret store can’t connect to its provider | 10 minutes | Warning |
A synced secret that fails to sync keeps its last values, and pods that read it get no new values until a sync succeeds. A synced secret that never synced has no Secret, so pods that need it can’t start. A store that is not connected stops updating every synced secret that uses it. The error message on the cluster’s Secrets page shows the cause. See External Secrets.
Valkey alert pack
Section titled “Valkey alert pack”Install Valkey from Alerts → Packs to add ten editable rules. They use the existing Valkey exporter and evaluate once a minute, keeping each database and pod separate.
| Rule | Condition | Duration | Severity |
|---|---|---|---|
| Metrics endpoint unreachable | Discovered exporter cannot be scraped | 2 minutes | Critical |
| Server unavailable | Exporter cannot connect to Valkey | 2 minutes | Critical |
| Memory usage high | Reported memory usage above 90% of nonzero maxmemory | 10 minutes | Warning |
| Connection saturation | Connected clients above 80% of maxclients | 5 minutes | Warning |
| Rejecting connections | New rejected connections in the preceding 5 minutes | 2 minutes | Critical |
| Replica disconnected | Replica link to primary is down | 2 minutes | Critical |
| Replication traffic stalled | Connected replica has received no primary traffic for over 30 seconds | 5 minutes | Warning |
| Snapshot save failed | Latest background RDB save failed | 2 minutes | Critical |
| AOF write failed | Latest AOF write failed while AOF is enabled | 2 minutes | Critical |
| Core metrics missing | Reachable exporter lacks server status, or an online server lacks core metrics | 10 minutes | Warning |
Replication rules follow the reported role, including after promotion. Traffic inactivity measures the replication link, not replay delay. AOF write alerts apply only when AOF is enabled.
Memory usage is compared with Valkey’s configured maxmemory, not the container
memory limit. Adjust the threshold for caches intentionally kept near capacity.
Eviction rates and cache hit ratios have no default alerts because their expected
values depend on the workload.
Endpoint alerts cover discovered targets. Use Cluster Health for pod and node failures and Storage for PVC pressure. The Valkey pack does not monitor Sentinel quorum or object-storage backup success; its RDB rule reports local snapshot failures. Use the database Backups tab to inspect stored backups and restore a snapshot to test one.
Node Exporter and host alerts
Section titled “Node Exporter and host alerts”The host infrastructure pack depends on the Node Exporter add-on and VictoriaMetrics.
When node monitoring is enabled for VictoriaMetrics, Edka installs the Node Exporter add-on and configures VictoriaMetrics to scrape host metrics. Once both Node Exporter and VictoriaMetrics are installed, Edka can install the host infrastructure alert pack automatically.
Host alerts cover node-level pressure that Kubernetes object state alone cannot show:
| Alert | Fires when | For | Severity |
|---|---|---|---|
| Host root filesystem usage high | A node’s root file system is above 85% used | 10 minutes | Warning |
| Host inode usage high | A node’s root file system is above 85% of its inodes | 10 minutes | Warning |
| Host CPU usage high | A node’s CPU is above 90% busy | 15 minutes | Warning |
| Host RAID array degraded | A software RAID array on a node is missing a drive | 5 minutes | Critical |
| Encrypted local storage usage high | A node’s encrypted local storage is above 85% used | 10 minutes | Warning |
| Host IO pressure high | Every task on a node waited on IO for more than 20% of the time | 15 minutes | Warning |
A degraded array keeps running on the drives it has left, and the next drive failure loses its data. Replace the failed drive in Hetzner Robot, then add it back to the array.
Notifications
Section titled “Notifications”Firing custom alert rules appear in global notifications with the severity and labels from the rule. Once a firing series has gone without a match for the rule’s keep firing for period, the notification occurrence resolves and remains available in notification history until retention expires. A value that dips below the threshold for one evaluation and comes back keeps the alert firing instead of resolving it and firing it again.
Read/unread state is separate from alert resolution.
A rule that cannot be evaluated cannot fire, so Edka raises a notification for the cluster when:
- a rule’s query returns an error, with the last error message
- rules have not been evaluated for three of their intervals, and at least 5 minutes, while Edka Agent is still reporting
When Edka Agent itself stops reporting, its own notification covers the cluster instead.