What actually happens to your applications when the Kubernetes control plane dies? In this demo we break a cluster on purpose: we delete the control plane instance directly from the Hetzner console, watch the impact on a live application, and then recover the cluster with an Edka rollback.
The setup
The cluster is intentionally simple:
- one control plane node, which also runs the embedded etcd datastore
- two worker nodes
- Metabase connected to a Postgres database, serving real traffic
To measure the impact, three things run during the entire test:
- Siege sends two concurrent users to the Metabase endpoint and reports availability
kubectlwatches the cluster state- a log stream from one of the worker nodes shows what the node sees
Deleting the control plane
We delete the control plane instance from the Hetzner console. Since this node runs etcd, the entire cluster state goes with it: every Deployment, Service, Secret, and certificate.
The effect is immediate:
kubectlstops responding- the worker node logs fill up with connection errors
- Metabase keeps serving traffic
That last point surprises people. Worker nodes do not need the control plane to keep running the pods they already have. Containers stay up, existing Service routing keeps working, and the load balancer keeps sending traffic to healthy workers.
What you lose is control. No deployments, no scaling, no rescheduling if a pod or node fails, no new certificates. The cluster is frozen in its last known state, and every minute it stays frozen increases the risk that a routine failure turns into real downtime.
Recovery without backups
If there is no backup of the etcd state, there is no way to rebuild the control plane as it was. The usual path is to provision a new cluster and migrate the workloads. That can be done safely, but it takes time, from hours to days depending on how much state and configuration your cluster carries.
Rolling back with Edka
Every Edka cluster can upload scheduled etcd backups. Each backup set contains an encrypted etcd snapshot, the PKI bundle required for recovery, and restore metadata. With that in place, a lost control plane becomes a rollback instead of a rebuild.
From the cluster’s backup list we pick the latest backup set and choose Roll Back Cluster. Edka detects that the control plane instance is missing, provisions a replacement, and restores the cluster state from the backup.
Backup sets are encrypted before upload. This cluster uses a customer-provided AGE key, so the rollback asks for the matching AGE private key. The key is used only for this operation and is not stored.
In this demo, the cluster API is reachable again about 90 seconds after the rollback starts. The worker nodes reconnect on their own, kubectl works again, and Siege reports uninterrupted availability for the application across the entire test.
Practical notes
A few things worth knowing before you rely on this:
- A rollback restores the state captured in the selected snapshot. Anything changed after that snapshot is lost, so your backup frequency defines your recovery point. Edka supports snapshots every 24 hours, or every 6 hours on paid plans. Pro plan can opt for hourly snapshots. You can also run on-demand backups at any time.
- This demo uses a minimal cluster. Highly dynamic clusters can require restarting some components after a rollback to get everything fully in sync.
- The same backup sets also cover other failure modes: restoring a deleted cluster or cloning a cluster for testing. See Cluster Backups and Restore for the full picture.
Try it yourself
You can enable etcd backups when creating a cluster or later from Cluster Settings → Backup. If you want customer-controlled encryption, add your own AGE public key in the backup settings.
Breaking a test cluster on purpose is a cheap way to find out what your recovery actually looks like. We recommend trying it before you need it.