Node OS Updates

Cluster nodes and NAT gateway members install security updates every day. Fixes to the kernel, libc, OpenSSL or systemd only take effect after a reboot, so Edka runs update rounds that update and reboot these hosts one at a time. Rounds run when you start one, with the hosts and the order you pick, or on their own in the cluster’s maintenance window.

Path: Clusters -> your cluster -> Settings -> Maintenance

Availability

OS updates run on clusters created with Debian 13, Ubuntu 24.04 or Ubuntu 26.04, on cloud nodes, metal pools and pools of existing servers. They are not available on satellite clusters, whose server you update yourself, or on clusters created with another image.

Defaults

Setting	Default
Security updates	On
Reboots	Manual
Maintenance window	Tuesday to Thursday, 03:00 to 06:00

The window uses the time zone of the cluster’s location:

Location	Time zone
Falkenstein (fsn1), Nuremberg (nbg1)	Europe/Berlin
Helsinki (hel1)	Europe/Helsinki
Ashburn (ash)	America/New_York
Hillsboro (hil)	America/Los_Angeles
Singapore (sin)	Asia/Singapore

Host OS status

Maintenance lists every node and NAT gateway member with its role, kernel and one of these states:

• Up to date: no pending updates and no pending reboot.
• N updates: packages waiting to install, with the number of security updates among them.
• Reboot pending since: installed updates that take effect after a reboot. Hover the state to see which packages need it.
• Updating: the host is part of the running round.
• Not reported: the host has not published its status yet.

NAT gateway members show the role NAT gateway. In a gateway with two members, the role also says which one is active, the member that carries the outbound traffic of the private nodes, and which one is on standby.

Each host works out its status after boot, every hour, and after the daily package update. A node publishes it on its Node object, and a NAT gateway member’s agent sends it to Edka with the gateway status, every minute. The Nodes view in Explorer marks nodes with a pending reboot or an update in progress.

How a round works

A round updates its hosts one at a time, in its order. A round you start covers the hosts you pick, in the order you set. An automatic round covers the hosts with pending updates or a pending reboot, in the default order:

1. Control-plane nodes.
2. Workers, by pool.
3. NAT gateway members, the active member last, so the outbound traffic moves once.

These rules hold in every order:

• One host updates at a time, so at most one control-plane node is down and etcd keeps its quorum. A control-plane node starts only while every control-plane node is Ready.
• Before a node’s drain, Edka waits up to an hour for the PodDisruptionBudgets of its pods to allow a disruption. The node stays schedulable while it waits, and is skipped after the hour.
• Before the active NAT gateway member updates, the outbound traffic moves to another healthy member. Open connections through the gateway break when it moves. A gateway with a single member stops outbound traffic for about a minute while that member reboots.
• On a cluster with a NAT gateway, cloud workers wait until the cluster keeps their route through the gateway across a reboot. Edka sets that up on its own.
• A node that fails after its reboot, or a NAT gateway member whose update fails, stops the round. The remaining hosts are not updated.

For each node, Edka:

1. cordons the node and evicts its pods. PodDisruptionBudgets apply, DaemonSet pods stay, and emptyDir data is deleted.
2. installs all pending package updates.
3. reboots the node when an update needs it. A node that reached the internet before the reboot has to reach it again within five minutes after it.
4. makes the node schedulable again once it is Ready.

For a NAT gateway member, Edka installs the pending updates, reboots the member when an update needs it, and checks that it forwards traffic and reaches the internet again.

The cluster autoscaler does not remove a node while it is in a round. A node that was cordoned before the round stays cordoned after it.

A cluster with one control-plane node loses the Kubernetes API for a minute or two while that node reboots. Workloads on the workers keep running.

Automatic updates

To turn on automatic updates, set Reboots to Automatic, in the maintenance window and click Save settings. Edka then starts a round once a node or NAT gateway member has needed a reboot, or had security updates waiting, for more than a day. After a round finishes, the next automatic round waits at least a day.

A node with a fix waiting for a known exploited vulnerability gets a round in the next window without the one-day wait.

Nodes only start updating inside the window. A node that is updating when the window closes finishes, and the rest of the round continues in the next window. In an automatic round, a worker whose pods have no room on the other nodes is skipped, and the round shows which pods held it back.

To change the window, set Days, Start, Length (1 to 12 hours) and Time zone under Maintenance window, then click Save settings. A window ends by midnight. A round that waits for the window follows the new one.

Update the OS now

1. Open Settings -> Maintenance and click Update OS.
2. Every host with updates is picked, in the default order. Clear the hosts to leave out, and move the others up or down to set the order. Default order puts the picked hosts back in the default order.
3. Check What happens for each host. A node also shows how many CVEs its update fixes, from its vulnerabilities. A host shows a note when pods have no room on other nodes and wait for it to come back, when pods without a controller would stop its drain, when the Kubernetes API or the outbound traffic pauses while it reboots, or when the outbound traffic moves twice because the active NAT gateway member comes before the standby.
4. Click Start update and confirm with your passkey if asked.

A host that cannot be updated now is listed under the others with the reason, and cannot be picked:

• the node has not reported its OS status, or is not Ready.
• the worker reaches the internet through the NAT gateway, and the cluster does not keep that route across a reboot yet.
• the NAT gateway member carries the outbound traffic, and no other member is healthy enough to take it over.

Hosts that are up to date are counted at the bottom.

Maintenance shows the round with each host’s state: Waiting, Updating, Done, Blocked or Skipped. When every host is done, the round shows under Last update.

Update and reboot one host

1. Open Settings -> Maintenance.
2. Click Update and reboot, the icon at the end of the host’s row. For a node, the dialog lists the pods that move and any that would stop the drain. For the active NAT gateway member, it names the member the outbound traffic moves to.
3. Click Update and reboot.

A node goes through the same steps as in a round: drain, package updates, reboot and uncordon. It reboots even when no update needs it. A node can be rebooted once it has reported its OS status, is Ready, and no round is running on the cluster. Explorer -> Nodes has the same action for a node.

Cancel a round

Click Cancel update on the running round. Hosts that have not started are skipped. A node that is updating finishes first, so it is not left drained.

Turn off security updates

Turn off Security updates to stop the daily install of security updates on every node. Update rounds still install all pending updates on the nodes they cover.

Notifications

• A manual cluster with a node or NAT gateway member that has needed a reboot for more than seven days, with the number of CVEs the reboot fixes.
• A round that could not update a node, with the reason.
• A round that stopped because a node did not come back after its reboot, or came back and its update failed, for example without internet access, or because a NAT gateway member’s update failed.

Troubleshooting

A node is blocked because its pods could not be moved. The drain waits up to 15 minutes. A PodDisruptionBudget that allows no disruptions, or a pod without a controller, holds it back. Check the budgets and the bare pods on the node:

kubectl get pdb -A
kubectl get pods -A --field-selector spec.nodeName=<node> -o wide

Edka uncordons the node and moves on. Run Update OS again once the pods can move.

The update did not start. The rounds run as the os-update-server and os-update-agent Plans of the system-upgrade-controller:

kubectl -n system-upgrade get plans,jobs,pods

A node did not come back. When a node is not Ready 10 minutes after its reboot started, 20 minutes on a metal node, Edka resets the server once. If it is still not back, the round stops and the remaining nodes are not updated.

A node has no internet access after its reboot. The round stops, the remaining nodes are not updated, and the node stays cordoned so no pods land on it. The node’s reason in the round shows the default route and nameservers it came back with. Once the node reaches the internet again, make it schedulable and run Update OS for the remaining nodes:

kubectl uncordon <node>

A node’s update failed after its reboot. The round stops the same way and the node stays cordoned. The reason shows the last line the update wrote.

A node shows Not reported. The status comes from the node’s edka.io/os-status annotation, which the edka-os-status.timer systemd timer on the node writes:

kubectl get node <node> -o jsonpath='{.metadata.annotations.edka\.io/os-status}'

A NAT gateway member shows Not reported. The member’s agent reads the status from /var/lib/edka/os-update/status.json, which the edka-os-status.timer systemd timer on the member writes. Edka installs both with the OS update policy, which it sends to every cluster on its own. The member’s status in Settings -> NAT shows whether its agent reports.

The active NAT gateway member cannot be updated. The outbound traffic moves only to a healthy member. Check the other member in Settings -> NAT. A member that is updated in the same round reports healthy within a minute or two, and the round waits for it.