Encrypted Local Storage

A node pool with encrypted local storage splits the disk of each server in two. A root partition of a fixed size holds the operating system. The rest of the disk is a partition encrypted with LUKS2, and volumes of the local-path-encrypted storage class live on it.

Edka stores the disk key of each node. The node does not keep it, so the partition is locked after every boot until Edka unlocks it. Each node also gets a recovery key encrypted to the cluster’s age public key. With the matching age identity, you can unlock a node yourself.

Before you start

• The pool is a worker pool on Hetzner Cloud, a metal pool or a pool of existing servers. Encrypted local storage is not available on satellite clusters.
• The pool keeps a fixed size. Autoscaling is not available for pools with encrypted local storage.
• The server type has room for both partitions. The encrypted partition gets what is left of the disk after the root partition.
• You have an age key pair for the cluster’s recovery key. See the next section.

Create the recovery key

The recovery key is an age key pair. The quickest way to get one is Generate & download key next to Recovery on the pool form. Your browser creates the pair, downloads the identity file and fills in the public key. Edka keeps no copy of the identity, so store the file where you can reach it without Edka, for example in your password manager.

To create the pair yourself, install age, then run:

age-keygen -o edka-recovery-key.txt

The command prints the public key, which starts with age1. That is the key you give Edka. The file holds the identity that decrypts the recovery keys of the nodes. Edka never receives it, so keep it where you can reach it without Edka, for example in your password manager.

To print the public key of an identity file again:

age-keygen -y edka-recovery-key.txt

Set the recovery key

A cluster needs a recovery key before its first pool with encrypted local storage.

On a new cluster, set Disk to Encrypted local storage on a worker pool of the create form, and paste the public key into Recovery. Edka saves it with the cluster.

On an existing cluster, set it from the page of the new pool:

1. Open the cluster, go to Settings → Node Pools and click Add Node Pool.
2. Set Disk to Encrypted local storage.
3. Paste the public key into Recovery and click Save key.

Edka saves the key for the cluster right away, before you apply the pool. The review does not apply a new pool with encrypted local storage until the cluster has a saved key.

The recovery key is fixed once the cluster has a pool with encrypted local storage, because every node’s recovery key is encrypted to it. Until the first such pool is created, you can still change it in Recovery. To use another key later, delete the pools with encrypted local storage and set the new key when you create the next one.

Create a pool with encrypted local storage

1. Open the cluster, go to Settings → Node Pools and click Add Node Pool.
2. Set up the pool as for any other pool: provider, name, location, and the server type or the servers.
3. Set Disk to Encrypted local storage.
4. Set Root, the size of the root partition in GiB. The default is 40 and the minimum is 30. The root partition holds the operating system, container images and container logs.
5. Check Recovery. If the cluster has no recovery key yet, paste the public key and click Save key.
6. Set the size, labels and taints.
7. Click Review changes. The review lists the pool with Encrypted local storage and its root size. Click Apply to create it.

Disk and Root are fixed once the pool exists. Its page shows Encrypted local storage and the root size. To change either, create another pool and move the workloads to it.

The cluster gets the local-path-encrypted storage class with its first pool with encrypted local storage. Edka installs the Local Path Storage Class add-on for it when the cluster does not have it yet.

Use the local-path-encrypted storage class

Name the storage class in a PersistentVolumeClaim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
    name: data
spec:
    storageClassName: local-path-encrypted
    accessModes:
        - ReadWriteOnce
    resources:
        requests:
            storage: 10Gi

The storage class is not the cluster default. Pick it in each claim, or in the storage class field of a database, app or deployment.

• The volume lives on one node. Kubernetes creates it on the node where the first pod that uses it is scheduled, and only nodes of pools with encrypted local storage qualify. These nodes carry the label storage.edka.io/local-encrypted=true. From then on, every pod that uses the volume runs on that node.
• Pods need to fit on those nodes. When the pool has taints, the pods that use the volume need tolerations for them.
• The size is not a limit. All volumes on a node share its encrypted partition. A claim’s size cannot be raised later.
• Deleting the claim deletes the data. The storage class reclaims volumes with Delete.

PostgreSQL

To keep a PostgreSQL database on encrypted local storage:

1. On General, set Storage class to local-path-encrypted. With the WAL volume on, WAL storage class can use it too.
2. Check that the cluster has at least as many nodes with encrypted local storage as the Instance count. Each instance runs on its own node.
3. If the pool with encrypted local storage has taints, pick it on Placement as the Node pool and keep Tolerate node pool taints on. A database pinned to a pool needs that pool to have encrypted local storage.

Edka refuses a database that cannot fit and says which of these is missing.

The storage class is fixed after creation, and Storage size (Gi) cannot grow on local-path-encrypted. To move an existing database, create a new one on local-path-encrypted with Clone an existing cluster on the Import tab.

After a reboot

The encrypted partition is locked after every boot of the node: an OS update round, Update and reboot, or a reset from the Hetzner Console. Edka unlocks it once the node reports its new boot.

While the partition is locked:

• Pods with local-path-encrypted volumes on the node cannot start. Once the partition is unlocked, Edka restarts the pods that wait for it when a controller such as a Deployment or a StatefulSet owns them. Other pods start when the kubelet retries their volume mounts.
• Explorer → Nodes marks the node disk locked. Hover the mark to see the last error, or open the node’s Disk tab.
• Edka retries the unlock every minute. After 10 minutes, it sends a critical notification, Encrypted disk on <node> is locked, with the last error, and retries every 5 minutes from then on.

OS update rounds

An OS update round waits for the partition of a rebooted node to be unlocked before it moves to the next host. If the partition is still locked 20 minutes after the reboot, the node is Blocked and the round stops, so no other node reboots into a locked disk. The remaining hosts are not updated. Unlock the node, then run Update OS for the rest.

The Disk tab

Open the cluster’s Explorer → Nodes, click a node and open its Disk tab. Nodes of pools with encrypted local storage say encrypted disk in the list. The tab shows:

• Partitions: the root partition and, on a node of a pool with encrypted local storage, the encrypted partition, each with its size, usage and mount point. The encrypted partition’s usage shows once it is unlocked.
• Volumes: the volumes on the node’s disk (local-path, local-path-encrypted) and the network volumes attached to it, such as Hetzner Cloud Volumes, with where each lives and the pods on the node that use it. A pod marked waiting has not started yet.
• Disk recovery key: the age public key every node’s recovery key is encrypted to.

While the encrypted partition is locked, the tab starts with how long it has been locked, why Edka could not unlock it, and the ways to unlock it.

Usage comes from the cluster’s monitoring.

Unlock a node from the console

While the partition is locked, the node’s Disk tab offers two ways to unlock it. Both need cluster write access, and Edka has to reach the node over SSH.

Unlock now has Edka send the node’s disk key right away instead of at the next retry. Use it once whatever kept Edka from unlocking the node is fixed.

Unlock with the recovery key is for when the stored disk key is missing or no longer opens the partition:

1. Paste the age identity into the field, or click Choose file and pick the identity file.
2. Click Unlock.

The identity stays in your browser. It decrypts the node’s recovery key there, and Edka passes only that key to the node, once, without storing it.

When Edka cannot reach the node, click Unlock over SSH when Edka cannot reach the node for the command to unlock it by hand with the node’s address filled in.

Unlock a node by hand

Unlock a node yourself when Edka cannot reach it, or when the cluster no longer connects to Edka. You need:

• the age identity file of the cluster’s recovery key.
• the cluster SSH key from the SSH Access card. See Access & Security.
• age and jq on your machine.
• SSH access to the node as root. A node without a public IP, for example behind a NAT gateway, needs a machine that reaches the cluster’s private network.

Run:

ssh -i ssh-key-<cluster-name>.pem root@<node-ip> \
  'cryptsetup token export --token-id 0 "$(blkid -U "$(cat /etc/edka/local-disk.uuid)")"' \
  | jq -r .recovery | base64 -d \
  | age -d -i edka-recovery-key.txt \
  | ssh -i ssh-key-<cluster-name>.pem root@<node-ip> /usr/local/sbin/edka-local-unlock

The first command reads the node’s encrypted recovery key from the partition header. age decrypts it with your identity, and the last command unlocks the partition with it.

The command prints unlocked, or already_unlocked when the partition was open, followed by the node’s boot ID. The pods that wait for the partition start when the kubelet retries their volume mounts. To start one right away, delete it and let its controller create it again:

kubectl delete pod -n <namespace> <pod>

The partition locks again at the next boot.

Remove nodes

Scaling a pool down, deleting the pool and deleting the cluster all remove nodes. Before Edka removes a node, it erases the node’s encrypted partition, recovery keyslot included, when it can reach the node, and then deletes the node’s disk key. Copy any data you want to keep off the node’s volumes first.

Servers of metal pools and pools of existing servers keep their disks after they leave the cluster. The erase is what makes the old partition unreadable there.

Troubleshooting

The pool page says encrypted local storage is not available on this cluster yet. Contact support to enable it for the cluster.

A pod is stuck in ContainerCreating and its events show a mount error for a local-path-encrypted volume, such as path does not exist. The partition of the pod’s node is locked. Check the node in Explorer → Nodes:

kubectl describe pod -n <namespace> <pod>

A new claim stays Pending with create process timeout after 120 seconds. The node picked for the volume is locked. The provisioner retries until the claim binds.

A deleted claim leaves its volume Released with VolumeFailedDelete. The volume’s node is locked. The provisioner deletes the data once the partition is unlocked.

A claim stays Pending and no node is picked. The cluster has no Ready node with encrypted local storage, or the pod’s tolerations and node selectors keep it off those nodes. List the nodes that can hold the volume:

kubectl get nodes -l storage.edka.io/local-encrypted=true

A node stays disk locked. Open the node’s Disk tab in Explorer → Nodes, or the notification, to read the last error. Then unlock it from the console, or by hand when Edka cannot reach it.

Related

• Update a Cluster: managing node pools.
• Node OS Updates: update rounds and reboots.
• PostgreSQL: storage and placement of a database.
• Explorer: the Nodes list and node tabs.