Skip to content

Provisioning the cluster

Following the steps of tinkerbell and elsa, raya is built on top of Fedora CoreOS. Mostly for reducing costs, it is deployed on Hetzner Cloud.

Building a CoreOS image for Hetzner

While Hetzner Cloud is a great provider, it sadly does not provide a CoreOS image its VMs can boot from. This forces us to build one ourselves, and we do so using Packer.

The Packer template we use is image/fcos.pkr.hcl. Because Fedora CoreOS provides artifacts for Hetzner for its releases, the template mostly consists of downloading them and burning them onto a throwaway server’s disk (image/install.sh) using dd. As a consequence, CoreOS itself never boots during the build, which is why Ignition still runs on first boot of every server made from the snapshot. Once the disk is written, a snapshot is published, with labels allowing us to use it from a Terraform configuration. This is what the hcloud_image.fcos data source is used for (declared in cluster.tf).

image/build.sh is a script that can be used to build the image iff it does not already exist. If it does, calling the script returns without building anything.

Which image is burnt is decided by the image/fcos.pin file, which specifies the stream (among stable, testing, or next, we use stable for now), version and sha256 hash of the image. image/fcos.pkr.hcl, image/build.sh and cluster.tf all read it, making it the source of truth everybody trusts.

Once the image is burnt onto the disk, the latter is mounted and k3s is copied in /usrlocal/bin, ready to be used. Which versioned is provisioned is decided by the image/k3s.pin, using the same logic applied to image/fcos.pin.

Provisioning the VMs

All the VMs that make up raya are declared via the same pattern inside the cluster.tf file. The starting point is a Butane configuration file, written as a Jinja template file thanks to the NikolaLohinski/jinja provider. Using templates allows us (1) to share configuration snippets among the various VMs, and (2) to inject Terraform variables when building the plan. This configuration file is transpiled to an Ignition config using the poseidon/ct provider. The Ignition config is fed to the VM by passing it via the user_data field of a new hcloud_server resource.

Warning

Ignition configs are likely to embed secrets. Terraform does not treat user_data as a sensitive field, and ct does not mark its rendered result as sensitive either, even when its input is. By default, Terraform will therefore output user_data (at least partially) with terraform plan.

To prevent that, we systematically mark it as sensitive using the sensitive() built-in, in order to avoid leaking secrets via the CI logs.

Users

We create one user, core, with a list of SSH public keys that are authorized to log into the VMs.

Instead of providing SSH public keys verbatim, we fetch them from GitHub. The logic is implemented in github_keys.tf, and relies on the fact that for a given GitHub user $user, https://github.com/$user.keys returns the list of public keys this user can use (one per line).

As a consequence, giving access to the VMs to someone becomes as simple as adding their GitHub handle to var.authorized_users (declared in variables.tf, set in prod.tfvars).

Warning

Adding a new handle to var.authorized_users will change the Ignition config of the VMs making up raya, forcing a complete redeployment. The current declaration of local.authorized_keys ensures a stable order among terraform plan calls for this reason.

Eventually, we will want to migrate to constructing the ~/.ssh/authorized_keys file at startup instead.

Private network

The VMs are attached to the nodes subnet defined in network.tf (see Private network for more details about the subnet itself). Attaching a VM in Terraform only gets it a second network interface. That is not enough in and of itself, as by default CoreOS does not configure that interface.

As a consequence, our Ignition config ships a NetworkManager keyfile for it. The interface is enp7s0 (see Hetzner documentation). The address is statically assigned, with 10.0.0.1 as gateway. This is injected by Terraform to avoid duplicating the information between the cluster.tf file (when attaching the VM to the subnet) and this file. An explicit route sends 10.0.0.0/8 through the gateway. This makes other subnets of the network (if we ever create one) reachable and not just this one. IPv6 is disabled.

Finally, the MTU is set to 1450 (Hetzner’s private network MTU). The MTU is the one to get right. Leaving it to a default value (1,500 bytes being the most standard value) does not fail in an explicit way and can even look healthy (small packets flow). Large transfers would hang, though.

Note

Any systemd unit which requires the private interface to be configured should order itself after network-online.target (and Wants= it). The may-fail=false in the keyfile’s [ipv4] section is what makes that target wait for this interface in particular.

[Unit]
Description=Something that needs the private network
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/ping -c1 10.0.1.10

[Install]
WantedBy=multi-user.target

Wants= and After= are both needed: the first pulls the target into the boot, the second orders against it. After= alone silently does nothing if nothing else requested the target.

Identity of the cluster nodes

Agents are declared in deploy/kube-system/agents.json. Their position give the VM their name (agent-$i), their private address (10.0.1.$(( 20 + i ))), and the record published for it (a$i under the managed zone).

The identity of the node joining the cluster also depends on the VM initial state, whether it runs on the control plane VM or an agent VM. That is, the node running on the agent-0 VM (resp. control-plane) receives a unique name like agent-0-896ef0a5 (resp. control-plane-896ef0a5), which changes the next time agent-0 (resp. control-plane) is replaced by Terraform. The short SHA256 suffix is computed by Terraform from the future VM’s configuration (Ignition config, k3s version, etc.) in cluster.tf1.

This naming scheme allows the cluster to detect when a VM has been replaced. Seeing a node agent-0-ca817dde marked as NotReady and another node agent-0-896ef0a5 marked as Ready is enough for the cluster to know that the former no longer exists. raya has a CronJob named node-reaper responsible for detecting such a situation and cleaning up: deleting the node will cascade into deleting zombie VolumeAttachment objects2.

Public DNS

raya assumes every DNS zone it touches lives in Google Cloud DNS, in the same project. That includes the parent zone (local.dns_parent_zone) it edits to delegate its own subdomain.

Information

By force of habit, I am defaulting to OVH as my domain name registrar, but I am hosting my DNS zone on Cloud DNS. See this GitHub project for more information.

raya owns one zone, google_dns_managed_zone.primary (defined in dns.tf), named after var.cluster_managed_subdomain under the parent zone. It holds the records for the VMs. To make that zone reachable, google_dns_record_set.delegation writes the NS record for it into the parent zone3.

The google provider needs credentials with the following IAM rights:

  • roles/dns.admin, to manage both zones and their records.
  • roles/iam.serviceAccountAdmin, to create the service accounts external-dns and cert-manager authenticate as (see The kube-system namespace).
  • roles/iam.serviceAccountKeyAdmin, to mint those service accounts’ keys, which are what travel in the Ignition config.
  • roles/resourcemanager.projectIamAdmin, to grant those same service accounts roles/dns.admin on the project. This requires to enable the Cloud Resource Manager API.

The last three exist because the cluster does not reuse these credentials: it gets identities of its own, holding roles/dns.admin on this project.

When provisioning raya’s VMs, the following records are created:

  • A record for cp.$google_dns_managed_zone.primary.dns_name pointing to the control plane.
  • A record for a$i.$google_dns_managed_zone.primary.dns_name pointing to the ith agents in the agents.json list.

  1. In reality, this is a little more complicated. The Ignition config sent to the agents as user_data contains the name of the node they will use to join the k3s cluster, so claiming we are using the user_data to decide the name introduces a circular dependency. In reality, we run the template a first time, using mock data for some key fields, to get a sha256 digest we can use to run the template a second time. ↩

  2. node-repear is also responsible for handling the removal of agents. ↩

  3. Looked up by name among the project's zones. ↩