Skip to content

Provisioning the cluster

Following the steps of tinkerbell and elsa, raya is built on top of Fedora CoreOS. Mostly for reducing costs, it is deployed on Hetzner Cloud.

Building a CoreOS image for Hetzner

While Hetzner Cloud is a great provider, it sadly does not provide a CoreOS image its VMs can boot from. This forces us to build one ourselves, and we do so using Packer.

The Packer template we use is image/fcos.pkr.hcl. Because Fedora CoreOS provides artifacts for Hetzner for its releases, the template mostly consists of downloading them and burning them onto a throwaway server’s disk (image/install.sh) using dd. As a consequence, CoreOS itself never boots during the build, which is why Ignition still runs on first boot of every server made from the snapshot. Once the disk is written, a snapshot is published, with labels allowing us to use it from a Terraform configuration. This is what the hcloud_image.fcos data source is used for (declared in cluster.tf).

image/build.sh is a script that can be used to build the image iff it does not already exist. If it does, calling the script returns without building anything.

Which image is burnt is decided by the image/fcos.pin file, which specifies the stream (among stable, testing, or next, we use stable for now), version and sha256 hash of the image. image/fcos.pkr.hcl, image/build.sh and cluster.tf all read it, making it the source of truth everybody trusts.

Once the image is burnt onto the disk, the latter is mounted and k3s is copied in /usrlocal/bin, ready to be used. Which versioned is provisioned is decided by the image/k3s.pin, using the same logic applied to image/fcos.pin.

Provisioning the VMs

All the VMs that make up raya are declared via the same pattern inside the cluster.tf file. The starting point is a Butane configuration file, written as a Jinja template file thanks to the NikolaLohinski/jinja provider. Using templates allows us (1) to share configuration snippets among the various VMs, and (2) to inject Terraform variables when building the plan. This configuration file is transpiled to an Ignition config using the poseidon/ct provider. The Ignition config is fed to the VM by passing it via the user_data field of a new hcloud_server resource.

Warning

Ignition configs are likely to embed secrets. Terraform does not treat user_data as a sensitive field, and ct does not mark its rendered result as sensitive either, even when its input is. By default, Terraform will therefore output user_data (at least partially) with terraform plan.

To prevent that, we systematically mark it as sensitive using the sensitive() built-in, in order to avoid leaking secrets via the CI logs.

Users

We create one user, core, with a list of SSH public keys that are authorized to log into the VMs.

Instead of providing SSH public keys verbatim, we fetch them from GitHub. The logic is implemented in github_keys.tf, and relies on the fact that for a given GitHub user $user, https://github.com/$user.keys returns the list of public keys this user can use (one per line).

As a consequence, giving access to the VMs to someone becomes as simple as adding their GitHub handle to var.authorized_users (declared in variables.tf, set in prod.tfvars).

Warning

Adding a new handle to var.authorized_users will change the Ignition config of the VMs making up raya, forcing a complete redeployment. The current declaration of local.authorized_keys ensures a stable order among terraform plan calls for this reason.

Eventually, we will want to migrate to constructing the ~/.ssh/authorized_keys file at startup instead.

Private network

The VMs are attached to the nodes subnet defined in network.tf (see Private network for more details about the subnet itself). Attaching a VM in Terraform only gets it a second network interface. That is not enough in and of itself, as by default CoreOS does not configure that interface.

As a consequence, our Ignition config ships a NetworkManager keyfile for it. The interface is enp7s0 (see Hetzner documentation). The address is statically assigned, with 10.0.0.1 as gateway. This is injected by Terraform to avoid duplicating the information between the cluster.tf file (when attaching the VM to the subnet) and this file. An explicit route sends 10.0.0.0/8 through the gateway. This makes other subnets of the network (if we ever create one) reachable and not just this one. IPv6 is disabled.

Finally, the MTU is set to 1450 (Hetzner’s private network MTU). The MTU is the one to get right. Leaving it to a default value (1,500 bytes being the most standard value) does not fail in an explicit way and can even look healthy (small packets flow). Large transfers would hang, though.

Note

Any systemd unit which requires the private interface to be configured should order itself after network-online.target (and Wants= it). The may-fail=false in the keyfile’s [ipv4] section is what makes that target wait for this interface in particular.

[Unit]
Description=Something that needs the private network
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/ping -c1 10.0.1.10

[Install]
WantedBy=multi-user.target

Wants= and After= are both needed: the first pulls the target into the boot, the second orders against it. After= alone silently does nothing if nothing else requested the target.

Identity of the control plane

To join a k3s cluster, an agent needs to prove it knows a token that is usually generated by the control plane on its first boot, alongside its own certificate authorities the first time it starts. This approach is cumbersome, because it requires to instrument the control plane to upload its token to a vault of sorts and the agents to wait for the secret to be available.

To avoid this cumbersome procedure, we take a different road: we let Terraform generate both the token (using hashicorp/random) and the CA (using hashicorp/tls), and we embed the relevant secrets in the Ignition configs of the VMs (depending on their role).

Generating the CA in addition to the token means we can provide enough information to the agents so that they do not need to trust the control plane endpoint on first connection. An agent will receive exactly two things: the address of the control plane, and a token in k3s's full form,

K10<sha256 of the server CA certificate>::server:<secret>

and nothing else.

The two halves of that token do two different jobs. The secret proves the agent may join. The hash is what removes the need to trust the endpoint: on first contact the agent downloads the CA bundle from the control plane without validating the certificate it is offered, hashes what it received, and compares that against the hash embedded in its own token.

Identity of the agents

Agents are fully specified by their position in deploy/fleet/agents.json. That position gives the VM its name (agent-$i), its private address (10.0.1.$(( 20 + i ))), the record published for it (a$i under the managed zone), and the node password they use to join the cluster.

Public DNS

raya assumes every DNS zone it touches lives in Google Cloud DNS, in the same project. That includes the parent zone (local.dns_parent_zone) it edits to delegate its own subdomain.

Information

By force of habit, I am defaulting to OVH as my domain name registrar, but I am hosting my DNS zone on Cloud DNS. See this GitHub project for more information.

raya owns one zone, google_dns_managed_zone.primary (defined in dns.tf), named after var.cluster_managed_subdomain under the parent zone. It holds the records for the VMs. To make that zone reachable, google_dns_record_set.delegation writes the NS record for it into the parent zone1.

The google provider needs credentials with the following IAM rights:

  • roles/dns.admin, to manage both zones and their records.
  • roles/iam.serviceAccountAdmin, to create the service accounts external-dns and cert-manager authenticate as (see The kube-system namespace).
  • roles/iam.serviceAccountKeyAdmin, to mint those service accounts’ keys, which are what travel in the Ignition config.
  • roles/resourcemanager.projectIamAdmin, to grant those same service accounts roles/dns.admin on the project. This requires to enable the Cloud Resource Manager API.

The last three exist because the cluster does not reuse these credentials: it gets identities of its own, holding roles/dns.admin on this project.

When provisioning raya’s VMs, the following records are created:

  • A record for cp.$google_dns_managed_zone.primary.dns_name pointing to the control plane.
  • A record for a$i.$google_dns_managed_zone.primary.dns_name pointing to the ith agents in the agents.json list.

  1. Looked up by name among the project's zones. ↩