Puffmatic - OpenBSD Autoinstall Generator

Abstract

This article covers automatic OpenBSD installation over the network and from USB thumb drive.

First part briefly discusses the problem at hand and motivations.

Second part discusses the procedure to manually prepare images and file sets, diving in the methodology.

Third part introduces puffmatic - a script that automates above steps.

If you are familiar with unattended installation, you can skip intial discussion and jump stright to puffmatic.

If you are curious how it works, I recommend going through whole article, so you can troubleshoot if anything goes sideways.

Part 1 - Why?

Motivation

OpenBSD has a six-month release cycle, accompanied by a one-year support cycle, which promotes regular updates. I run OpenBSD on my personal laptop and several servers, meaning I confront the possibility of a system upgrade every six months.

I have a reluctance toward upgrades; I simply don’t trust them. While I haven't had any negative experiences with the OpenBSD upgrade process, that’s mostly because I have never relied on it.

Instead of upgrading, I prefer to perform fresh installations.

Configuring a new system can be a laborious task, but since I use Ansible to manage all my systems for both automation and disaster recovery, that concern is minimized.

The only truly "tedious" aspect of this process is the installation itself, followed by a system bootstrap to prepare it for configuration (install python3 from ports and few other quirks).

Fortunately, OpenBSD offers autoinstall(8), and install.site(5) allowing me to automate this part to streamline the process.

Objectives

I want to support two scenarios:

  • Automatic installation from a USB stick to provision my physical hardware, such as the APU4 server or my laptop;
  • Semi-automatic network installation over HTTP

The latter case requires some clarification. Although OpenBSD supports direct automatic installation from bsd.rd booted via TFTP, I run several VPS instances, and my provider does not offer TFTP capability nor local http server. I am willing to provide the URL to auto_install.conf when prompted by the installer. Let's call it semi-automatic.

Overall, system provisioning is divided into two phases:

  1. installation of base system + some initial configuration
  2. further configuration using ansible, conducted over ssh

Phase 1 - autoinstall(8)

During this phase, I want to provision a bootable system with root SSH access and a minimum set of utilities required by ansible.

At a minimum, I need to provision sshd host keys (so I don't have to clear my known_hosts after each upgrade) and configure the network interfaces. This can be achieved using site set archives.

Once the system has network access, I can leverage rc.firsttime to install python3, which is required by ansible during phase 2.

I am intentionally limiting this phase to the bare minimum, as system configuration using shell scripts can quickly balloon in complexity and reliability issues.

Phase 2 - configuration

In this phase, I'm working with a fresh install accessible via ssh. I can execute ansible scripts and securely run arbitrarily complex provisioning procedures.

Part 2 - How?

In this section, I delve into the intricate details of the entire procedure without utilizing any automation. This comprehensive exploration enables readers to grasp the technical challenges that must be addressed.

Autoinstall from USB

This variant is realtively uncomplicated:

  1. grab install77.img installation image from one of the mirrors
  2. add site file sets to install77.img
  3. add auto_install.conf, disklabel to bsd.rd
  4. patch boot.conf to run from patched bsd.rd
  5. run it

Below is the breakdown of the whole procedure.

Adding site77.tgz and site77-${hostname}.tgz

To mount the image, simply use vnd(4), copy the necessary files, and you're done. Preparation for the site setup is detailed in OpenBSD FAQ, so we won’t cover it here.

My installation process divides the file sets into "common" (shared by all machines) and "host," which is specific to each individual host. Therefore, I need to add two tarballs.

  vnconfig vnd0 install77.img
  mount /dev/vnd0a /mnt
  cp site77*.tgz /mnt/7.7/amd64/
  umount /mnt
  vnconfig -u vnd0

Patching bsd.rd

This one is a bit more involving.

Ramdisk kernel filesystem is emedded in compressed bsd.rd and cannot be mounted directly. Additionally, install77.img comes with 2 bsd.rd files: one in /bsd.rd and another in /7.7/amd64/bsd.rd. The installer boots the latter by default and I don't want to modify file sets, so we need to patch boot.conf.

We need to:

  • gunzip bsd.rd
  • extract filesystem from decompressed bsd.rd
  • mount the filesystem image using vnd(4)
  • add auto_install.conf and disklabel to the ramdisk root
  • put patched filesystem back into decompressed bsd.rd
  • compress bsd.rd
  • copy it to /bsd.rd (root of install77.img, not host system)

Patching boot.conf

By default, the installer image loads using /7.7/amd64/bsd.rd. We don't want to modify that file. Instead, we will patch and boot the /bsd.rd found in the installation image root.

To avoid the need for manual selection of the file at the bootloader prompt, we can update /etc/boot.conf.

Since my system is headless and lacks a video console, I also want to enable the com0 console. My final boot.conf is as follows:

stty com0 115200
set tty com0
set image /bsd.rd

Provision USB stick

Insert the USB stick. Run dmesg to identify the device you will be writing to. Wiping your hard drive by accident can teach you to do backups, but that's not our goal here today. For this example, I will assume sd1 is the target device.

  dd if=install77.img of=/dev/sd1c bs=1M

Pull it out, plug it in, reboot and after a while, you should end up with a running system.

Autoinstall over (wild) network

This section covers installation via the Internet. Yes, the Internet. I want to retrieve response files from a machine within the network, as I do not have the luxury of provisioning installation artifacts in a walled garden.

Security considerations

Autoinstall over open internet may be regarded as reckless, so let's dive into some security considerations first to gauge the risk.

There are 4 types of installation resources to consider:

  1. response file
  2. disklabel file
  3. upstream file sets (their big tarballs)
  4. site sets (our own small tarballs)

Let's see if serving those files can be problematic.

Response file

The response file contains two types of secrets:

  • root password
  • Disk encryption passphrase (optional)

By default, the system is configured to prevent remote root login using password authentication, making the provisioned secret unusable without direct access to the system console. We can provision a temporary password and change it later during the phase 2 configuration.

User authentication for ansible access is based on PKI and we provision public ssh key, which by definition is not secret.

The disk encryption passphrase (optional) is another sensitive piece of data. However, it is not usable unless an attacker has either a disk dump or, again, direct access to the system console. The passphrase can also be changed later using bioctl(8), so the window of opportunity is time-limited.

If an attacker gains access to the system console during installation and compromises your empty machine, it indicates that your provider account has been breached and you are in much bigger troubles anyway.

The response file will be served over HTTPS (encrypted) and we can utilize basic authentication to prevent public access.

Additionally, we can further limit access to files served by httpd using firewall IP filtering.

I find that threat level acceptable to me. You make your own judgment.

Disklabel & upstream file sets

No secrets here, period. Let's move on.

Site sets

The situation here is a bit more complicated. The OpenBSD installer does not support basic authentication for installation sets, which limits our options for protecting these files from unauthorized access during installation window.

We do have TLS encryption, allowing us to secure these files using a "secret path" technique, similar to AWS S3. Without knowledge of the path, it is impossible to reach the served files in the first place.

The installer retrieves the secret location from the response file, which is delivered relatively securely, protected by a combination of TLS + basic authentication + IP ACL.

If this is still insufficient, we can always choose not to include sensitive data in the site sets and provision that through SSH during phase 2.

Overall, I find the threat level acceptable as well.

Setup

To serve installation files, we need:

  1. httpd(8) - included in the base system, although any other web server will suffice.
  2. Public IP address. I'm serving my installation files from a BT/EE residential DSL address (my personal laptop). Although the IP is dynamic, it tends to remain stable for weeks, which is "good enough" for my use.
  3. Proper TLS certificate - The OpenBSD installer will not work with self-signed certificates. I obtained mine from Let's Encrypt using the DNS-01 challenge method. It's valid for three months.
  4. DNS - I temporarily added an install.* domain to my zone, pointing to my residential IP address during installation.

Overall, I have a valid installation server accessible via a valid hostname with proper TLS and basic authentication.

My servers have fixed IP addresses that never change, making defining IP ACL straightforward.

This is fine. I'm ok with the events that are unfolding currently.

Boot & install

I'm using Hetzner, which have OpenBSD stock images available in offer. I'm booting my VPS from cd77.iso and point it to my installation server sitting on my desk at home:

Welcome to the OpenBSD/amd64 7.7 installation program.
(I)nstall, (U)pgrade, (A)utoinstall or (S)hell? a
Could not determine auto mode.
Response file location? [https://100.64.1.2/install.conf] https://user:pass@install.example.com/install-alfa.conf
(I)nstall or (U)pgrade? i
Fetching https://user:pass@install.example.com/install-alfa.conf
Performing non-interactive install...
...

That's all folks!

Part 3 - Automate it with puffmatic

Here we are. Over time, we have collected a variety of auto_install.conf response files, a few siteXY.tgz tarballs, and some peculiar configuration quirks for various edge cases. Maintaining these configurations by hand has become quite tedious.

Introducing puffmatic - a Python script that:

  • takes "domain configuration" as input
  • downloads file sets from the OpenBSD mirror
  • patches installer images with auto_install.conf
  • generates response files and site sets for hosting over HTTP

You maintain and version a set of YAML configuration files. The script processes these config templates, fills in the details, and packages your site files for you.

Want to jump stright into example? Check example.com README.md.

Prerequisites

This script - apart from python3 packages - uses only utilities available in base install.

However, it requires priviledged access to perform certain operations:

  • mount - we need to mount filesystem images
  • umount - we need to unmount them when done
  • vnconfig - required to attach files as drives
  • install - required to copy files into mounted installer root

To avoid running entire script as root, it leverages doas to execute only those selected commands. Here is required doas.conf:

permit nopass user cmd mount
permit nopass user cmd umount
permit nopass user cmd vnconfig
permit nopass user cmd install

You could argue that running install as root effectively opens the system for abuse and that's true. If concerned, create a dedicated account to build installation sets. Most of the script doesn't require root access, so I tried to limit it as much as possible using doas technique.

It works for me. Don't shout at me, ok?

Domain configuration

Configuration is divided into domains. Literally domain names, like example.com, which we describe here. Configuration direcotory ("domain"), will contain few elements:

config.yaml
Main configuration file. The file itself is well documented, so we spare detailed fields description. Here we configure mirror URL, where to put output files and how to host them.
site
this directory contains files that will be placed in siteXY.tgz file set. This directory is accompanied by site.mtree, which describes file attributes. Consult mtree to learn more about this tool.
templates
template install.conf.j2 with some configuration overrides. This response template file will be used by default, unless there is a host specific one.
hosts
here we store host-specific configuration, like alfa for alfa.example.com and bravo for bravo.example.com. Host specific configuration can provide own site directory (that one will be packaged as site77-${hostname}.tgz), config.yaml overrides or complementary values and boot.conf.
install.conf.j2
autoinstall response file. This file can be provided either as a Jinja2 template (it will be populated using values from merged config.yaml files) or stright as install.conf, used verbatim.

Running the generator

There are two modes available: network installation or USB stick. Generator must be inside the "domain configuration" directory.

USB

To generate auto-installable USB image, run puffmatic-usb:

  puffmatic-usb $HOSTNAME

Here, $HOSTNAME must match one of the host configuration directories in the hosts/ folder. The resulting file will be named install77-$HOSTNAME.img and placed in the output/img directory. This image must be written to the USB thumb drive using dd. The install77.img file will be downloaded if it doesn't already exist. This process may take some time, so the initial run could be longer. Subsequent runs will utilize the cached installer image file.

Network

To generate HTTP site sets, use the following command:

  puffmatic-net

Note that there are no arguments required. This command will process all hosts in the ${PWD}/hosts/ directory and output relevant response, disk labels, and site sets in the output/sets and output/resp directories. Those output directories are suitable for serving with httpd(8).

Network installation requires a complete set of files, so puffmatic-net will mirror the entire OpenBSD release directory for the configured architecture. This process involves downloading approximately 2.5GB of data, so the initial mirroring (using rsync) will take some time. Subsequent runs will utilize cached files.

Messed up? Just delete output/ and start again.