Week 9: Configuration control with NixOS and Kairos. Tools: blxrep, udpspeeder and more

In this weeks issue!

• Configuration control with NixOS and Kairos
• Tools: blxrep, udpspeeder, Node-RED, RAG and below
• Random: LTE/5G vulnerabilities and Mr. Beast

Configuration control with NixOS and Kairos

A client asked me about configuration control and brought up Ansible, Puppet and so on. I'm not sure if configuration control is the right term, but given the context my client was asking in, I immediately thought of NixOS and Kairos. Although these two are very different there are some commonalities:

• Unified declarative configuration, meaning you can configure your whole OS using the same syntax (although you may have to add support for some settings)
• Immutable, meaning you don't get configuration drift at runtime (with caveats)
• Atomic updates and rollback, meaning if an update goes wrong the system will be fine and you can select previous generations at boot

There is quite a large group of software that comes into the configuration management and infrastructure as code (IaC) categories that will control the state of a Linux installation. The idea being that you can automate installing and updating one or more Linux devices. There are a number of systems which can install and configure Ubuntu, these act as an external controller and usually use an agent, such as sshd, to control the box once it is up and running. During installation something like cloud-init can be used to get the initial configuration.

This isn't exactly how NixOS and Kairos work. NixOS is in fact a stand alone Linux distribution which is built upon the Nix language and package manager. Kairos is a meta-distribution which transforms other distributions into immutable systems with generations and rollback. They both allow creating customized ISO and VM images.

Kairos (started by my good pal Ettore Di Giacinto , who also created LocalAI) uses container technologies to take a distribution like Ubuntu and turn it into an immutable OS. The resulting image still uses the Ubuntu kernel and to my knowledge there is no container runtime.

It uses file system overlays at runtime to allow rollback. At build time it uses Dockerfiles and so it's essentially using Dockerfiles to package a bare metal OS. The overall system configuration is done using a cloud-init file in a similar fashion to how NixOS uses Nix to configure everything (see below). The cloud-init syntax can be extended using bundles, which themselves are created with Dockerfiles and are kind of like meta-packages.

Nix is actually a functional programming language that is intended for reproducibly building and packaging software. You can install the Nix package manager on Linux, Mac and maybe Windows at some point. It's in fact the largest and most up to date OSS repository in the world by a very long way. Only AUR comes close and that is actually still less fresh than a NixOS stable release from over 3 years ago.

The Google and Apple app stores are larger, but Nix is catching up. Perhaps Docker hub is also bigger, but there is a very low bar to adding containers on Docker hub.

NixOS is a Linux distribution built with Nix. Almost everything on NixOS can be configured from a single Nix file. The base Nix system is partially immutable and supports rolling back to previous generations. I started using NixOS because I was fed up of reinstalling and configuring machines from scratch as well as maintaining lots of different config files. I also loved the Nix package manager, especially because I could just type "nix run nixpkgs#some-program" or "nix shell nixpkgs#some-program" and try out some program without installing it or using a container.

NixOS also has the steepest learning curve I've ever encountered in a Linux distribution. You aren't just learning an unusual distribution, you are also learning an unusual programming language and its libraries. A lot of software does stupid stuff like download binaries that link to a specific libc in a specific location. This does not work with Nix unless you implement a workaround and this is usually not that easy. The fact then that NixOS has more than just a few users is a testament to how incredibly powerful it is. There has to be a strong incentive to use this software because it is bloody difficult to get into it.

I vaguely remember a post from someone saying "I finally wrote my first Nix flake/package and it only took me two years!". Obviously the story is quite different with Kairos which still requires some mental adjustments, but will be more familiar to a lot of system admins who have already been exposed to Kubernetes, cloud-init and whatever distro they decided to wrap.

It could actually make sense to wrap NixOS with Kairos because the level of immutability is different between them, but there are a lot of details to sort through before deciding something like that. Indeed there is a lot of depth to this topic.

Relevant Links

• https://nixos.org/
• https://kairos.io/

Tools

• blxrep - disk replication tool that uses eBPF tracepoints
• udpspeeder - Wraps UDP packets (e.g. VPN traffic) providing forward error correction codes making any UDP traffic more reliable
• Node-RED - Low code event-driven programming. This is great for writing programs that you can then hand off to non-coders, especially electrical and mechanical engineers who despise anything text based. It has a module for Modbus which I covered in my previous newsletter.
• below - Like top, but for CGroups
• ebpf-usb and ebpf-usb-inspector - Spy on USB devices. I don't know that these work given their age, but as is often the case with eBPF you can adapt scripts to your own purposes
• Kotaemon - An open source RAG UI that actually works

Random

• 119 vulnerabilities in LTE/5G core infrastructure
• HOW TO SUCCEED IN MRBEAST PRODUCTION