Background

I’ve got a couple of computers in the home. One machine in particular is a semi-powerful gaming machine. The weird thing about this machine is that it’s stowed away in a storage room, not a very suitable place to sit and actually use it.

To be honest, I’m not really much of a gamer, and so I mostly use its capabilities for work and other heavier tasks. But it’s always nice to be able to use graphical applications in a performant way. And I do in fact use it for running Zwift with my virtual trainer.

Before switching to Hyprland running on Wayland I used various SPICE and RDP setups that worked well enough running on an X server. However, with my limited patience I didn’t manage to find any suitable solutions working well on Wayland. And when I found people mentioning solutions Sunshine with Moonlight always seemed to be high up on the list of options.

I managed to get Sunshine running on NixOS with minimal effort due to the Sunshine NixOS module. Then on both my NixOS laptop and other Windows desktop computer I’ve installed Moonlight.

The issue

However, after upgrading to NixOS 25.11 I noticed that Moonlight no longer managed to capture input (there were several other issues, but more on that later).

Inspecting the sunshine.service logs I noticed the following warnings:

Warning: Unable to create virtual mouse: Permission denied
Warning: Unable to create virtual keyboard: Permission denied

With a smoking gun I managed to google my way to realize that Sunshine uses udev to emulate HID¹ devices for input capture through the /dev/uinput device.

An sure enough, when I checked the permissions on /dev/uinput I saw that it was only read-writable by root:

❯ ls -l /dev/uinput
crw------- 1 root root 10, 223 Dec 11 22:36 /dev/uinput

Although it might seem tempting to lean up and just chmod 666 /dev/uinput, you don’t want to give everyone access to /dev/uinput, as that would be a security risk. But Sunshine needs access to it, and the NixOS module spins up Sunshine instances through a systemd user service running as your unprivileged user.

Not sure what has changed to cause this to break now, I don’t know. But there must be a simple way to work around it, right?

The fix

As covered on the Sunshine troubleshooting page users need to be a member of the input group when running Sunshine unprivileged. However, this doesn’t really help if /dev/uinput is only read-writable by root.

Luckily NixOS makes it easy to add custom udev rules through it’s configuration system. Adding the following udev rule to my NixOS configuration:

services.udev.extraRules = ''
  KERNEL=="uinput", MODE="0660", GROUP="input", SYMLINK+="uinput"
'';

This creates /dev/uinput with the right permissions.

❯ ls -l /dev/uinput
crw-rw---- 1 root input 10, 223 Dec 11 22:36 /dev/uinput

Then I had to add myself to the input group:

users.users.myme.extraGroups = [ "input" ];

And with that, Moonlight is able to pass captured input to Sunshine and I’m able to control my stowed away gaming machine once again.

Footnotes

Background

As a developer, when something crashes on my computer I’m inclined to fix it. Obviously that comes with the trade of writing your own software, but we are also empowered with the knowledge and skill to find ways to contribute to fixing issues in programs and apps we didn’t write ourselves.

In the world of open source we even have the superpower of modifying programs someone else wrote, then build and run it without ever involving the upstream maintainers. This is obviously great, but also cursed, in a way. Knowing you are able to fix something can be frustrating when you feel you don’t have the time or energy to do so.

I will admit as I’m getting older my threshold for debugging issues in other people’s software is getting higher. Lazy, I know. But when I experience weird crashes in off the shelf applications I quickly turn to search engines to see if someone else has experienced similar issues.

Before LLMs we relied on people to continously publish their issues and solutions online for this to work. Now this has obviously changed quite significantly. LLMs provide great help just for you in your particular situation, but the rest of the world won’t see a trace of this interaction (besides the model providers, of course.)

So I thought to get back into writing I’d take some of the issues I experience and try to document how I fixed them (or abandoned trying.)

The crash

I choose to use Firefox as my main browser and have been happy with it for the last decade or so. One prerequisite for me to use a piece of software however, is that it works™.

For the last few weeks I’ve noticed that both Firefox and Thunderbird has been crashing when I leave work with my laptop. It happens the second I unplug the USB-C cable connecting two external displays and a keyboard and presents the following error:

Crash Annotation GraphicsCriticalError: |[0][GFX1-]: (hyprland) Wayland protocol error: wp_color_manager_v1#46: error -1: Invalid output (2)
(t=3.49975) [GFX1-]: (hyprland) Wayland protocol error: wp_color_manager_v1#46: error -1: Invalid output (2)

I’m not going to admit to how long I kept up with just restarting Firefox (and had to unlock my password manager browser extension with a stupidly long password.) But eventually I couldn’t take it any longer.

First I seriously considered switching to Chromium just because of my laziness. But it didn’t even take an hour before I realized I didn’t really enjoy that experience.

Time to scour the internet for solutions!

The bug report

Turns out there’s already an upstream issue reported on Bugzilla, Firefox’s issue tracker.

The report states that Firefox crashes on Wayland upon losing a display. Sound like we got it!

Looking closer at the issue could be a generic Wayland and Firefox issue, but the only reports that I’ve found mention Hyperland (my window manager.) To resolve this issue let’s see if there are any issues or fixes for Hyperland in particular.

And there is!

https://github.com/hyprwm/Hyprland/pull/11916

The fix

When there are bugs in software that you use and you don’t build everything yourself from source, you’re by default at the mercy of package maintainers to get fixes into upstream packages.

The same applies for the default experience on NixOS. However, since packages (or derivations) on NixOS basically are defined through their build recipe it’s trivial (once you know how) to patch software yourself.

The fix for Hyprland#11916 is already merged and applied in main upstream, but there hasn’t been a release containing it yet. Regardless, we can quite easily replace the source of the hyprland package in a NixOS configuration by specifying an overlay pulling down a source tree containing the fix.

From my personal NixOS configuration in myme/dotfiles:

# overlay.nix
{
  hyprland = nixpkgs.legacyPackages.${prev.system}.hyprland.overrideAttrs ({
    version = "0.51.1-patched";
    src = prev.fetchFromGitHub {
      owner = "hyprwm";
      repo = "hyprland";
      fetchSubmodules = true;
      rev = "ab11af9664a80df70fe3398810b79c4298312a33";
      hash = "sha256-dSAPRyKzxM+JodX5xBCmpVrVYWjYpNPPiSySaI4W+rQ=";
    };
  });
}

Hydra builds packages in the maintained NixOS release channels, but once we create an overlay we generally won’t find existing builds. What nix does when applying the configuration is that it doesn’t find a cached derivation and proceeds to build it on the spot.

After a nixos-rebuild switch --flake .# I now have a laptop setup where Firefox no longer crashes when I unplug USB-C cables connected to external displays. Great success!

Once the fix makes it into the next NixOS stable release I can remove the overlay and use the stock package.

If you’re not already at the point where all your little AI agents are churning out code for you — well, I guess it’s true if you’re extensively prompting via text too — chances are you’re spending a significant time bashing on your keyboards still?

Thought so!

⌨ Key background information

Personally I’m not in the “fancy keyboard camp”¹. I’m not sporting some split-keyboard variant, configured with some less-than-usual key layout (my layout “family” is QWERTY). I do require easy access to some weird “A”s and “O”s with circles and lines through them to fully articulate my native tongue² — Norwegian — in writing.

The standard no layout is horrible for coding, due to how all the special characters are laid out and require modifiers, so I mostly try to find and apply the us alt-intl-unicode variant where available (which is most Linuxes, but not stock on Windows, afaik³).

What I can say is that I don’t go very overboard with my keyboard setup. I’m reluctant to build habits on top of very unconventional setups, for a couple of reasons:

1. I’ve been fortunate enough to never experience anything resembling RSI.
2. Until I have to worry about point number 1, I think it’s nice to stick close to how most computers are setup by default.

That said, I do find ergonomics and efficiency on the keyboard to be very important. Which is why I also do opt for having “regular” tenkeyless or mini mechanical keyboards for my desktop setups, because it’s just more comfortable. It is also why I use vim emulation in any editor I use as well as my browser.

🕹 Control yourself

If you’ve used vim emulation modes — or vim itself, obviously — for any extended amount of time, you know there’s a whole lot of <esc> action going on. The Escape key is used to switch from most modes back to <NORMAL> mode (used for general text navigation through vim motions.) On most keyboards one might say this is a bit of a stretch. It gets your fingers off the home row and generally hampers your flow.

I know there are many out there that for instance remap the CapsLock key to Escape, or even to Control (I mean, WHO’S USING CAPSLOCK, RIGHT!?) I don’t fancy that too much, as I mentioned I don’t want my setup to be too custom.

Instead, I’ve built my escape around what I think is a fairly unknown feature of terminal emulators: Hitting ^[ fires off an <esc> (that is Control+[).

But isn’t that, uhm, like pressing two keys to get an escape?

Well, yes, indeed it is.

But the big difference is, although you’re pressing two keys at once, your hand position is almost not affected by reaching for the two keys with each of your pinkies. This felt so strange to me when I started incorporating it, but now I much prefer keeping my hands still at the cost of pressing two keys at once.

If you find yourself exclusively in vim, another <esc> alternative is the classical ^c or Ctrl+c. It’s not an option for me who switch frequently between vim and emacs, as C-c (yet another notation for ^c)⁴ is a very common prefix for many sequential “key chords”.

In evil-mode it does work to use the traditional emacs “quit” binding of C-g to escape back into <NORMAL> mode. But — and I bet you guessed it — it does not translate well back into vim. Which is a muscle memory issue I still struggle with on a frequent basis.

🍒 Terminal escapes? There are dozens of us!

There are a bunch of these Ctrl+<key> sequences — sitting at various levels of useful — when typing on a typical terminal emulator⁵:

^[

Escape

^i

Tab (Rarely used it)

^m

Carriage return (nah, not for me)

^j

New line (well, it’s somewhat of an emacs thing)

^c

Interrupt current command (I bet you know this)

^d

End of file (this one too!)

^w

Delete previous word (careful doing so in your browser)

^l

Clear screen

^h

Backspace

^k

Kill/delete to the end of line (super useful)

^a

Move cursor to the start of line

^e

Move cursor to the end of line

^/

Undo

^_

Redo

Who am I kidding, I’m not going to list them all.

Like I said, some of these are mostly a curiosity and not all that useful in everyday, programming life. Others are surprisingly handy, and you should consider to get them into your typing habits.

I would suggest to have a look in the READLINE section of man 1 bash. Similarly to vim bindings, you find readline bindings in quite a number of places too, so mastering both is a useful skill to possess — even in 2025.

Footnotes

At work, I’ve set up a few Raspberry Pi 400s as signage devices. These devices display a web browser in “kiosk” mode, showing a roulette of information pages, live reports, and various system statuses.

Setting up the boxes, I probably could’ve used off-the-shelf solutions like Anthias (never tried), but where’s the fun in that?

Instead, the whole setup is powered by a simple ~200 lines NixOS configuration combined with a bespoke signage web application implemented using vanilla HTML, CSS, and some simple JavaScript.

The Raspberry Pis aren’t exactly computational powerhouses and in my post on building NixOS for the Raspberry Pi 3B I explained how I use emulated¹ compilation via binfmt_misc from my more powerful laptop or desktop computers. This avoids doing most of the distribution assembly work on the Raspberry Pi (which would take forever) and it also keeps most development dependencies off the target systems. This makes for a leaner deployment in general, which is good given NixOS’s notorious appetite for disk space.

All that said, this post isn’t about the signage device configuration and the application I’ve made, but a fun little detail in the capabilities of nix and the exchange of its pre-built² artifacts.

🕸 Look at the mesh I’ve made

We don’t have much physical infrastructure at work besides the WiFi mesh network. There’s no corporate network or other VPNs. Pretty much everything is cloud-hosted. The signage R-Pis are basically the local infrastructure at the moment (lol). In any case, to access the devices from home, I’ve set them up on a tailnet.

I’ve used various means of getting NixOS configurations onto remote hosts, mainly deploy-rs. However, NixOS supports remote system profile activation without any additional dependencies through nixos-rebuild. I won’t go into details as there’s already an excellent Haskell for all post on the topic. So for these devices I simply stick with nixos-rebuild.

One pain point is that copying nix closures from my host machine to the Raspberry Pis in the office is slooow. Dead slow. 💀

Also, as I’m iterating on configurations and upgrades, I don’t usually want to mess with all devices at once, so I typically pick out one I use for experimentation. Luckily, nix will only copy things that have changed, and this means that most of the required derivations make it to my target device incrementally until I’m done experimenting. However, when I want to apply this final configuration to the other devices, I would need to copy them all… again.

The devices themselves, however, are on the same network, and so copying between them should be a fair amount quicker. What if I could copy common dependencies in the configurations directly among the devices, saving me time as well as not having to remember to keep my laptop on the same tailnet (I use several)?

As long as we know what we’re after and nodes know how to communicate with each other, nix alone can serve as its own little distributed binary cache.

That’s preeetty dope.

🚧 NixOS rebuild

First off, we need to build the complete configurations for the devices we wish to deploy. This is trivial with a flake configuration where the various hosts are defined together. Building a specific one is as simple as using nixos-rebuild build:

nixos-rebuild build --flake .#baard-open-space

This leaves the system activation package linked under ./result:

❯ ls -l result
lrwxrwxrwx 1 myme users 101 May 28 19:34 result -> /nix/store/h16i6zjrjcv0wrd2dl9n3m0g4xqjcn4a-nixos-system-baard-open-space-23.11.20240525.9d29cd

How do we find shared derivations between the various configurations?

🌳 nix-tree

nix-tree is a great little tool for browsing the dependency graphs of nix derivations: the derivation’s closure. It provides a TUI reminiscent a file browser where it allows you to dig down into the dependency graph of derivations provided on the command line:

nix-tree shows the derivation’s closure size and sorts the results from largest to smallest. This is useful as I would like to avoid copying as much as possible from my machine at home that has already been copied to one of the Raspberry Pis in the office. Inspecting the activation packages we can see that etc is the largest, while the system-path is the second largest.

I also would like to avoid copying stuff that is specific to a single host’s configuration because it’s unusable by any other host. Navigating around in nix-tree it’s clear that there are certain host specifics in etc. This is not surprising as the hostnames differ, etc. However, everything within the system-path is identical and the closure hash is the same.

Another even simpler (and more precise) way of determining shared derivations is using nix-store --query directly:

❯ nix-store -q --references ./result* | cut -d'-' -f2- | sort | uniq -c | sort -n
      1 append-initrd-secrets
      1 bash-5.2-p15
      1 bash-interactive-5.2-p15
      1 coreutils-9.3
      1 extlinux-conf-builder.sh
      1 findutils-4.9.0
      1 firmware
      1 getent-glibc-2.38-66
      1 glibc-2.38-66-bin
      1 glibc-locales-2.38-66
      1 gnugrep-3.11
      1 initrd-linux-6.1.91
      1 kernel-modules
      1 kmod-31
      1 lingering-users
      1 linux-6.1.91
      1 local-cmds
      1 mounts.sh
      1 net-tools-2.10
      1 nixos-system-baard-entrance-23.11.20240525.9d29cd2
      1 nixos-system-baard-open-space-23.11.20240525.9d29cd2
      1 setup-etc.pl
      1 shadow-4.14.2
      1 shadow-4.14.2-su
      1 systemd-254.10
      1 system-path
      1 update-users-groups.pl
      1 users-groups.json
      1 util-linux-2.39.2-bin
      2 etc
      3 perl-5.38.2-env

By passing both system derivations to nix-store --query --references, we’re getting the union of all referenced derivations. Since we also get the hash in the nix store paths, any derivation name that appears only once is either an identical, shared dependency, or it’s specific to one of the two devices.

I’ve yet to explore the possibilities of nix-store / nix store sub-commands like diff-closures, which would most likely be able to provide even more precise results with regard to which closures are identical between two derivations. Neither have I spent any effort digging into other tools specializing in nix deployment. For instance, Colmena supports parallel deployment, but I’m unsure if it has any features related to copying derivations between two or more remote hosts.

🍝 Copy/pasta

Once we’ve determined the derivation(s) we want to copy we can use nix-copy-closure. It allows us to copy a derivation and its dependency graph in its entirety from one Raspberry Pi host to another.

Without further ceremony:

nix-copy-closure --from 10.20.30.40 /nix/store/gxz15i14ig73maiy2zpaj4myhl9gckyi-system-path

Once the entire system-units from the example above have been transferred, we need to perform the actual activation of the next NixOS generation. This can be done using a regular nixos-rebuild switch with a remote target host.

nixos-rebuild switch --use-remote-sudo --target-host 10.20.30.40 --flake .#baard-open-space

For a little more ergonomics, I use a small bash script that also asks to restart the display manager of the signage device (to apply window manager configuration changes, etc.):

#!/usr/bin/env bash

set -eo pipefail

# Ensure there's a hostname argument
if [ $# -ne 1 ]; then
    echo "Usage: $(basename "$0") <hostname>"
    exit 1
fi

# Fetch all possible hostnames into an array
hosts=()
while IFS= read -r line; do
    hosts+=("$line")
done < <(nix flake show --json 2>/dev/null | jq -r '.nixosConfigurations | keys[]')

# Ensure the hostname is valid
if [[ ! " ${hosts[*]} " =~ " $1 " ]]; then
    echo "Invalid hostname: $1"
    echo ""
    echo "Use one of:"
    printf '  %s\n' "${hosts[@]}"
    exit 1
fi

# Build and deploy the system
nixos-rebuild switch --use-remote-sudo --target-host "$1" --flake ".#$1"

# Restart the display server if the user wants to
read -p "Restart the display manager? (y/N) " -n 1 -r

if [[ $REPLY =~ ^[Yy]$ ]]
then
    ssh "$1" sudo systemctl restart display-manager.service
fi

Which is invoked via the configuration flake as an app:

{
  # ...

  outputs = {
    # ...

    apps = let
      deploy = pkgs.stdenv.mkDerivation {
        name = "baard-deploy";
        src = ./deploy.sh;
        dontUnpack = true;
        installPhase = ''
          mkdir -p $out/bin
          install $src $out/bin/deploy
        '';
      };
    in {
      deploy = {
        type = "app";
        program = "${deploy}/bin/deploy";
      };
    };
  };
}

nix run .#deploy baard-open-space

After running the remote-to-remote closure sync, the deployment only copies a fraction of the required system dependency derivations.

How cool is that?

Footnotes

Automation 🤖

I’ve invested leisure time to save time when I’m working hard, not to save keystrokes but to save mental energy for the project at hand.

• John D. Cook

In much of the work that I do I try to automate to save mental energy. Automation, to “techy” people, often means to apply some sort of repeatable application of machinery or code in order both free us from manual labor and potentially increase our effective throughput. I’m using a more general interpretation of “automation” when I say “I try to automate to save mental energy” – an interpretation more in line with what John D. Cook points towards in his post quoted and linked to above.

Yes, writing scripts, applying tools and utilities to common everyday programming problems are undeniably “automation”. However, automation in its purest sense isn’t reserved for silicate or metallic machinery only. Biology automates through instinct and practice.

What this means to me in practice is that I relatively often invest a significant amount of time learning new things, practicing various skills, and iterating on my workflows. Not necessarily to save on some imaginary “net sum of time spent”¹ – although Randall Munroe of XKCD has shared some thoughts on the topic of – but to ensure I can maintain a certain level of focus in my work when I’m “in the zone”.

Focus 🔍

Why is focus so important? Well, programming is hard and I’m sure also most non-programmers acknowledge the importance of focus when doing work requiring a certain level of immersion.

This little gem from MonkeyUser.com illustrates “focus” for a programmer perfectly:

I’ve often used the analogy of building a house of cards when explaining to non-programmers how we build mental models of the problems we are solving. Like a house of cards these mental models take considerable time and effort to construct, but topple over and collapse from even the smallest breath of air (interruptions).

Our mental models dissolve faster than cotton candy in water when we’re distracted at an inappropriate time.

How about focus in my work? Sure!

The draft of this post is over a year old. I guess I got distracted!

Distractions ⚡

Distractions; the arch enemy of every coder. Distractions break our focus by ripping us out of our flow states. It doesn’t help that programming environments are littered with potential distractions. They come in many shapes and forms, to name a few:

• ⚡ Physical interruptions from our surroundings (meetings, colleagues, notifications).
• 🧠 Internal interruptions through impulsive thoughts (spontaneous ideas, procrastination).
• 🤷 Missing preconditions to perform a task (unclear objectives, yak shaving, tech debt).
• ➰ Long feedback loops (handovers, difficulty testing, ++).
• 🥪 An empty stomach.

Some distractions are obvious, others are more subtle.

When we talk about distractions it’s often the “big and obvious” ones that people think of. Those are the distractions where you’re interrupted by someone or something in your surroundings. It may be somebody “just popping over to ask a question” or some meeting looming on the horizon about to evacuate you from your warm and cozy “flow zone”.

In recent years I’ve also grown more conscious of less obvious forms of distractions. Things like small papercuts and annoyances which I previously might have accepted simply as a natural part of “work”. By not acknowledging these minor nuisances as distractions I’ve never stopped to consider how they may negatively affect my productivity or cognitive ability. However, tiny inconveniences can add up to make solving issues more difficult than it needs to be.

Much can be done to reduce the distractions within an organization, but introducing transformational change can be hard for an individual alone. Organizational chaos might even be outside of our control. What we can control to some extent is how we interface with the environment around us.

I find many negative effects of distractions – large and small – can be greatly reduced through personal organization and mastery of your tools.

Organization 📓

I’m personally a big fan of Getting Things Done (GTD), Zettlekasten methods (using emacs with Org Mode and Org-roam), and various other “productivity life-hacks”.

I strongly believe everyone should invest a bit of time and effort into learning a few organizational tools. I don’t have strong opinions on which, but I believe it’s crucial to learn them well enough and integrate them into your daily workflow so that they stay out of your way. It doesn’t even have to be anything digital. A paper notepad is honestly perfectly acceptable too. However, I believe tighter integration into e.g. code editors increases the possibility for distraction free note-taking and organization.

I feel an organizational tool hits the sweetspot when you hardly notice yourself using it.

In my case I’ve achieved such ergonomics using emacs as both my primary code editor and note-taking application. Org mode lets me capture all kinds of fleeting notes and Org-roam complements it by allowing me to quickly refile things into a personal knowledge graph. TODO lists and priorities fit nicely into the same workflow, as does time-tracking.

There is A LOT of literature on personal organization, so I don’t want to spend too much time on this topic in this post. Let’s move on to another critical tool in the battle against distractions: mastery.

The value of mastery 🧙

I briefly touched on “practice makes perfect” in produce, don’t consume and how practice can be incorporated into constructive efforts. Writing this post I’ve also come to realize that another trait of practice and mastery is how automating our skill through practice helps us cancel out distractions in order to maintain focus.

Mastering our tools is important as it allows our brains to focus on the task at hand. If we are forced to spend a significant amount of our brain power learning programming language syntax, editor bindings or APIs then we have less mental energy to spend on solving actual problems. Battling complexity on multiple fronts increases cognitive load and often a loss of focus.

A dancer who doesn’t know the basic moves of a dance style will have great trouble connecting motions in the choreography while maintaining rhythm and composure. Similarly, a programmer who constantly stumble on basic syntax or interpreting compiler errors will have a harder time constructing a mental model of the problem she’s solving.

But we do not necessary have to become specialists to become effective. In fact, becoming and staying a proper expert in more than a handful of fields is a fools errand in modern software development. Ecosystems evolve too quickly, new tools and practices come and go. Not to mention the looming paradigm shift of artificial intelligence and how it might render significant aspects of conventional programming obsolete².

What I find useful is to learn enough about a wide variety of topics to build a basic intuition for when specific technologies or methodologies will help solve the problems we’re faced with. You’ll then eventually start picking out what matters to you and your expertise will grow somewhat organically³. Also try to surround yourself with smart people complementing your own abilities.

Practice and repetition is crucial to become a master in just about anything. A musician spends time with her instrument to improve, programmers spend time with their “tech stacks”, editors and tools. It’s a matter of starting somewhere. Experts weren’t born experts. And even though talent or determination allow some to progress faster than others nobody who achieve mastery within any discipline will admit to not having worked hard – or for a long time – to acquire the skill they possess.

Just as a musician is likely to butcher an unfamiliar instrument, a programmer dropped into a new development environment will most definitely experience some initial regression in their productivity. This is normal, but it is also the way we learn. The difficulty is knowing (or choosing) what to learn, and when. In the middle of a high-stakes project is probably not the right time to pick up a new language or framework⁴.

Whether you consider yourself a specialist or a generalist I believe software development is quite unique in the way many skills are transferable across environments. Some skills invested transcend their technologies almost entirely due to their pervasiveness or general applicability.

Typing speed

Muscle memory and habits are both important abilities of the body and mind to increase our efficiency and reduce our cognitive load. Making things more automatic allows us to complete repeatable tasks faster or do so without sacrificing our ability to reason about the problems we’re currently solving.

Most seasoned programmers know that we spend significantly more time reading and navigation through code than actually changing existing code or writing new code. For this reason it’s easy to assume that learning effective typing is secondary to learning how to structure programs really well. I don’t disagree, yet I have the impression that many of the really amazing programmers I know are also great typists.

I believe there might be a very simple reason for this correlation – besides the obvious take that great programmers spend a lot of time in front of their computer, thus become great typists. No, I believe by being able to generate many programs fast you are also able to attempt many possible solutions faster. Difficult problems require thinking, I know. But few things beat the speed of finding code problems as actually passing code through a compiler or an interpreter and executing it.

I would say it’s worth putting effort into improving your typing. Not with the end-goal “writing as fast as possible” in mind of, but rather to interface as efficiently as possible with the computer. Why then not also try to improve how you type? Let’s take vi/vim and modal editing as an example.

Vim bindings

In (neo)vim there are keybindings bound to operations which operate on various textual structures.

Bindings may operate on “bodies” of text like letters, words, sentences or paragraphs. Or they operate on pairs of quotes, parentheses and brackets. They can work on locations like the beginning and end of a buffer. Or even metadata not represented in the buffer itself, like the location of compilation errors from an external tool or spelling errors from a spell checker.

Once vim motions are internalized it’s amazing how efficient it feels to “delete all word” (daw) or “change in paragraph” (cip) without breaking a sweat. The great strength and power of vim motion bindings come from how a few handfuls of general-purpose operations translate across different types of textual motions, from prose to any style of programming and markup languages.

There are many ways people can argue that learning vim-style modal editing helps their productivity. It can, however, be somewhat tricky to determine what exactly people benefit from it. Personally I’m certain the time I invested into learning motion bindings and compound operations many years ago has reaped dividends way past the initial investment.

Optimize the common paths

But how? By letting me type faster? What has typing and vim motions to do with avoiding distractions?

Everything!

The keyboard is still my main interface to the computer. As long as it continues to be it matters to me to wield it well.

Similarly to vim motions most mainstream $SHELL prompts default to emacs-style readline bindings⁵. To me that was motivation enough to put some effort into internalize the most common readline keyboard bindings on the CLI (for emacs users this is trivial).

Raise your hand if you know how to undo edit operations in readline ✋ (yes, C-/ and C-u C-/ works in your prompt!)

Programming is littered with micro-distractions, remember? Not only do you increase your efficiency at typing out actual programs by learning effective text navigation and manipulation (as well as text generation through snippets and generative A.I.). You also do so while sparing your brain from having to think about these trivial “problems” and road-blocks.

This is just as true in your editor as outside of it.

It’s not uncommon to find myself thinking about my next steps while doing some other thing like text navigation or manipulation using either vim or readline keybindings. By now they’re so ingrained in muscle memory that I more often that not think about what I need to do, not how. It just happens.

And let me say that the point of this section is not to boast or rave about vim and emacs bindings. The point is that by learning your tools really well they eventually get out of your way and you’re free to tackle the actual problems you’re faced with. It’s even better if these skills are applicable in a wide variety of applications and environments – try to stick to defaults.

Braaains 🧠

I don’t mind learning, but I want to choose – within my power – when, how and how much to learn. After all, learning is all about rewiring our brains. It’s mental exercise, and like physical exercise learning can be very exhausting. The same goes for problem solving – it’s our brains applying our existing knowledge in order to achieve something new.

Knowing when to invest time into learning, automation or optimization is not trivial. Done prematurely there’s the risk of sinking costs into efforts that won’t pay off, yet stubbornly sticking to old ways could be impeding future efficiency. A big part of maturing as a software developer is knowing (or more likely, sensing) when the time is right for you.

The biggest bottleneck of software development is still the (human) brain. It’s a precious resource, but it can only do so much and distractions greatly reduce its capacity.

Through practice and mastery, by building and honing physical and internal tools, we can offload our brains to focus much more of its effort on the problems we’re solving. By allocating more brainpower into solving problems we’re also more likely to succeed in solving some of the hard ones, and to more gracefully recover from distractions when they occur.

Now, what was I doing again…?

Footnotes

Prelude

I grew up with the internet.

It’s mind-boggling how much it has changed since the first time my dad and I plugged in our brand new i486 computer into some dial-up modem in the early 90s. We had no idea what we were doing, and I have very few recollections from this first household machine. Ironically, back then I was too physically active, too much into sports and outdoor activities. My pre-adolescent ants-in-my-pants attention span wouldn’t let me learn enough about computers to have them ensnare me.

Fast-forward to today: While short-form dopamine-triggering micro-blogs and reels on X, TikTok, Instagram et. al. are eating away our attention span, the siren song of hours-long podcasts, streams and in-depth blog posts lure us into never-ending rabbit holes of deep knowledge and insight. Short-form is addictive. Long-form is addictive. But we need to set our own boundaries because we can’t ever consume it all.

Back in the early 90s this internet thing was new to basically everybody. You had to spend significant effort actively seeking relevant content, given you knew where to look. I mean, just getting and staying online was a challenge:

Martin, are you using the … internet!? I need to make a phone-call!

A little under a decade later the situation had improved, slightly. Me and my high-school friends – at night-time, on our home ISDN connections – would round-robin share the responsibility of downloading the last week’s Futurama episode (in RealMedia) and burn it out to a CD-R(W) and bring it to school.

These days content jumps into our face and we seem to have an insatiable appetite for it. A thirst that’s driving a gold rush of content producers, many only in search of fame and fortune. As a consumer it’s a wild luxury to have instant news, knowledge and entertainment at our fingertips. However, self-regulation is vital to not lose ourselves indulging in this luxury.

Perhaps our relatively new-found superpower of having incomprehensible amounts of human knowledge, culture and talent directly accessible is overstimulating and perverting some age-old survival trait. There was a time where not learning certain abilities, latest events or social movements within a tribe would in the very least reduce our likelihood of mating success. In the utmost consequence ignorance would lead to exclusion or an inability to self-sustain and ultimately a certain death. Knowing meant surviving.

From the start of the information age and beyond the consequences of not knowing much less so imply our own personal demise. It’s just that our genes and brains have not had the time to adapt. Meanwhile, the content produced and the technological advances making it accessible show little sign of slowing down.

<pause-for-dramatic-effect class="spinner manual-breathing" />

But enough about me waving my proverbial stick and waddling my way through some intuition-based reverse engineering of anthropological behavior. Hopefully it’s illustrative for where I’m coming from and helps back up the underlying motivation for what’s to come below.

Produce, Don’t Consume

I don’t remember exactly when, but somewhere within the last couple of years I felt an urge to set myself a strategic long-term goal: Default to produce rather than to consume.

I meant this in a general sense, basically to focus less on being an active or passive receiver of input and focus more on trying to be a producer of output (or outcomes). It would imply engaging less with multi-media: Less TV and shows. Digesting less social media and to a certain degree blogs. In social settings to be engaging and a driver of conversations¹. Another example could be to spend less time in awestruck paralysis watching virtuosic guitarists, more time “violating” my guitar. And obvisouly actively programming more than consuming programming content.

What’s wrong with being a consumer, you may ask? Well, in a way, nothing.

And here I need to be perfectly clear: My motivation never stemmed from any idealistic standpoint against the state of the modern internet nor the World in general, as the prelude might have lead you to believe. Yes, there is enough content out there to drown everybody tenfold. But I don’t necessarily have an issue with that. In fact, looking back I don’t know what made me set myself this goal in the first place. What I do know is that it was my intuition trying to tell me something. It was definitely a deep and vague sensation. A gut feeling.

Content inflation

“But, all this talk of information overload” – you may wonder – “wouldn’t urging yourself and other people to produce more simply lead to an acceleration of content inflation?”

First off, I did just state that I don’t necessarily have a huge issue with the amount of content on the internet.

Secondly, well sure, if we keep sharing everything we do or create then we’re obviously contributing – perhaps negatively – to the cacophony. Yet it’s not on us to decide what might appeal to others. I’m of the opinion that certain things are good to keep to yourself, it’s either very personal or simply knowing you did it is fulfilling enough.

In the end, a piece of good work should fill us with a sense of pride, and few other things are as inspirational and likely to spawn new ideas as reciprocal engagement. It’s important to distinguish between what’s sensible self-censorship and what’s self-deprecation due to a lack of confidence. Sharing and getting feedback is how we learn and improve, and as we improve we also gain confidence².

Furthermore, consuming content is voluntary. Producers shouldn’t feel a responsibility catering for the (potential) lack of self-regulation of their consumers. I mentioned that consumption is addictive to a degree – and some people might require help to escape it – but in the end it’s our responsibility to regulate and protect ourselves from information overload. Algorithms try to “help” us, and do to a large extent, but also trap us in echo chambers of monotony in order to drive engagement. I sincerely hope we continue to evolve the tools that help us navigate the information jungle, discovering new pockets of friendly people and relevant-to-us content, without click bait sensationalism.

Practice makes perfect

As I’ve further reflected around this strategy³ I’ve started forming an understanding of what my intuition was trying to tell me: “Produce, Don’t Consume” really was derived from the famous saying Practice makes perfect.

I wasn’t imposing on myself to create more content, be more visible or aspire for anything in particular. Rather, I was urging myself to try to stay active and constructive. I wanted to make sure I continued to apply my existing knowledge while picking up some new along the way. It was the kid in me remembering the joy of learning through experimentation and creation. And I was protecting myself from stagnation.

By defining myself as a producer I was forcing myself to actively do stuff™. “Repeatedly doing stuff” is what we commonly refer to as practicing. The acts of producing and practicing relate to one another, although the two are definitely not synonymous. Depending on what you produce, you end up practicing in various ways:

• Repeatedly producing similar things, you practice and maintain the skills that go into producing that thing.
• Producing something you have not produced before practices your innovative and problem-solving ability.
• By adopting a producing mindset, you generally practice staying active and constructive.

Another way to look at it is that production is an offensive strategy – you take the initiative and act. Whereas consumption is defensive – you react based on external events or stimuli. This directly maps to the concepts of active and passive learning.

I can’t stress this enough: You learn better by doing, and by doing you produce. So in reverse, by choosing to produce you effectively practice; be that existing skills or learning new ones. Doing both you maintain and evolve.

One can evolve ones abilities through consumption and passive learning too, but I find it serves best as the catalyst to inspire and ignite the much more effective skill building that comes from practical application. Some skills simply can’t be learned solely through consumption.

Nobody learns to swim or drive a car just by reading a book. Likewise, you will never achieve programming fluency by reading programming books and blogs alone. Good books are essential learning tools, but you are yourself responsible for internalizing their intuition and knowledge into your brain through doing.

Outcomes?

So what’s the result of this? What did I produce?

Honestly? My public-facing output has not changed much at all. From a tech perspective my blogging frequency has stayed equally infrequent. I have not made any significant public contributions to open-source. I have not engaged more on any social medium. But that’s what you see (if you were looking my way).

Professionally I can look back at a couple of years with new job opportunities where I’ve been able to significantly increase the impact of my own efforts towards engineering progress. Privately I’ve enjoyed some simple home- and personal-improvement projects, like woodworking and picking up my guitar almost every day – if only for a few minutes. I’ve also been spending considerable amounts of time on private and personal coding projects⁴.

The “produce, don’t consume” strategy has been defining in how I’ve chosen to spend my time and manage my interactions over the last few years.

In my opinion it wasn’t that much about what I ended up producing. What I have realized, however, is that simply by having a simple strategy I’ve been able to maintain a steady course and make decisions with a new level of confidence.

Software is weird. In its basic form, software is this very abstract thing with seemingly little connection to the real world. Yet graphical applications running on a hardware device like a modern phone feel very real, even to people who have no appreciation of what happens within this “black box”. Some pieces of software is even able to control hardware to perform real actions in the physical world.

As programmers, some of the code that we write is supposed to model real world concepts, and as a result the code often reflects this in its structure and form. Other pieces of code are purely abstract and solve “metaphysical” problems that only exist within the realm of computers. Code solving abstract problems might also serve as necessary foundations to build more tangible “real world modelling” on top of.

At the end of the day code has no obligation or enforced requirement to resemble the problem it’s supposed to solve – and sometimes there isn’t even a “real” problem to compare it to in the first place. Of course, code which is able to somehow communicate this correlation tends to be considered more “readable” and “maintainable” simply due to familiarity. But the purpose of the systems we build is first and foremost “to model”, not simply “to be”¹.

Software is a means to an end and as long as the behavior of our model solves our problem well enough users tend to be satisfied.

• Me @RightNow

Performance considerations aside, computers do not care at all about the structure of the programs they’re evaluating. A legacy spaghetti mess of a codebase might solve a problem just as well as any well-crafted system. The cost of poorly constructed solutions only becomes relevant when we factor in the fact that humans tend to have to come back and make changes to them.

This highlights one of the more hidden, but considerable challenges of being a programmer: Having to translate real world requirements into abstract constructs that – when evaluated by a computer – model our problem domain to a satisfactory degree. And we have to do this while also considering how we best can model our abstractions so both our future selves and our peers are able to make sense of them.

Much of the online discourse about the complexity of programming tend to focus on complexity associated with programming languages, tools and technology stacks that we use. Not so much on how we can more effectively understand requirements and translate these into code. Neither how we can ensure that these requirements can be communicated and maintained over time as our system evolves. This is not to say there are no such efforts. I myself very much enjoyed attending the 2023 edition of DDD Europe where this exact problem is front-and-center.

Learning a programming language syntax and semantics can be both time-consuming and mind-bending². The same goes for tools, services and infrastructure that’s central to building modern applications and services. And not only that! Many of these technologies change at a rapid pace, so even keeping up with updates to things we already though we knew can feel overwhelming. But let’s not also forget or neglect our responsibility to understand the people and the world around us – and how crucial this understanding is to ensure quality and purpose in the majority of the code that we write.

Yeah, programming is hard! I mean, like, really hard!

Last month I published the blog post Git Commands You Probably Do Not Need. It seemed to have caused a bit of a buzz as it peaked on the orange site and other aggregators. I received quite a bit of pleasant feedback from it, which I’m thankful for.

I write blog posts because I enjoy writing and I don’t feel like I have any agenda besides the creative outlet it provides. I also have no immediate goal of defining myself as a content creator simply because I do not have any expectation for this blog to ever generate any significant revenue. I try to keep a healthy balance between not feeling pressured into producing content while maintaining the necessary personal discipline required to finish off a post.

That said, I did feel a rush knowing that stuff I’d written struck a chord with the community.

Being in the “15 minutes of fame” limelight there’s no doubt there would have been a periodic surge of traffic to the site. I wouldn’t know the numbers though, because I’ve been reluctant to add any analytics or metrics to the site at all. In the end, curiosity got the best of me, and so I went in search of a privacy-minded, light-weight solution able to provide me with some basic visitor insights.

Google Analytics & access logs 🪵

Several years ago I did have Google Analytics integrated and running on the site for a while. However, with the increased focus on privacy through GDPR, cookie consent and what have you I saw no reason to keep tracking users for the relatively low volume traffic I was receiving. After all, at that point I was running a static website off a Linode nanode VPS with an nginx web server serving static files. If I wanted to check for traffic, I had the basic access logs from nginx¹.

A little under a year ago I wrote another another post about my dotfile history which seemed to attract the attention of the interwebs briefly appearing on the orange site front page. At that time my little Linode server was happily serving off the static website and I was able to get some sense of the page views based on the nginx access logs.

GitHub Pages 📃

Some time later I decided to simplify my blog deployment by hosting it off GitHub Pages. Although I had no big issues with the Linode hosting it just didn’t make much sense to me with regards to the manual management of deployment and operations vs. the reliability and performance of Pages. Adding a GitHub workflow triggering a nix build of the static assets then deploying them was just too simple to ignore:

The major disadvantage of switching to GitHub Pages was to forfeit any kind of application insights. It didn’t bother me at first, because all I wanted was to ensure that the blog was up². However, once a post blew up it would be interesting to at least have some numbers reflecting the amount of traffic going to the site.

Plausible Analytics 📈

I must admit that I didn’t spend much effort researching the alternatives for privacy-minded, cookie-less analytics solutions. Coincidentally, I recently found A Brief Comparison of 10+ Best Google Analytics Alternatives on Lobsters which does seem to go into a decent selection of analytics alternatives.

Before reading the above summary I had already decided on Plausible. Primarily because of a good first impression and some second-hand experiences from others.

Hosted vs. self-hosting

Plausible provides a hosted service for what seems like a decent price. What sold it to me though was the fact that it’s open source and quite simple to self-host. I wanted to go down the self-host route because I still have some VPS servers for other minor things which should quite easily manage the additional overhead of running the Plausible backend service.

If you’re here just for adding some analytics then I would definitely suggest to give the hosted solution a go, both for simplicity and for financial support to the Plausible team.

Setup

The github.com/plausible/hosting repository contains the necessary docker compose setup to spin up the self-hosted setup, as documented in the Self-Hosting section of the Plausible documentation. The steps are replicated here for my own sake. I would direct anybody eager to try out self-hosting Plausible for themselves to first and foremost consult the official documentation.

Here are the steps:

1. Clone the hosting repo

   $ curl -L https://github.com/plausible/hosting/archive/master.tar.gz | tar -xz
   $ cd hosting-master

2. Add required configuration

   Generate a secret key for the application by setting SECRET_KEY_BASE in plausible-conf.env.

   $ openssl rand -base64 64 | tr -d '\n' ; echo

   Also, set the BASE_URL by specifying the base url of the instance, including the URL scheme (e.g. https://stats.example.com).

3. Start the server

   $ docker-compose up -d

At this point the service should be accessible on http://localhost:8000. It’s highly advisable to setup a reverse proxy with SSL in front though.

Graphs

And bam! Stats are live and stuff seems to work:

My numbers won’t impress anybody at this rate, but again, that’s not what I’m here for.

Updating Plausible

The self-hosting docs states that the self-hosted version is “somewhat of a LTS”. This means self-hosting using the Docker Hub containers won’t contain the latest and greatest Plausible features. It also means that updates will be less frequent. However, there will be a need to upgrade Plausible at some point, which is basically just a matter of pulling new Docker images and restarting the services:

$ docker-compose down --remove-orphans
$ docker-compose pull plausible
$ docker-compose up -d

Conclusion

There’s not really much to conclude after enabling and running these analytics only for a few days. Getting started with self-hosting Plausible was a breeze and I see numbers come in. Hopefully the numbers are fairly accurate. My personal browser privacy plugins denies the Plausible hits from my browser to be registered. Which means that many page views will never reach my backend. I don’t mind. It’s motivating just to see my articles having any kind of reach, and I hope people will continue to find some of them either insightful or interesting.

Footnotes

Ah, git! Love it, hate it. Few things are as central to the modern software development workflow as source-control management (SCM) tools. Although there have been and still are plenty of alternatives to git in the world of SCMs, none other seem quite as prevalent both in open-source and the enterprise.

Regardless of how central git has grown to be for many (most?) software developers, I frequently get the impression that people have a tendency to shy away from anything beyond the relatively basic functionality it provides.

Since its very inception git has been notorious for its often unfriendly, inconsistent and occasionally hostile command line interface:

In this post I’ll present a few git commands and operations I run or have run on occasion, in no particular order, that the majority of git users out there might not ever need.

The empty commit ∅

git is designed to track content and changes to that content over time, so creating empty commits doesn’t sound like a very productive or sensible thing to do. Not adding any content might seem like a waste.

Yet, I know off the top of my mind at least two occasions where creating an empty commit can make quite a bit of sense:

1. Initializing new repositories.
2. Triggering continuous deployment pipelines.

1. Initializing a new repository

My main reason for using the --allow-empty flag to git is to create an initial commit in a repository before I have any content for it.

Why?

Primary reason for me is that git rebase is somewhat troublesome when wanting to rewrite the first commit of a repository. The rebase command centers around working against an <upstream> commit¹, and by definition the first commit of a branch has no upstream. In later versions of git the --root option has been added to remedy this, but I still like to use the opportunity to start any new line of history with an empty commit:

❯ git commit --allow-empty -m "Initial commit"
❯ git show --stat 592f9dd06d1013e4b5300311e6c1b7033c17ab9b
commit 592f9dd06d1013e4b5300311e6c1b7033c17ab9b
Author: Martin Øinæs Myrseth <myrseth@gmail.com>
Date:   Thu Dec 21 21:25:24 2017 +0100

    Initial commit

2. Trigger a build

Depending on your continuous integration/deployment setup chances are that automated work is scheduled based on updates to some git repository. Alas, flaky network and computation resources, tests and dependency management often come in the way of a flawless pipeline execution.

Creating empty git commits and pushing them to a PR might be all you need to scratch that “let’s just try it once more, without changing anything” itch.

Pushing locally 💨

It’s hard to use git for long without encountering git push, the git sub-command for transferring (copying) stuff from your local repository to some other, remote git repository. The format of the command I’m sure most people are familiar with is:

git push <remote>

Or for the more adventurous:

git push <remote> <refspec>

Now what if I told you it’s possible to push a remote reference to your local repository, effectively reversing the direction of the push operation?

git push . origin/main:main

Why on earth would you ever want to do this? Haven’t you heard of git pull?

Well, first off git pull is a combination of two git operations: git fetch followed by either a git merge or a git rebase depending on user configuration. In other words, a fetch followed by an integration².

Since the integration can result in conflicts, git will only allow you to integrate into the current branch. After all, conflicts must be resolved in the worktree, and the worktree sees only the current branch (by definition). This is not always desirable.

Let’s say that you have been working for a long-ish period of time on a feature branch, leaving your local copy of the project’s main branch significantly out of date. For whatever reason (diffing, cherry-picking, etc) you want to update main with the latest upstream changes, but you don’t want to navigate off your feature branch³. One reason might be to avoid invalidating much of a slow-running incremental build.

In this case we specifically do not want the integration phase of a git merge or git rebase, rather we just to update a branch without having to check it out first⁴. As it happens, this exactly what git push does: When git push pushes to a remote repository, there is no opportunity to resolve conflicts, instead the update either works (fast-forward, or maybe you’re using --force) or it is rejected: there is no integration phase.

We just need to turn the direction around: Instead of pushing a local branch into the remote repo, we want to push a remote branch into the local repo. The dot (.) target takes care of the local repo part, but we need to cheat a little to get the remote branch. As already mentioned, git fetch will fetch remote branches into the local repo, and update the corresponding remote-tracking branches in your local repo[^remote-tracking-branches]. From there, you can now use the remote-tracking branch as the source of your `git push`:

# update remote-tracking branches (incl. origin/main)
git fetch

# Update main branch in this repo to point the same place as origin/main
git push . origin/main:main

git provides the same branch protection rules as when pushing to remote repositories. Which is to say that it denies non-fast-forward pushes by default, but allows overrides through +<commit>:<ref> refspecs, or the --force and --force-with-lease command line options.

Commit ranking 🥇

Perhaps in need for something to serve as the year-end bonus rounds tie-breaker? What better way to settle the implicit battle between your peers of “who’s providing most value” than by having a “git commit count” showdown?

No, that’s a horrible idea …

Yes, yes it is. Any sensible developer knows that nothing good ever comes out of placing merit in lines of code changed or number of commits committed.

But should you, God forbid, ever be in need to know (for the sake of curiosity) who’s been committing the most to a repository, here’s git rank:

git shortlog -s -n --no-merges

Configure it as an alias in ~/.config/git/config with:

[alias]
  rank = "shortlog -s -n --no-merges"

and simply run:

git rank

As a quick example, behold, here’s the horrendous output from my own dotfiles repository, where I’ve been able to make commits under different names and identities:

❯ git shortlog -nse
   567  Martin Øinæs Myrseth <myrseth@gmail.com>
   322  Martin Øinæs Myrseth <mmyrseth@cisco.com>
   142  Martin Myrseth <mm@myme.no>
    14  Martin Myrseth <myrseth@gmail.com>
     4  Martin Øinæs Myrseth <mm@myme.no>
     3  Martin Myrseth <myme@map>
     2  Martin Øinæs Myrseth <myme@Tuple.localdomain>
     2  Martin Øinæs Myrseth <myme@map.localdomain>

It’s painful to read, I know. Try to imagine the pain and embarrassment it is for me to share it. And have you been unfortunate or careless enough to get yourself into a similar situation, please read on. I’ll revisit this problem in the filter branch section below.

Cat file 🐈

This is more of a git party trick than anything to actually make much use of. But I should say I have made use of git cat-file -p on a couple of occasions to help people visualize and actually grok git’s data model.

As the name hints at, the cat-file command outputs information about git objects. I’ve personally only used cat-file with the -p (pretty-print) flag, which first determines the type of the object before printing it out. Let’s start off with inspecting the HEAD commit:

$ git cat-file -p HEAD
tree 9491ada70010d722646b674d2e2a26521628df94
parent 9d7e5a6490c9f560f54fee9e1af5d72429bb26c7
author Martin Myrseth <mm@myme.no> 1665439490 +0200
committer Martin Myrseth <mm@myme.no> 1665439490 +0200

Delete Docker deploy action

We see the main metadata that git associates with a commit: the repository file structure (tree), a parent commit SHA1 reference, author information and finally the commit message after a blank line. Let’s dig further by passing the SHA1 (9491ad..94) of the tree associated with the HEAD commit:

$ git cat-file -p 9491ada70010d722646b674d2e2a26521628df94
040000 tree 6d71faa5d70085c5d07228d8fa522fb712253b6d    .github
100644 blob e09fe0dc282fdcaff06bcc6a9bbf57cbfc845eb4    .gitignore
100644 blob da7e7945524871726071f919690c9c9f6c1e173d    README.md
100644 blob e6be557357c3fe2e3cce6f1b7b9b3c9c55981a16    default.nix
040000 tree 4f69a79c432cde80b4a1c486974b03cab84b45b9    docker
100644 blob 2f8aacd9efa3cfdf9e5f2860fa7226b510ed83bc    feed-cors.conf
100644 blob 14b9e2dd0a41aa932c1f4bb5938519547f37f82c    flake.lock
100644 blob eeae336837db94ca62255a7e5fa7f32ae3363716    flake.nix
100644 blob f1f8ef836b3b9b9ea011a43972a28ffaa713c868    image.nix
040000 tree 5cad033d973f19ece938c33c3bb912eb63dc3305    site
040000 tree 49dc35d8e519f02f6f1a647f437226af198d225a    ssg
100644 blob 60dede4bba8cd9479b0bec49048da1397e14f352    todo.org

The result of printing a tree is what looks like a directory listing of the contents of that “tree” directory. Each directory entry is represented as some mode bits, an object type, the SHA1 of the object and the name of the entry. Trees may contain other tree objects to create a directory structure, or blob objects which contain file contents.

Finally, let’s inspect one of the blob in the output, like .gitignore (e09fe0..b4):

$ git cat-file -p e09fe0dc282fdcaff06bcc6a9bbf57cbfc845eb4
.stack-work
_cache
public
dist-newstyle
.ghc.environment.*

# nix-build
result

Which prints out the actual content of .gitignore the way it was committed into the current HEAD commit.

Wait? What? So everything is just text files?

Conceptually, yes. However, modern git does a lot more to optimize storage (re)usage and whatnot to ensure that a repository stays as small as possible. There are other, scarier objects lurking under .git/objects in a git repository.

The git parable

As I said in the beginning of this section, I’ve used cat-file to help myself and others understand the git object model. Learning all the details of that model isn’t the purpose of this section (or post) though. However, if reading this ignited some form of curiosity on your part I would gladly recommend the talk “The Git Parable” which dives deeper into the git object model, as presented by my good friend Johan Herland:

Use case

Now, why would you want to use cat-file? (Except you wouldn’t, but let’s just play along here)

I was deep into some refactoring and clean-up of a set of template files used for various messages sent out from a system. Each template directory would contain a set of files for each message template. I’ve been working with the files for a while when a feeling grew on me that several of these templates seemed to be fairly similar, identical in fact.

At this point I had already been making some work-in-progress commits, which would definitely get in the way for any attempt at checking if there were identical template directories in my working copy. I wanted to compare the contents of the template directories at the point before I started making my changes.

The primary tool for checking differences to files in git is obviously the git diff command. It can easily check the differences between files stored in the git history. Typical usage of diff is to compare a single path across various versions. However, looking closer at it’s synopsis we can see that there are a couple of call signatures that might do somewhat what we need:

NAME
       git-diff - Show changes between commits, commit and working tree, etc

SYNOPSIS
       git diff [<options>] [<commit>] [--] [<path>...]
       git diff [<options>] --cached [--merge-base] [<commit>] [--] [<path>...]
       git diff [<options>] [--merge-base] <commit> [<commit>...] <commit> [--] [<path>...]
       git diff [<options>] <commit>...<commit> [--] [<path>...]
       git diff [<options>] <blob> <blob>
       git diff [<options>] --no-index [--] <path> <path>

Primarily git diff <blob> <blob> which would let us compare any two git blob objects. There’s also a note under “DESCRIPTION” which states:

Just in case you are doing something exotic, it should be noted that all of the <commit> in the above description, except in the --merge-base case and in the last two forms that use .. notations, can be any <tree>.

Which means that also git diff <blob> <blob> should do something along the lines that we want. And surely, doing something similar to the following yielded an empty diff (where HEAD~3 is the commit I based my work on):

❯ git diff HEAD~3:some/templates/path/ HEAD~3:some/templates/other-path/

So I had found one pair of templates that were identical, and which could be coalesced into one. But what if there were more? Using git diff alone I would have had to compare all permutations of template directories to see if the results were empty.

No time for that…

Instead we can use cat-file to simply dump all the hashes of every sub <tree>. Then we can use a familiar shell pipeline to group the hashes and count them:

❯ git cat-file -p HEAD~3:some/templates/ \
  | awk '{ print $3; }' \
  | sort \
  | uniq -c \
  | sort -rn
      2 af83bb357f2b8dc42f6c9f07620140590dc7fd44
      2 228182da5a0ffcf4c0d263bfa54852176f0250d2
      1 ef1a471185c2092e6708349fa710702dd416f892
      1 e453cb9d3dddbdd46a65c811068352ac40941fcd
      1 e3df1181dae478172a7ae6bbc1618a3af2151db4
      1 de0f6cd53ea97cb100a74c812f75c0d4844c0efa
      1 d7f239da6283c927dad650599d49639ddc761465
      1 d7d8f5aa3571ea2392028e353ad958d778d2bee0
      1 cc03005d684b5735da337a6e5ca9765751943d7d
      ... # A bunch more

Et voilà! We clearly see that there are not just one pair of duplicate templates, but two!

Orphan commits 🤷

Every commit in a git repository has a reference to its parent, which is the commit that chronologically came immediately before the commit. For merge commits the number of parents are greater than one.

Well, that’s not 100% accurate. As discussed in the empty commit the initial commit of a branch is somewhat special: it has no parents. Commits without any reference to a parent is called an “orphan” commit. In most repos there would only be one such commit, the initial commit.

However, git is by no means limited to a single orphan commit. The default behavior when creating a new branch is that the new branch is based on some start-point. Using git checkout --orphan (or the currently unstable git switch --orphan) it’s possible to start off a completely new and independent line of history totally disconnected from the rest of the repository.

The main use-case I’ve had for git’s support of this functionality is not to start “orphaned” histories, but rather absorb the history of a branch from another, unrelated repository. It’s very useful when coalescing many smaller repositories into a monorepo or when vendorizing some library.

Merging histories

As a synthetic case-study let’s import the doomemacs history into my dotfiles repo!

First let’s create a new worktree so that we don’t mess up my actual files:

❯ git worktree add ~/code/doomfiles doomfiles
Preparing worktree (checking out 'doomfiles')
HEAD is now at a0b32f8 machine: deque: Setup nginx with rtcp.myme.no
❯ cd ~/code/doomfiles

Doing a git log of the most recent commits we can see that they’re all mine:

❯ git log --oneline --graph -5
* 0f1f6cd machine: map: Enable podman
* 46099b9 emacs: Add React fn-component snippet
* 2e75458 ssh: Update hosts
* 445ade4 machine: deque: Set SSH port
* bf0a552 flake: Add utils as "apps"

Another “little known” feature of git is that it’s trivial to fetch “a random” upstream repository without adding an explicit git remote. This can be quite useful when e.g. checking out some incoming one-off contribution. Just pass the remote url to fetch directly:

❯ git fetch git@github.com:doomemacs/doomemacs
remote: Enumerating objects: 118606, done.
remote: Counting objects: 100% (20/20), done.
remote: Compressing objects: 100% (17/17), done.
remote: Total 118606 (delta 4), reused 15 (delta 3), pack-reused 118586
Receiving objects: 100% (118606/118606), 26.98 MiB | 6.80 MiB/s, done.
Resolving deltas: 100% (82950/82950), done.
From github.com:doomemacs/doomemacs
 * branch                HEAD       -> FETCH_HEAD

As the output states, the result of the fetch is placed in the special git ref FETCH_HEAD. We can use this ref to refer to the doomemacs commit that was fetched when we wish to merge the histories.

Now, git won’t let us merge without warning us first:

❯ git merge FETCH_HEAD
fatal: refusing to merge unrelated histories

Easily enough we can add the --allow-unrelated-histories telling git we’re being quite serious right here:

❯ git merge --allow-unrelated-histories FETCH_HEAD
Auto-merging .gitignore
CONFLICT (add/add): Merge conflict in .gitignore
Auto-merging README.md
CONFLICT (add/add): Merge conflict in README.md
Recorded preimage for '.gitignore'
Recorded preimage for 'README.md'
Automatic merge failed; fix conflicts and then commit the result.

Pfffft, conflicts … Let’s get on with our lives by simply resetting the conflicting files to the imported versions #yolo.

❯ git checkout --theirs -- .gitignore README.md
❯ git add .gitignore README.md
❯ git commit -m 'Pulling in Doom Emacs!'
Recorded resolution for '.gitignore'.
Recorded resolution for 'README.md'.
[doomfiles 11826ae12] Pulling in Doom Emacs!

And that’s about it! Let’s inspect the result:

❯ git show
commit 11826ae125834cc4e2263172275d8c51bca11d63 (HEAD -> doomfiles)
Merge: a0b32f85f e96624926
Author: Martin Myrseth <mm@myme.no>
Date:   Thu Jan 19 01:13:17 2023 +0100

    Pulling in Doom Emacs!

We can see that the commit is a merge commit, where one parent is a0b32f85f from my dotfiles while the other parent e96624926 is the current⁵ HEAD from the doomemacs repo.

We have successfully merged the histories of my dotfiles repository with Doom Emacs!

As stated previously, this can be quite useful when pulling in e.g. an experimental repository, vendorizing some dependency or similarly constructing a monorepo from separate smaller repositories.

The next section is about one (of several) times I’ve found this useful myself.

Dotfiles bankruptcy

I agree that the previous example of absorbing Doom Emacs into my dotfiles is kind of silly, but it illustrates possibilities.

Stepping away from synthetic examples I also would like to show one of a few occasions where I’ve made use of it to solve a real-world use-case.

Let’s step back into my dotfiles.

With our new knowledge about orphan commits we may wonder if there is a way to easily query for them in a git repository. And there sure is:

❯ git log --all --max-parents=0
commit 6fa853118711f557a911b98f00d5c4a2eb3ded71
Author: Martin Myrseth <mm@myme.no>
Date:   Mon Jan 17 21:44:43 2022 +0000

    nixos: Initial commit

commit 61a3f80babec8c1339391462590dafe7ff30fe7f
Author: Martin Myrseth <mm@myme.no>
Date:   Wed Feb 10 11:59:23 2016 +0100

    Inital import of tuple

There is not one, but two commits in the dotfiles repository which doesn’t have any parents.

1. The real “Initial import” created at the beginning of time⁶.
2. The much more recent “nixos: Initial commit”.

The second commit was the beginning of my attempt to move my machine configurations towards a fully NixOS managed declarative setup built on top of flakes. I’ve already covered this journey in another post which also links to the state of my configuration management before that migration.

In any case, when starting my configuration rewrite I wasn’t yet sure if I would want a clean slate or eventually port it into my dotfiles. In the end I figured I could have both, by simply pulling in the experiment into my already existing history.

Eventually my experiment had matured to the point where I was convinced I had what I wanted. It was time to import it into the dotfiles repository. Following pretty much the same steps as in the previous section I ended up with the following merge commit:

❯ git show 79977b007099390a53e11f540e178f6285137206
commit 79977b007099390a53e11f540e178f6285137206
Merge: ad28da4 841eec3
Author: Martin Myrseth <mm@myme.no>
Date:   Wed Feb 2 00:19:24 2022 +0100

    nixos: Declare dotfile bankruptcy

Filter branch 🌿

I tend to work in a number of git repositories across various machines. I also split work between my personal projects and anything related to $DAYJOB. I do not want to taint the git history in work repositories with personal email addresses and other “unprofessionalism”.

Turns out I do though. Remember the painful output from commit ranking?

Oh, the embarrassment! It’s unbearable!

This example is was from my dotfiles history before I cleaned it up. I occasionally setup new machines, and my dotfiles repo is the first thing I pull in after the machine boots. Every host typically needs some form of tweaking, and not realizing I haven’t setup my git configurations correctly, I start patching and committing configurations for the new host.

Next thing I know I’ve completely missed the fact that I’ve been committing with all kinds of ad-hoc user settings inferred from git without letting me know.

The filter-branch cleanup

Most people familiar with rewriting git history know about git rebase and git rebase --interactive, which allow operations like “moving” (or replaying) commits onto new parents, rewriting commit messages, re-assigning author information, as well as making changes to the source tree.

Perhaps less familiar to people is the git filter-branch command, which is sort of the hydrogen bomb of history rewriting. I urge you to heed the glaring warning that meets you in man git-filter-branch(1) and perhaps consider alternative solutions like git-filter-repo:

WARNING
       git filter-branch has a plethora of pitfalls that can produce non-obvious
       manglings of the intended history rewrite (and can leave you with little
       time to investigate such problems since it has such abysmal performance).
       These safety and performance issues cannot be backward compatibly fixed
       and as such, its use is not recommended. Please use an alternative
       history filtering tool such as git filter-repo. If you still need to use
       git filter-branch, please carefully read the section called “SAFETY” (and
       the section called “PERFORMANCE”) to learn about the land mines of
       filter-branch, and then vigilantly avoid as many of the hazards listed
       there as reasonably possible.

Warning aside, a few factors lead me to believe this was what I wanted in this particular scenario:

1. All the faulty commits were fairly recent, I wouldn’t touch very old history.
2. I’ve had experience running filter-branch from way back and felt confident choosing it again.
3. The manpage has the exact use-case exemplified.

With a few modifications from this Stack Overflow answer:

❯ git filter-branch --env-filter '
WRONG_EMAIL="martin@machine.localdomain"
NEW_NAME="Martin Myrseth"
NEW_EMAIL="martin@example.com"

if [ "$GIT_COMMITTER_EMAIL" = "$WRONG_EMAIL" ]
then
    export GIT_COMMITTER_NAME="$NEW_NAME"
    export GIT_COMMITTER_EMAIL="$NEW_EMAIL"
fi
if [ "$GIT_AUTHOR_EMAIL" = "$WRONG_EMAIL" ]
then
    export GIT_AUTHOR_NAME="$NEW_NAME"
    export GIT_AUTHOR_EMAIL="$NEW_EMAIL"
fi
' --tag-name-filter cat -- --branches --tags

I don’t think there’s much reason to invest a whole lot of effort into understanding all the ins and outs of filter-branch. There are most likely always better options to solve the problems it too can solve, so try your best to avoid it.

Following are some other workarounds to avoid committing with a broken user configuration or ensuring that faults are at least concealed in command outputs.

Git templates and pre-commit hooks

Before all of the other mitigations outlined in the sections below I used to have a .gittemplates folder containing a few git hooks that would be added to every newly created repository. One of these hooks was the pre-commit hook which checked that I had a properly configured user.name and user.email.

#!/usr/bin/env bash
if !(git config user.name &> /dev/null && git config user.email &> /dev/null); then
    echo "Please setup your repository with a user.name and user.email" >&2
    exit 1
fi

If I ever forgot to properly setup particularly the user.email for a specific repository then git wouldn’t let me commit without annoying me with a warning. Since I rarely change my name (I haven’t yet), I would hardcode user.name into my user-global git configuration.

Due to the chicken-and-egg problem, these hooks weren’t created for my dotfiles repo on new hosts because they’re in the dotfiles repo. It’s a while since I abandoned this approach alltogether as it’s obsoleted by the solution of the next section.

No second guessing please!

One of the git defaults I’m not very fond of is the user.useConfigOnly configuration which is false by default. Here’s its excerpt from man git-config(1):

user.useConfigOnly
    Instruct Git to avoid trying to guess defaults for user.email and user.name,
    and instead retrieve the values only from the configuration. For example, if
    you have multiple email addresses and would like to use a different one for
    each repository, then with this configuration option set to true in the
    global config along with a name, Git will prompt you to set up an email
    before making new commits in a newly cloned repository. Defaults to false.

I guess the documentation outlines my “default” use-case, which is to use different email addresses for the repository I work in. With the following configuration git will refuse to commit when user configuration is missing, thus obsoleting my pre-commit hook:

[user]
  name = "Martin Myrseth"
  useConfigOnly = true

Git conditional configuration

It’s hard to argue against the fact that the best way to solve any problem, is to not have the problem in the first place. Using some “clever” conditional configuration sections it’s possible to include additional configurations for e.g. repositories within specific sub-directories on the filesystem, ensuring that there never is a partial user configuration.

Once I became aware of this configuration trick I took more care as to where I placed repositories on disk. Making sure to have separate directories for personal and work related repos. With this repository directory layout, it’s possible to have a conditional section in gitconfig which applies additional configurations to any repository matching the predicate (i.e. placement on disk):

[includeIf "gitdir:~/code/work/"]
    path = "./work_config"

Any repository under ~/code/work will include the configuration from ./work_config, which may contain something like the following:

[commit]
    gpgSign = true
[tag]
    forceAnnotated = true
    gpgSign = true
[user]
    email = "martin@day.job"
    signingKey = "martin@day.job"

.mailmap

Although the filter-branch command allows a full cleanup of the history of a git repository, it shouldn’t be understated the potential damage and inconvenience such an operation has on the repository integrity. Rewriting history has the viral effect of changing SHA1 sums of all subsequent commits, leading to parallel histories (old vs. new). This is most likely not what you want for public histories.

On the other end of the spectrum git provides a rather convenient and non-destructive feature to solve this particular issue through its mailmap support. Quoting the man gitmailmap:

If the file .mailmap exists at the toplevel of the repository … it is used to map author and committer names and email addresses to canonical real names and email addresses.

The man page of gitmailmap contains syntactical examples of mailmap entries. To correct a simple incorrect email one can add an entry on the format:

<proper@email.xx> <commit@email.xx>

The .mailmap can also correct user.name issues as well as correct specific commits and so on. Here’s the .mailmap file from my dotfiles which fixes up a few of my past mistakes:

Martin Myrseth <mm@myme.no> <mm@myme.no>
Martin Myrseth <mm@myme.no> <myrseth@gmail.com>
Martin Myrseth <mm@myme.no> <mmyrseth@cisco.com>
Martin Myrseth <mm@myme.no> <myme@Tuple.localdomain>
Martin Myrseth <mm@myme.no> <myme@map.localdomain>

Octopus merge 🐙

I must admit, I never use this, but I remember being amazed the first time I learned about the many-parent merge ability of git long ago.

I would assume most people live their life thinking a merge commit is just the combined result of two somewhat related histories. Ideally two histories that forked off one another in (hopefully) the not too distant past.

Yet, we’ve already seen and debunked the fact that histories have to be “somewhat related” in order to be merged. That’s what the “absorb some other repository” functionality covered in the orphan commits section was all about.

I guess then it comes as no surprise that the assumption of merges only ever having just two parents is also not a hard limitation.

Tentacles

Let’s see how we can create a many-parent merge commit, called an “octopus merge”, by starting off a new repository and adding a bunch of branches to it:

❯ mkdir octopus
❯ cd octopus/
❯ git init
Initialized empty Git repository in /home/myme/tmp/octopus/.git/

❯ git commit --allow-empty -m 'Initial commit'
Author identity unknown

*** Please tell me who you are.

Run

  git config --global user.email "you@example.com"
  ...

Ah … right. Forgot about that 🤦

❯ git config user.email 'dave@tentacle.org'
❯ git commit --allow-empty -m 'Initial commit'
[main (root-commit) 9ff0a71] Initial commit

At this point we have a new git repository with a single main branch containing a single empty commit:

❯ git log --all --oneline --graph
* 9ff0a71 (HEAD -> main) Initial commit

Let’s create some branches with content:

❯ git checkout -b tentacle
Switched to a new branch 'tentacle'
❯ date > tentacle.txt
❯ git add tentacle.txt
❯ git commit -m 'Add day of tentacle.txt'
[tentacle 4dadc16] Add day of tentacle.txt
 1 file changed, 1 insertion(+)
 create mode 100644 tentacle.txt

Yay, one limb (aka branch) in place!

❯ git log --all --oneline --graph
* 4dadc16 (HEAD -> tentacle) Add day of tentacle.txt
* 9ff0a71 (main) Initial commit

But creating limbs is tedious. Let’s push the fast-forward button:

for count in nine eight seven six five four three two one; do
  limb="${count}tacle"
  git checkout -b "$limb" main
  date > "${limb}.txt"
  git add "${limb}.txt"
  git commit -m "Add ${limb}"
done

Switched to a new branch 'ninetacle'
[ninetacle 3f7a95e] Add ninetacle
 1 file changed, 1 insertion(+)
 create mode 100644 ninetacle.txt
Switched to a new branch 'eighttacle'
[eighttacle e9cd39a] Add eighttacle
 1 file changed, 1 insertion(+)
 create mode 100644 eighttacle.txt
Switched to a new branch 'seventacle'
...
Switched to a new branch 'sixtacle'
...
Switched to a new branch 'fivetacle'
...
Switched to a new branch 'fourtacle'
...
Switched to a new branch 'threetacle'
...
Switched to a new branch 'twotacle'
...
Switched to a new branch 'onetacle'
[onetacle c78c58a] Add onetacle
 1 file changed, 1 insertion(+)
 create mode 100644 onetacle.txt

And we got ourselves a bunch of limbs!

❯ git log --all --oneline --graph
* e9cd39a (eighttacle) Add eighttacle
| * e310cbc (fivetacle) Add fivetacle
|/
| * 44ad755 (fourtacle) Add fourtacle
|/
| * 3f7a95e (ninetacle) Add ninetacle
|/
| * c78c58a (HEAD -> onetacle) Add onetacle
|/
| * 6be7cf4 (seventacle) Add seventacle
|/
| * a54e5c1 (sixtacle) Add sixtacle
|/
| * 3b1a5da (threetacle) Add threetacle
|/
| * bb79112 (twotacle) Add twotacle
|/
| * 4dadc16 (tentacle) Add day of tentacle.txt
|/
* 9ff0a71 (main) Initial commit

Time to assemble our squid:

❯ git merge tentacle ninetacle eighttacle seventacle sixtacle fivetacle fourtacle threetacle twotacle onetacle -m 'Assemble squid'
Fast-forwarding to: tentacle
Trying simple merge with ninetacle
Trying simple merge with eighttacle
Trying simple merge with seventacle
Trying simple merge with sixtacle
Trying simple merge with fivetacle
Trying simple merge with fourtacle
Trying simple merge with threetacle
Trying simple merge with twotacle
Trying simple merge with onetacle
Merge made by the 'octopus' strategy.
 eighttacle.txt | 1 +
 fivetacle.txt  | 1 +
 fourtacle.txt  | 1 +
 ninetacle.txt  | 1 +
 onetacle.txt   | 1 +
 seventacle.txt | 1 +
 sixtacle.txt   | 1 +
 tentacle.txt   | 1 +
 threetacle.txt | 1 +
 twotacle.txt   | 1 +
 10 files changed, 10 insertions(+)
 create mode 100644 eighttacle.txt
 create mode 100644 fivetacle.txt
 create mode 100644 fourtacle.txt
 create mode 100644 ninetacle.txt
 create mode 100644 onetacle.txt
 create mode 100644 seventacle.txt
 create mode 100644 sixtacle.txt
 create mode 100644 tentacle.txt
 create mode 100644 threetacle.txt
 create mode 100644 twotacle.txt

The end result is the most wonderful git graph ever!

We’ve managed to create a new commit in our repository with no less than ten parents. We can also confirm this using git show:

❯ git show
commit 442b9a2852fc2707517690f1a994c1c5a38ac20b (HEAD -> main)
Merge: 4dadc16 3f7a95e e9cd39a 6be7cf4 a54e5c1 e310cbc 44ad755 3b1a5da bb79112 c78c58a
Author: Martin Myrseth <dave@tentacle.org>
Date:   Fri Jan 20 01:09:57 2023 +0100

    Assemble squid

Use-cases

Honestly, in practice I haven’t found a single valid use-case for octopus merges which aren’t already covered by sequencing a series of merges, one after the other. Perhaps there are some integration use-cases out there which really let’s the octopus merge strategy shine. Let me know!

I should also note that the octopus merge strategy is quite conservative and bluntly refuses to merge anything which doesn’t trivially apply without conflicts. I imaging trying to juggle changes and their origins during a merge resolution to be quite the mess.

One thing I like though about the octopus merge is that it quite visually shows how simple the git graph model really is. It has helped me build intuition about what goes on during a merge operation in git.

The dishonest merge 🤞

While on the topic of merges, I’d like to quickly break down some of the misconception(?) that merge commits are something special in git.

It might be true that there’s some special sauce involving merge-bases and heuristics in order to determine the merge result of joining multiple histories. But once a commit with multiple parents have been made there’s no requirement that whichever tree is associated with a merge commit to make any kind of sense with regards to the merge operation its parent relationship reflects.

Let’s continue from where the octopus merge left off and see that we’ve got all ten *tacles in place:

❯ ls -la
total 52
drwxr-xr-x 3 myme users 4096 Jan 20 01:09 .
drwxr-xr-x 7 myme users 4096 Jan 20 00:42 ..
-rw-r--r-- 1 myme users   32 Jan 20 01:09 eighttacle.txt
-rw-r--r-- 1 myme users   32 Jan 20 01:09 fivetacle.txt
-rw-r--r-- 1 myme users   32 Jan 20 01:09 fourtacle.txt
drwxr-xr-x 9 myme users 4096 Jan 20 01:09 .git
-rw-r--r-- 1 myme users   32 Jan 20 01:09 ninetacle.txt
-rw-r--r-- 1 myme users   32 Jan 20 01:09 onetacle.txt
-rw-r--r-- 1 myme users   32 Jan 20 01:09 seventacle.txt
-rw-r--r-- 1 myme users   32 Jan 20 01:09 sixtacle.txt
-rw-r--r-- 1 myme users   32 Jan 20 01:09 tentacle.txt
-rw-r--r-- 1 myme users   32 Jan 20 01:09 threetacle.txt
-rw-r--r-- 1 myme users   32 Jan 20 01:09 twotacle.txt

There’s nothing stopping us at this point to delete everything introduced by merging all the tentacles and amending the HEAD commit:

❯ git rm *.txt
rm 'eighttacle.txt'
rm 'fivetacle.txt'
rm 'fourtacle.txt'
rm 'ninetacle.txt'
rm 'onetacle.txt'
rm 'seventacle.txt'
rm 'sixtacle.txt'
rm 'tentacle.txt'
rm 'threetacle.txt'
rm 'twotacle.txt'

❯ git commit --amend -C HEAD
[main 8494ef5] Assemble squid
 Date: Fri Jan 20 01:09:57 2023 +0100

All files are gone:

❯ ls -l
total 0

Yet the default view of git show of the merge doesn’t hint at anything suspicious:

commit de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (HEAD -> main)
Merge: 4dadc16 3f7a95e e9cd39a 6be7cf4 a54e5c1 e310cbc 44ad755 3b1a5da bb79112 c78c58a
Author: Martin Myrseth <dave@tentacle.org>
Date:   Fri Jan 20 01:09:57 2023 +0100

    Assemble squid

While asking it to also include the merge commits it’s fairly obvious that somebody have been messing around with the merge resolution:

❯ git show --pretty=oneline -m --stat
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from 4dadc16d89758ed1625223286e1218b63c988313) (HEAD -> main) Assemble squid
 tentacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from 3f7a95ecac18a92451f7e205c8ea0bb2366c2e97) (HEAD -> main) Assemble squid
 ninetacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from e9cd39ad4664b04f29263250396ec1b270e4eeb8) (HEAD -> main) Assemble squid
 eighttacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from 6be7cf4b00f640a32d61a9e205e0b4a1e18b3bb8) (HEAD -> main) Assemble squid
 seventacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from a54e5c16f807a3f9aad8dd0c5187abcc9e6b6c7d) (HEAD -> main) Assemble squid
 sixtacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from e310cbcfecaa3cb6f08084a64c18318f7552a8a7) (HEAD -> main) Assemble squid
 fivetacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from 44ad755cc07047ee3dd25c5170aa9d4dde60475c) (HEAD -> main) Assemble squid
 fourtacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from 3b1a5da6c6e5b2d0b93517dda20c3295ed893374) (HEAD -> main) Assemble squid
 threetacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from bb791123be4bd03a0c6427d1990cd57898dd9793) (HEAD -> main) Assemble squid
 twotacle.txt | 1 -
 1 file changed, 1 deletion(-)
de3e016de71484e62e6ac7e6dda08fe7f9d85af4 (from c78c58a2debbab2d88ed0e747a54f4d750f8378f) (HEAD -> main) Assemble squid
 onetacle.txt | 1 -
 1 file changed, 1 deletion(-)

In the end, a merge commit in git tracks a tree⁷ – like any other commit – and it only extends on the parent commit metadata by including one parent field for all commits that serves as inputs to the merge operation. Furthermore, it places no constraints onto the changes made to the tree associated with that commit. Which basically gives a committer full “artistic freedom” as to what should be the result of a merge, ranging from the trivial “sum of all differences” or minor conflict resolutions to absolutely wild rewrites that had absolutely nothing to do with the differences that went into a merge to begin with.

Rounding off

I’m sure that many of these features of git are by no means news to the readers of this post, and I’m not exactly sure what pushed me towards writing it in the first place. If anything, it’s a recollection of (silly) things I’ve done in the past. Hopefully it could also inspire people to go learn tools that serve as their daily drivers beyond just the basic or core functionality.

I’m a believer that not everything we learn or do has to necessarily have some obvious usefulness in and of itself. Often when learning tools, techniques, programming languages, and everything else in the field of software, I find that going off on tangents can help build intuition about core concepts, ultimately leading to a deeper understanding. Of course, the few times this peripheral knowledge is of actual use in real-life situations it’s even better.

I do place great value in utility, but I also like to remind people to have fun, experiment, and to build simply for the sake of building. Which, while typing out this summary, reminded me of this recent post – “Take your pragmatism for a unicycle ride” – which appeared on my favorite tech aggregator site the other day. A post which also touched on the importance of developer energy. That’s something I consider very central to my own motivation and mental well-being. If there’s fun to be had in learning – or building – we’re much less likely to burn out from it.

Footnotes

I’ve been building this site statically using Hakyll since the beginning and I haven’t found any significant reason to move off it yet. I figured that since there are some topics that keeps repeating I wanted to start tagging my posts. Hakyll supports tags out of the box and with some help from this post it didn’t take me too long to get a decent initial implementation in place.

Adding tags to posts

Hakell allows defining some metadata for each post. In this metadata it’s trivial to add a new tags key, which should define a list of comma-separated tags:

---
title: This is some post
tags: NixOS, Haskell, ...
---

Once all posts have been tagged it’s all about letting Hakyll know what to do with them.

Finding tags

First off Hakyll needs to know from which files to lookup tags (all posts) and where to place the output of each tag list. For this Hakyll provides the buildTags function:

λ: :i buildTags
buildTags ::
  MonadMetadata m => Pattern -> (String -> Identifier) -> m Tags
        -- Defined in ‘Hakyll.Web.Tags’

In my static site generator (ssg) I apply it to the following arguments:

main = do
  -- snip snip
  tags <- buildTags "posts/*" (fromCapture "tags/*.html" . toLower)

Basically this means for all files under ./posts and generate files for each tag under ./tags. A personal preference of mine is to have the file names be lower case, so each tag is converted to lower case for the file name only.

Generating tag post lists

Hakyll can generate one static page for each tag found. The posts are found, filtered based on visibility and sorted before the tags file is rendered with the same template as the “all posts” listing:

tagsRules tags $ \tagStr tagsPattern -> do
  route idRoute
  compile $ do
    posts <- loadAll tagsPattern >>= filterM postIsNotPreview >>= recentFirst
    let postsCtx =
          constField "title" tagStr
            <> listField "posts" postCtx (return posts)
            <> defaultContext

    makeItem ""
      >>= loadAndApplyTemplate "templates/posts.html" postsCtx
      >>= loadAndApplyTemplate "templates/default.html" postsCtx
      >>= relativizeUrls

Navigating to tags list

At this point Hakyll will build tags/*.html pages containing all posts with the given tags. However, there’s no way to find these pages through navigation. In order to make each listing of topics defined by a tag reachable I could choose to have a good old “tag cloud” overview page, using something like renderTagCloud. To start off though I settled on just adding a list of tags to each post, which allows the user to click into similar posts.

Hakyll has various functions for rendering tags into HTML where the simplest is tagsField:

λ: :i tagsField
tagsField :: String -> Tags -> Context a

This function only renders a comma-separated set of <a> anchors, which doesn’t give all the semantics and styling options I would like. Instead, there’s the more general tagsFieldWith which accepts a few additional parameters:

λ: :i tagsFieldWith
tagsFieldWith ::
  (Identifier -> Compiler [String])
  -- ^ Get the tags
  -> (String -> (Maybe FilePath) -> Maybe H.Html)
  -- ^ Render link for one tag
  -> ([H.Html] -> H.Html)
  -- ^ Concatenate tag links
  -> String
  -- ^ Destination field
  -> Tags
  -- ^ Tags structure
  -> Context a
  -- ^ Resulting context

For the most part I’d like the same behavior as tagsField, so I applied the same arguments while providing a separate renderLink function which simply puts the links into <li> as well:

renderLink _ Nothing = Nothing
renderLink tag (Just url) = Just $
  H.li $ H.a ! A.href (H.toValue ("/" <> url)) $ H.toHtml tag

The whole rule ended up like the following:

match "posts/*" $ do
  route $ setExtension "html"
  compile $ do
    let
      renderLink _ Nothing = Nothing
      renderLink tag (Just url) = Just $
        H.li $ H.a ! A.href (H.toValue ("/" <> url)) $ H.toHtml tag
      tagsCtx = tagsFieldWith getTags renderLink mconcat "tags" tags
    postCompiler
      >>= loadAndApplyTemplate "templates/post.html" (tagsCtx <> postCtx)
      >>= saveSnapshot "content"
      >>= loadAndApplyTemplate "templates/default.html" postCtx
      >>= relativizeUrls

Finally, in order to output the list of tags for each post the rendered HTML must be injected into the post template. The tagsCtx places the rendered HTML into a field named tags and so not much more is needed than referencing it in the desired place in the template markup:

<article>
  <h1>$title$</h1>
  <ul class="tags">$tags$</ul>
  <section class="header">
    Posted on $date$
    $if(author)$
      by $author$
    $endif$
  </section>
  <section>
    $body$
  </section>
</article>

And that’s pretty much it! Now the posts have tags and it should be possible to see e.g. my list of Nix and NixOS related posts.

Hardware come and hardware go. About one year ago I moved my Home Manager setup from a Raspberry PI 3 Model B to an Intel NUC Core i3. After that migration my poor Raspberry Pi has been laying on a shelf, collecting dust.

Even a decommissioned Raspberry Pi could have some utility running on the home network. With some time to spare, last weekend seemed like the weekend for building RPi images. To get going quickly it’s possible to download pre-built NixOS SD card images from Hydra. Guaranteed to save me a lot of time, that was naturally not the path I chose. I wanted to build the image and machine configuration myself, because why not¹.

Building

The Raspberry Pi is quite a limited system when it comes to computing resources and you don’t really want to find yourself building much to any software on it natively. Instead it would be great to save some grief by building the configuration on a more powerful machine.

The configuration required to build a NixOS SD card image for the Raspberry Pi turned out to be really short and sweet. Here’s a minimal nixosConfigurations entry where a system configuration includes the necessary module to build SD card ISO images:

{
  description = "NixOS Raspberry Pi configuration flake";
  outputs = { self, nixpkgs }: {
    nixosConfigurations.rpi = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
        { config.system.stateVersion = "22.05"; }
      ];
    };
  };
}

In fact, these days I frequently find things I would assume to be “tricky”, “advanced” or “hacky” on other distros to be really elegant on NixOS. I’ve come to conclude that not only is NixOS awesome because of how its declarative, pure and composable nature result in highly reproducible builds and configurations, but more often than not people have managed to hide complex tasks behind elegant abstractions. I’m sure it’s a combination of the environment enabling and encouraging the smart people working on NixOS to easily build simple interfaces.

And on the flip side of that, I have full sympathy for all beginners to nix, clawing their way up the rocky cliff towards the fabled lighthouse of hope of eternal declarative reproducible software bliss.

The SD image attribute path

The SD card image derivation can be found under the following property in the flake configuration hierarchy:

nixosConfigurations.rpi.config.system.build.sdImage

While creating this configuration is trivial, chances are building it is not as simple as invoking:

nix build .#nixosConfigurations.rpi.config.system.build.sdImage

Most likely - depending on your build machine’s architecture - nix will greet you with an error message similar to the following²:

error: a 'aarch64-linux' with features {} is required to build
    '/nix/store/m9l1cvifh44nl7fscp7mjbl7qv88478g-stage-2-init.sh.drv',
    but I am a 'x86_64-linux' with features {benchmark, big-parallel, kvm, nixos-test}

Cross-compiling

The Raspberry Pi 3 is an ARM chip, requiring binaries targeting aarch64. As with the full SD image NixOS Hydra builds nixpkgs for aarch64-linux (and Apple aarch64-darwin). The binary cache on cache.nixos.org helps greatly speeding up the image generation as most stock derivations would be already built.

I use x86_64 machines exclusively (besides this one R-Pi). This poses the challenge of cross-compilation if I want to build the R-Pi SD card image. There are several options listed on the NixOS wiki’s ARM section for how to “natively” build the images.

The NixOS Wiki has a brief section on how to do native cross compilation setting nixpkgs.crossSystem.system to aarch64-linux. The nixpkgs sources for crossSystem states themselves, this is kind of a nonsensical settings. Why? Because (pure, as in isolated builds) cross-compilation requires at least two (or three) system architectures:

1. The build host’s architecture, on which the toolchain is built and run.
2. The host machine’s architecture, which is the target binary format of the build process and where the binaries will eventually run.
3. The target machine’s architecture, which is more related to toolchain build and compatibility.

Given this, the following snippet from the NixOS Wiki seems to be outdated and didn’t really work all that well:

{ ... }: {
  nixpkgs.crossSystem.system = "aarch64-linux";
}

The resulting toolchain did not seem to provide proper NixOS-style wrapped binaries and several toolchain paths (like typical system includes) and linker paths were broken.

Particularly, with flakes, the build (local) machine architecture shouldn’t be inferred from the system for the sake of purity. Rather, what seemed to have a lot more success in kicking off the (native) cross-compilation build of the SD card image was the following nixpkgs settings:

{ ... }: {
  nixpkgs.buildPlatform.system = "x86_64-linux";
  nixpkgs.hostPlatform.system = "aarch64-linux";
}

This resulted in better cache hits from the NixOS cache and many packages started building, saturating my machine’s CPUs with parallel builds. Sweet!

Unfortunately, while the cross-compilation of the aarch64 Linux kernel purred away through the build, the simultaneous build of samba failed due to missing headers. I did spent a wee bit of time trying to figure out why the #ifdef including crypt.h was set when waf listed it as missing in the configure step, but eventually I didn’t want to spend more time than necessary on this yak shave:

error: builder for '/nix/store/khs4kyxqa7gkd6wvg467d6a5s084zb7p-samba-aarch64-unknown-linux-gnu-4.15.9.drv' failed with exit code 1;
       last 10 log lines:
       > [2095/3718] Compiling libcli/named_pipe_auth/npa_tstream.c
       > In file included from ../../lib/talloc/pytalloc_util.c:25:
       > ../../lib/replace/replace.h:885:10: fatal error: crypt.h: No such file or directory
       >   885 | #include <crypt.h>
       >       |          ^~~~~~~~~
       > compilation terminated.
       >
       > Waf: Leaving directory `/build/samba-4.15.9/bin/default'
       > Build failed
       >  -> task in 'pytalloc-util.cpython-310-aarch64-linux-gnu.objlist' failed with exit status 1 (run with -v to display more information)
       For full logs, run 'nix log /nix/store/khs4kyxqa7gkd6wvg467d6a5s084zb7p-samba-aarch64-unknown-linux-gnu-4.15.9.drv'.

In the end the native cross-compilation was looking promising, but at the end of the day I left it at that.

Registering QEmu emulation as a binfmt wrapper

As an alternative to native cross-compilation it’s also possible to compile for aarch64 registering QEmu emulation as a binfmt wrapper. This can easily be done by adding this single definition to the build machine’s NixOS configuration:

{
  boot.binfmt.emulatedSystems = [ "aarch64-linux" ];
}

After a nixos-rebuild test the local machine should now be able to run binaries built for aarch64 and thus the tools required to build the NixOS configuration for the R-Pi SD images. I’ve tested this from both native NixOS on x86_64 and on NixOS running in WSL. Both worked great.

Now, one of the reasons for investing some effort into native cross-compilation is that emulation is significantly slower than any native counterpart. As part of my configuration I have e.g. some Haskell utlities that require compilation. Haskell isn’t exactly known for its ludicrous compilation speeds and overlooking the emulated GHC grind through each source file was like watching paint dry.

Customization

The image build from the configurations already shown will not contain any customizations. It would be a stock NixOS image, identical to the installers built by Hydra.

Wouldn’t it be great if we could customize the configuration and have that configuration be what we flash to the SD card without having to do any further nixos-rebuild commands after booting the R-Pi?

But of course, we’re talking about NixOS here after all! Here’s an incomplete example configuration illustrating various “configuration points” within the flake structure. The R-Pi image is basically nothing more special than a regular NixOS configuration, and thus it’s possible to pull in any existing NixOS configuration or module into this config:

{
  description = "NixOS Raspberry Pi configuration flake";
  outputs = { self, nixpkgs }: {
    nixosConfigurations.rpi = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        # SD card image
        "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"

        # Additional NixOS modules, like Home Manager or personal modules
        home-manager.nixosModules.home-manager
        ./some/presonal/module.nix

        # Inline configuration here
        ({ ... }: {
          config = {
            # ...like <hostname>
            networking.hostName = "bringebær-pai";
            system.stateVersion = "22.05";
          };
        })
      ];
    };
  };
}

In the case of my dotfiles I do have some abstractions over nixpkgs.lib.nixosSystem which builds configurations with some default modules based on my dotfile repository structure. I gave a walkthrough of this configuration in my previous post: NixOS: Confederation.

C.mpr.ss..n

By default, the SD image ISO is compressed using zstd. This is a reasonable default as it saves a significant amount of disk space:

nix-repl> :lf .
Added 8 variables.

nix-repl> :p nixosConfigurations.rpi.config.system.build.sdImage.compressImage
true

The image can be decompressed using the following command:

$ nix shell nixpkgs#zstd -c unzstd -o nixos.img ./result/sd-image/<name>.img.zst

No compression

With all my customizations and additional tools the image size is considerably bigger than the default. Expect that compressing the resulting .img using zstd takes quite a bit of time. Since I was intending on only building the SD image once, I skipped the zstd by setting config.system.build.sdImage.compressImage = false:

{
  description = "NixOS Raspberry Pi configuration flake";
  outputs = { self, nixpkgs }: {
    nixosConfigurations.rpi = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        "${nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix"
        {
          config = {
            system = {
              # Disable zstd compression
              build.sdImage.compressImage = false;

              # ... other configs

              stateVersion = "22.05";
            };
          };
        }
      ];
    };
  };
}

Surely enough, in the end the SD image was built and ready to be flashed onto a real SD card. Without concern of stripping down the configuration to bare minimums my image containing all my headless customizations and tools take close to 9GB of disk space. A minimal installation ends up somewhere around 2.5GB.

❯ ls -lh result/sd-image/
total 8.7G
-r--r--r-- 1 root root 8.7G Jan  1  1970 nixos-sd-image-22.11.20221031.d40fea9-aarch64-linux.img

Flashing the SD card

After plugging the SD card into the build machine there are several ways to figure out which device it ends up being assigned. Personally I just use dmesg or dmesg --follow for this, which provides some fairly obvious output:

[1514026.420241] mmc0: new ultra high speed SDR50 SDHC card at address e624
[1514026.420557] mmcblk0: mmc0:e624 AALCD 29.7 GiB
[1514026.425039]  mmcblk0: p1 p2

This means that the SD card device to use in the following commands would be /dev/mmcblk0.

One alternative to dmesg is to use lsblk, for which I don’t currently have any example output containing the SD card:

❯ lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1        259:0    0 476.9G  0 disk
├─nvme0n1p1    259:1    0   511M  0 part  /boot
└─nvme0n1p2    259:2    0 476.4G  0 part
  └─root       254:0    0 476.4G  0 crypt
    ├─vg-swap  254:1    0    16G  0 lvm   [SWAP]
    └─vg-nixos 254:2    0 460.4G  0 lvm   /nix/store
                                          /

Having gathered all pieces of information it should now be time to copy over the image to the SD card device. This can be done using the dd command, specifying the SD card image as the if (input file) and the device path as the of (output file).

sudo dd \
    if=result/sd-image/nixos-sd-image-22.11.20221031.d40fea9-aarch64-linux.img \
    of=/dev/mmcblk0 \
    status=progress \
    bs=4M

Booting

Power issues

The NixOS wiki warns about power issues related specifically to the R-Pi 3:

Especially with the power-hungry Raspberry Pi 3, it is important to have a sufficient enough power supply or weirdness may happen.

And surely enough, weirdness ensued.

I don’t quite remember what I used to power the R-Pi previously, but this time around I was using some regular USB adapters I had laying around. The R-Pi seemed to start up exactly as it should and once I had the SD image ready I was presented with the NixOS boot process and all seemed well. Power being an issue wasn’t in my mind at this point.

However, at some point during “Stage 2” of the boot process, everything went black and the R-Pi eventually entered a cyclic boot. This seemed to consistently happen right when the kernel changes or updates the framebuffer resolution and/or fonts (or whatever happens with that regular boot-time “flicker”). For the first couple of reboots I thought I might have mistakenly setup an X server on my image, and now the X server was causing some panic or whatever right when it was starting. Naturally my first though was that I had somehow messed up the image and it was corrupt in one way or another. To eliminate personal fuckups my first attempt at fixing this issue was to double check that one of the pre-built images managed to get me to the login prompt.

No such luck.

The same blackout then reboot happened in exactly the same way with the pre-built SD image. At this point I had spent quite a bit of time on this project and was feeling tired. Building images, flashing SD cards and booting Raspberry Pis has a long feedback loop, particularly with large images and slow SD card bandwidths. Not knowing which wild goose to chase it killed my spirit to push on. Better take a break!

A few days passed, but I eventually felt revigorated and came back with a fresh mind. I started by re-reading the NixOS Wiki on Raspberry Pi, and surely enough I was reminded of the importance of providing the R-Pi with sufficient, stable power.

Switching out the random USB adapter with something capable of consistently serving up to 2A of 5V seemed to get me to the long awaited login prompt on tty1.

Great success! However, logging in I did not see my familiar starship prompt, but rather a more stock bash prompt reminding me that I pulled all my configs from the SD image to rule out any personal settings messing up the boot process.

After another few iterations of rebuilding and reflashing the SD card - eventually growing the image to a (not so modest) 9 GB from the stripped down 2.5 GB of the plain image - I was able to log in to a system that felt like $HOME.

Remote deployment

Building the SD card image with all our customizations and tweaks saved us a ton of time. Once the R-Pi is up running we can’t keep re-flashing the SD card whenever we want to update our software. It’s of course possible copy over the configuration to the R-Pi, ssh in and issue nixos-rebuild switch there, but building my configuration itself the Raspberry Pi wouldn’t have finished by Christmas.

It’s much more desirable to keep using remote building and rather copy over all the store paths required for the new NixOS generation over to the R-Pi once they’re built. There are a few hurdles to overcome first before making that a reality.

Generating nix cache keys

In order for remote systems to accept derivations built on other machines they have to be signed in the nix store of the build machine or through the nix cache server. Generating a key is quite simple, following the steps on Distributed build or Binary Cache over at the NixOS Wiki:

$ nix-store --generate-binary-cache-key builder-name cache-priv-key.pem cache-pub-key.pem

Most people who’ve used nix for any significant time should be familiar with seeing pre-built store paths pulled down from cache.nixos.org. It’s also possible to use other caches (substitutors) like cachix or serving up a nix store directly from the local machine using e.g. nix-serve (or any of its many compatible clones).

As a security mechanism nix doesn’t allow using pre-built store paths from random hosts. To ensure that a path has been built by a trusted remote nix allows signing store paths. By default, nix is configured to trust the key used by Hydra:

❯ grep trusted-public-keys /etc/nix/nix.conf
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=

It’s possible to have nix sign all store paths that are built locally using a given private key. This can be done by adding the private key to /etc/nix/nix.conf:

secret-key-files <path-to-key>.pem

It’s also possible to sign all existing store paths after the fact:

$ nix store sign --all --key-file cache-priv-key.pem

Instead of explicitly signing store paths on the local machine it’s possible (and perhaps preferable) to sign the store paths when they are served through servers like nix-serve. In which case you shouldn’t invoke the command above or add the secret-key-files to /etc/nix/nix.conf.

🚀 deploy-rs

I haven’t been doing much of my own nix caching outside of simple experiments and using the cachix-action in GitHub workflows. The tool I chose to use for this was deploy-rs. It allows checking, building and deploying NixOS configurations to remote systems while also having some nice bells and whistles like automatic rollback on failure.

{
  description = "NixOS Raspberry Pi configuration flake";

  inputs = {
    # Add deploy-rs as Flake input from GitHub
    deploy-rs.url = "github:serokell/deploy-rs";
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.05";
  };

  outputs = { self, deploy-rs, nixpkgs }: {
    nixosConfigurations.rpi = nixpkgs.lib.nixosSystem {
      system = "aarch64-linux";
      modules = [
        # NixOS config
        # ...
      ];
    };

    # deploy-rs node configuration
    deploy.nodes.rpi = {
      hostname = "192.168.1.7";
      profiles.system = {
        sshUser = "myme";
        sshOpts = [ "-t" ];
        magicRollback = false;
        path =
          deploy-rs.lib.aarch64-linux.activate.nixos
            self.nixosConfigurations.rpi;
        user = "root";
      };
    };
  };
}

Once the deploy-rs configuration is in place the time has come for testing our first remote deployment. First we need to tell deploy-rs about our cache key before we can start the actual deploy step. This is just defining a LOCAL_KEY environment variable. Then running deploy with either zero arguments (defaults to all defined nodes) or with a flake path to a specific node:

$ export LOCAL_KEY=/var/nix/cache-priv-key.pem
$ deploy .#rpi

For generations to come!

Once the deploy-rs deployment succeeds it’s possible to list the NixOS generations on the R-Pi to see that we’re indeed able to activate new generations remotely.

Rounding off

So, what did we do?

1. Configured our build host to emulate aarch64 for building binaries for the Raspberry Pi.
2. Added our NixOS customizations to the minimal Raspberry Pi SD card configuration.
3. Built the SD card image and copied it over to a physical card.
4. Booted the Raspberry Pi (with sufficient power) into a batteries included NixOS installation.
5. Set up our build machine with a nix store cache key.
6. Configured deploy-rs with our Raspberry Pi as a node.
7. Successfully deployed NixOS to the Raspberry Pi 🚀

There are probably tons of things I can improve with this process, but at least it’s at the point where this is working smoothly.

Now I just need to find something to use this aging R-Pi for!

Footnotes

In the previous post I took a look back on my dotfiles journey through the last decade or so. That post started as a preface to this one, but grew to the point where it deserved a post of its own.

As I’ve journeyed deeper into the world of NixOS I have realized that there were some major issues with the way I managed machine configurations. What I’m presenting here is the outcome of that cleanup work. Now I have something I’m quite happy with and I rounded off the last post by saying:

I feel it’s a very exciting setup and would like to dive into the details, but those are saved for another post… Stay tuned!

The triggering moment for when I finally decided to refresh my system configurations turned out to be this tweet from Mitchell Hashimoto, which again led me to his YouTube video demonstrating his macOS virtual machine setup. Since I’ve recently been investing more time and effort into flakes and combined with an increasing annoyance of the state of my system configuration it was the perfect timing to start basing configurations fully on nix flakes. To begin with I started copying pretty much the same things Mitchell was doing, but I quickly started digging more into how others were building NixOS from flakes (see “Note:” section below). I have since borrowed whichever tricks and methods I found neat or useful, and so I’ve gathered inspiration from quite a few sources in the end.

The title of the post is obviously a word play on “configuration” and “federation”. I’ve gone through the process of uniting together the various bits and pieces of a previously unstructured configurations repository into something which feels much more consistent, not just as a source tree, but also the machine states it generates. Components can be shared across various deployments and architectures, yet configurable to accommodate for special needs. With higher guarantees for reproducibility the configurations are now also much more robust.

The flake

Let’s gets started by jumping straight in to the action by looking at the flake.nix.

The flake.nix is included as follows in its entirety, it’s not too long:

{
  description = "myme's NixOS configuration with flakes";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    nixos-wsl.url = "github:nix-community/NixOS-WSL";
    flake-utils.url = "github:numtide/flake-utils";
    agenix = {
      url = "github:ryantm/agenix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    home-manager = {
      url = "github:nix-community/home-manager/master";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    doomemacs = {
      url = "github:doomemacs/doomemacs";
      flake = false;
    };
    i3ws.url = "github:myme/i3ws";
    annodate.url = "github:myme/annodate";
    nixon.url = "github:myme/nixon";
    wallpapers = {
      url = "gitlab:myme/wallpapers";
      flake = false;
    };
  };

  outputs = { self, nixpkgs, flake-utils, home-manager, ... }@inputs:
    let
      system = "x86_64-linux";
      overlays = [
        inputs.agenix.overlay
        inputs.i3ws.overlay
        inputs.annodate.overlay
        inputs.nixon.overlay
        self.overlay
      ];
      lib = nixpkgs.lib.extend (final: prev:
        import ./lib {
          inherit home-manager;
          lib = final;
        });
    in {
      overlay = import ./overlay.nix {
        inherit lib home-manager;
        inherit (inputs) doomemacs wallpapers;
      };

      # NixOS machines
      nixosConfigurations = lib.myme.allProfiles ./machines (name: file:
        lib.myme.makeNixOS name file { inherit inputs system overlays; });

      # Non-NixOS machines (Fedora, WSL, ++)
      homeConfigurations = lib.myme.nixos2hm {
        inherit (self) nixosConfigurations;
        inherit overlays system;
      };
    } // flake-utils.lib.eachDefaultSystem (system:
      let pkgs = import nixpkgs { inherit system overlays; };
      in {
        # All packages under pkgs.myme.apps from the overlay
        packages = pkgs.myme.apps;

        devShells = {
          # Default dev shell (used by direnv)
          default = pkgs.mkShell { buildInputs = with pkgs; [ agenix ]; };

          # For hacking on XMonad
          xmonad = pkgs.mkShell {
            buildInputs = with pkgs;
              [ (ghc.withPackages (ps: with ps; [ xmonad xmonad-contrib ])) ];
          };
        };
      });
}

And as seen by nix:

❯ nix flake show .
git+file:///home/myme/code/dotfiles?ref=refs%2fheads%2fmain&rev=37e5dccd614bfb4b6e369697e7c285327ef59668
├───devShell
│   └───x86_64-linux: development environment 'nix-shell'
├───homeConfigurations: unknown
├───nixosConfigurations
│   ├───Tuple: NixOS configuration
│   ├───map: NixOS configuration
│   ├───nuckie: NixOS configuration
│   ├───qemu-server: NixOS configuration
│   ├───qemu-vm: NixOS configuration
│   └───vmware: NixOS configuration
├───overlay: Nixpkgs overlay
└───packages
    └───x86_64-linux
        └───git-sync: package 'git-sync-0d0s33l2..hhjz'

I try to keep the flake.nix “high level” and easy to navigate. So I factor out unnecessary details into helper functions and nix expressions that typically gets placed into ./lib.

Inputs

Everything under inputs are the inputs (aka. dependencies) of the configuration flake. To ensure reproducibility all inputs are locked in the flake.lock file which is managed by nix commands, but is also added as a git file to track its history of changes.

The inputs typically list other nix flakes that have a similar structure to the one from my dotfiles repo. The <input>.url field is resolved using nix registry symbolic identifiers and we can also tell nix that an input is not a flake by using <input>.flake = false. This lets a flake track and pin for instance any git repository, which is very convenient. In my case I use that for tracking Doom Emacs and my wallpaper repo.

Outputs

The outputs (more interestingly that the inputs) list the various derivations, configurations and environments that the flake can generate. In this case it’s the following:

• A nix overlay
• A set of x86-64 packages
• NixOS configurations
• Home Manager user profiles
• A development environment

The outputs section starts with the parameter list, or perhaps more correctly the destructuring of the input attribute set. Flake outputs should define a function which is applied to the attribute set of flake inputs. Only a few of the inputs are bound in the function parameter list since @inputs is used to pass on all the inputs to helper functions later on.

outputs = { self, nixpkgs, flake-utils, home-manager, ... }@inputs:

Nixpkgs and overlays

After the parameters come a few local variables: system, overlays and pkgs.

let
  system = "x86_64-linux";
  overlays = [
    inputs.agenix.overlay
    inputs.i3ws.overlay
    inputs.annodate.overlay
    inputs.nixon.overlay
    self.overlay
  ];
  pkgs = import nixpkgs {
    inherit system overlays;
  };

The system is hard-coded to x86-64¹ and used to instantiate pkgs from the nixpkgs input. In the “pure” world of flakes the system argument to nixpkgs cannot be deferred from the running system and must always be provided. The last one is overlays which is just a list of all the nix overlays in use. Most overlays come from the other input flakes, but it also includes its own overlay from self.overlay:

overlay = import ./overlay.nix {
  inherit home-manager;
  inherit (inputs) doom-emacs wallpapers;
};

The overlay of the flake just adds library functions, apps and packages that are bundled along with the dotfiles. This makes them available to the NixOS configurations and Home Manager profiles as well as possible to export using:

# All packages under pkgs.myme.apps from the overlay
packages = pkgs.myme.apps;

NixOS & Home Manager

I find that the next part of the flake.nix is quite interesting. This is the NixOS configurations and Home Manager user profiles generated from my dotfiles:

# NixOS machines
nixosConfigurations = pkgs.myme.lib.allProfiles ./machines (name: file:
  pkgs.myme.lib.makeNixOS name file {
    inherit inputs system overlays;
  });

# Non-NixOS machines (Fedora, WSL, ++)
homeConfigurations = pkgs.myme.lib.nixos2hm {
  inherit overlays system nixosConfigurations;
};

This makes use of a couple of (home grown) library functions that automatically generate host/machine configurations and associated user profiles from files on disk. Instead of enumerating all machines in the flake.nix there is an allNixFiles function that finds files and directories under ./machines/ and treats them as individual host configurations:

{ lib }:

dir:

let
  isNixFile = { name, type }: type == "directory" || lib.strings.hasSuffix ".nix" name;

in builtins.map (x: x.name) (builtins.filter isNixFile
  (lib.mapAttrsToList (name: type: { inherit name type; })
    (builtins.readDir dir)))

Host names are inferred from the basename of each directory entry under ./machines/:

❯ tree machines/
machines/
├── map.nix
├── nuckie.nix
├── qemu-server.nix
├── qemu-vm.nix
├── Tuple.nix
└── vmware.nix

The nixos2hm function extracts user profiles from all the NixOS machine configurations, exposing the Home Manager profiles for all users on all machines. This is particularly useful for non-NixOS environments so I can reuse “global” NixOS configurations even though I don’t build a complete NixOS system profile. As a more concrete example, I use the machine role configuration to determine whether or not the machine should have graphical tools installed or not. By defining a “mock” NixOS machine for these machines the Home Manager part of the configuration can still depend on the NixOS configurations and make decisions based on the values:

{ home-manager, lib }:

{ overlays, system, nixosConfigurations }:

let
  removeHostname = str: builtins.head (builtins.split "@" str);
  userAtHostConfig = { host, config }: (
    lib.mapAttrsToList
      (username: hmConfig: {
        name = "${username}@${host}";
        value = hmConfig.home;
      })
      config.home-manager.users
  );

in with builtins; (listToAttrs (concatMap userAtHostConfig
  (lib.mapAttrsToList (host: config: {
    inherit host;
    inherit (config) config;
  }) nixosConfigurations)))

Since nix flake show only lists homeConfiguration as “unknown” we can use the nix repl to list all user profiles pulled from the machine configurations:

❯ nix repl
Welcome to Nix 2.9.0pre20220530_af23d38. Type :? for help.

nix-repl> :lf .
Added 13 variables.

nix-repl> builtins.attrNames homeConfigurations
[
  "myme@Tuple"
  "myme@map"
  "myme@nuckie"
  "myme@vmware"
  "nixos@qemu-server"
  "nixos@qemu-vm"
  "nixos@vmware"
  "user@qemu-server"
]

Development shell

Finally there is the devShells. Flakes can define multiple shell development environments. The Haskell toolchain involved with hacking on xmonad is quite heavy, so I keep that in a separate shell environment called xmonad that’s not used by default. In the default shell I currently only expose agenix which I use to manage the small set of secrets (aka. semi-sensitive data).

devShells = {
  # Default dev shell (used by direnv)
  default = pkgs.mkShell { buildInputs = with pkgs; [ agenix ]; };

  # For hacking on XMonad
  xmonad = pkgs.mkShell {
    buildInputs = with pkgs;
      [ (ghc.withPackages (ps: with ps; [ xmonad xmonad-contrib ])) ];
  };
};

Machines

The configurations I push to github.com/myme/dotfiles is of course not the whole truth of the configurations I have for machines. Traditionally I’ve been keeping various machine configurations off in separate host branches. Each branch with a set of tweaks specific for that machines only. These tweaks are mostly mutually exclusive since that’s the whole point of not propagating it back to the shared main branch:

My workflow has been to make changes to a machine and try it out for anything from a couple of seconds to weeks or even months. Changes that are useful for other machines are rebased to the beginning of the “host branch” and the main branch is updated by a simple fast-forward to the rebased commit. All other host branches are then rebased in turn on top of the main branch. Ideally the set of tweaks should be few to minimize general configuration differences between machines as well as reducing the hassle of conflicts while rebasing.

The boilerplate of setting up a configuration for each machine is done with the makeNixOS function:

name: machineFile: { inputs, overlays, system }:

let
  inherit (inputs) self agenix home-manager nixpkgs nixos-wsl;

in nixpkgs.lib.nixosSystem {
  inherit system;
  modules = [
    ../system
    ../users/root.nix
    agenix.nixosModule
    nixos-wsl.nixosModules.wsl
    home-manager.nixosModules.home-manager
    machineFile
    {
      # Hostname
      networking.hostName = name;

      # Let 'nixos-version --json' know about the Git revision
      # of this flake.
      system.configurationRevision = nixpkgs.lib.mkIf (self ? rev) self.rev;

      # Nix + nixpkgs
      nix.registry.nixpkgs.flake = nixpkgs;  # Pin flake nixpkgs
      nixpkgs.overlays = overlays;
    }
  ];
}

It includes several upstream NixOS modules that are pulled in as flake inputs: nixos-wsl, home-manger and agenix. The ./system directory contains most of the defaults for a machine as well as custom NixOS configurations that mostly serve as high-level feature management for each machine:

options.myme.machine = {
  name = lib.mkOption {
    type = lib.types.str;
    default = "nixos";
    description = "Machine name";
  };
  role = lib.mkOption {
    type = lib.types.enum [ "desktop" "laptop" "server" ];
    default = "desktop";
    description = "Machine type";
  };
  flavor = lib.mkOption {
    type = lib.types.enum [ "nixos" "generic" "wsl" ];
    default = "nixos";
    description = "Linux flavor";
  };
};

The name is obviously used to give the machine a hostname, and so on.

The role determines if it’s a headless server machine, a laptop that requires e.g. battery and power management, or a desktop computer.

The flavor is used to specify the Linux flavor of the installation. Is this a NixOS machine, a “generic” Linux like Ubuntu or Arch, or WSL.

Generic config

Each machine configuration follow a pretty similar setup. The following is the configuration for map, a WSL installation running on a Microsoft Surface tablet:

#
# `map` is a Windows 11 machine and this configuration is for WSL on that host.
#
# Graphical apps are supported, but unfortunately not GL, see:
#
#   https://github.com/guibou/nixGL/issues/69
#

{ config, pkgs, ... }: {
  myme.machine = {
    role = "desktop";
    flavor = "wsl";
    highDPI = false;
    user = {
      name = "myme";

      # This maps to the `users.users.myme` NixOS config
      config = {
        isNormalUser = true;
        initialPassword = "nixos";
        extraGroups = [ "wheel" "networkmanager" ];
        openssh.authorizedKeys.keys = [];
      };

      # This maps to the `home-manager.users.myme` NixOS (HM module) config
      profile = {
        imports = [
          ../home-manager
        ];

        config = {
          home.packages = with pkgs; [
            mosh
          ];

          programs = {
            # SSH agent
            keychain = {
              enable = true;
              keys = [ "id_ed25519" ];
            };

            ssh = {
              enable = true;
              includes = [
                config.age.secrets.ssh.path
              ];
            };
          };

          myme.dev.haskell = {
            enable = true;
            lsp = false;
          };
        };
      };
    };
  };

  age.secrets.ssh = {
    file = ./../secrets/ssh.age;
    owner = config.myme.machine.user.name;
  };
}

Most of the configuration goes into the myme.machine configuration. This is because it also contains the home-manager configuration for each machine under myme.machine.user.profile which is mapped directly to the home-manager configuration options.

On hardware

I have a couple of machines running regular NixOS on hardware. My main work computer is a Lenovo P1 laptop and the configuration for it is not currently public. However, one useful part of its configuration is the use of nixos-hardware which is added as a flake input and provides useful configurations for the Lenovo P1’s quirks:

imports = [
  inputs.nixos-hardware.nixosModules.common-gpu-nvidia
  inputs.nixos-hardware.nixosModules.lenovo-thinkpad-p1
  inputs.nixos-hardware.nixosModules.lenovo-thinkpad-p1-gen3
  ./hardware.nix
];

…and don’t get me started on the nvidia graphics of that machine:

# NVidia - 😭
hardware.nvidia = {
  package = config.boot.kernelPackages.nvidiaPackages.beta;
  modesetting.enable = true;
  powerManagement = {
    enable = true;
    finegrained = true;
  };
};
services.xserver.displayManager.sessionCommands = ''
  ${lib.getBin pkgs.xorg.xrandr}/bin/xrandr --setprovideroutputsource NVIDIA-G0 modesetting
'';

So... if you're not particularly careful, you might screw up and order a dual graphics laptop (with nvidia). Next thing you know you're knee deep in X config/driver hell 😭#fmlhttps://t.co/tD0bxniErY

— Martin Myrseth (@ubermyme) March 24, 2022

The X server still craps itself when adding/removing external displays, so it’s not a very desirable setup for hot-desking at the moment. Besides the regrets of picking out a nvidia-based laptop, I’m quite happy with the power and screen of the P1 in general though.

I also have an Intel NUC that I use for home automation which runs NixOS directly on hardware. It’s not a graphical installation and there’s not really much more interesting to say about it.

VMWare

My daily driver in the days of home office is an AMD Ryzen desktop computer running Windows on the metal. It’s the same computer I use for home studio music production and most of the relevant music editing software I use is mostly macos and Windows only. I expect there would be quite a bit of suffering jumping onto Linux-based music production, and so I haven’t justified spending time on it.

Since the machine is powerful enough for my needs I don’t have any trouble doing most of my work in VMWare virtual machine and have been doing so for several years. There’s not much interesting to say regarding the VMWare-specifics in my NixOS setup, besides perhaps the guest tools:

# VM
virtualisation.vmware.guest.enable = true;

WSL

I’ve been making more and more use of Windows Subsystem for Linux v2 over the past months. Thanks to the excellent NixOS-WSL project I’m now able to run a pretty much complete NixOS installation on my Windows machines.

My configuration enables the NixOS-WSL configurations if the machine has a Linux flavor of wsl:

(lib.mkIf (config.myme.machine.flavor == "wsl") {
  wsl = {
    enable = true;
    automountPath = "/mnt";
    defaultUser = config.myme.machine.user.name;
  };
})

For WSL I also don’t include the boot and some networking parts of the configuration:

# Disable boot + networking for WSL
(lib.mkIf (config.myme.machine.flavor != "wsl") {
  # Boot
  boot.loader.systemd-boot.enable = true;
  boot.loader.systemd-boot.configurationLimit = 30;
  boot.loader.efi.canTouchEfiVariables = true;
  boot.kernelPackages = pkgs.linuxPackages_latest;

  # Network
  networking.networkmanager.enable = true;
  networking.firewall.enable = true;
})

QEmu

Even with the atomic rollbacks that NixOS provides it can be convenient to occasionally try out experimental configurations in a controlled environment such as a virtual machine. I’m already relying on VMWare or VirtualBox for some hosts and could of course use those virtualizers to test out configuration.

However, NixOS provides a very convenient sub-command to build QEmu virtual machines through nixos-rebuild build-vm. Through clever mounts of the host’s nix store the guest gets access to a read-only version of it. Also with the performance KVM provides it’s a very quick and lightweight way to spin up a configuration:

# QEmu
#
# Full graphical NixOS setup on QEmu.
#

{ config, lib, pkgs, ... }: {
  myme.machine = {
    role = "desktop";
    flavor = "nixos";
    user = {
      name = "nixos";

      # This maps to the `users.users.nixos` NixOS config
      config = {
        isNormalUser = true;
        initialPassword = "nixos";
        extraGroups = [ "wheel" ];
      };

      # This maps to the `home-manager.users.nixos` NixOS (HM module) config
      profile = {
        imports = [
          ../home-manager
        ];

        config = {
          myme.wm = {
            enable = true;
            variant = "xmonad";
            conky = false;
            polybar.monitor = "Virtual-1";
          };
        };
      };
    };
  };

  # Security
  security.sudo.wheelNeedsPassword = false;
}

From my dotfiles repo such a configuration can be built and run using:

nixos-rebuild build-vm --flake .#qemu-vm
./result/bin/run-qemu-vm-vm

Bringing NixOS to life inside QEmu running in NixOS in WSL (yo dawg, I heard you like virtualization):

Raspberry PI

I have a Raspberry PI that I’ve momentarily relieved from service as its purpose being a home automation driver has been replaced by a NUC. I intend to bootstrap the PI with NixOS at some point, but it’s not really on the top of my personal backlog. It would be a fun exercise though as it would allow me to test some other architectures for NixOS.

Should I end up doing this I’ll try to make sure I’ll write about it and add a link to it here.

Configuration

The Home Manager configuration entry point is home-manager/default.nix and should be familiar to those who’ve already used Home Manager:

{ lib, pkgs, ... }: {
  imports = [
    ./barrier.nix
    ./dev.nix
    ./emacs
    ./git.nix
    ./irc.nix
    ./nixon.nix
    ./spotify.nix
    ./tmux.nix
    ./vim.nix
    ./wm
  ];

  config = {
    # ...
  };
}

I’m slowly but surely trying to move to a structure where the ./home-manager directory and configurations are the same for all machines, but controlled through custom high-level configurations. This in an attempt to try to minimize the differences between machines. The alternative would be to include Home Manager configuration modules into each machine configuration, which does lead to more boilerplate and repetition.

I don’t really want to go into deeper details regarding how I structure my Home Manager stuff, because I don’t really think it’s unique in any way. Going in-depth sounds like a topic for a dedicated future post.

Secrets

I honestly don’t have many things I consider secrets in my configurations at the moment. However, since I’m trying to move more and more common configurations into my main dotfiles branch I do want to hide some configurations, like ~/.ssh/config hosts and whatnot.

I’ve found agenix to be simple enough for my needs so far. For my use-case I use ssh key infrastructure to encrypt secrets to each machine’s host key for sshd.

agenix needs a (by default) ./secrets.nix file containing the keys associated with each encrypted secrets file:

let
  hostKeys = {
    map =
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILaNxtQ37YaiXRXx+Ff3sPEbzsjA2i934r0Bl+eXVh3P root@map";
    tuple =
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMhSCm/KiFfhkTcLaza/GFrpPVEzIFhALxM6gBmNK3Gi root@Tuple";
  };

  userKeys = {
    map =
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII1Qsv8MA+cyu7n+4H1kpbVrAmOosJJxjPWAdl08YDvL myme@map";
    tuple =
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH+9tnNlMesGrK/lDvycgzyS4pPrsGqcGQP6yLCsr/LN myme@Tuple";
  };

in {
  # Files to manage - used by the agenix cli to encrypt/decrypt
  "./secrets/ssh.age".publicKeys =
    [ hostKeys.map hostKeys.tuple userKeys.map userKeys.tuple ];
}

The agenix command line tool can the be used to edit the secrets file (it’s available through the default devShell, remember?):

agenix -e ./secrets/ssh.age

For each machine the secrets file must have a corresponding entry in the NixOS configuration:

age.secrets.ssh = {
  file = ./../secrets/ssh.age;
  owner = config.myme.machine.user.name;
};

The secrets can then be included from the ~/.ssh/config by referencing e.g. the path attribute, like so:

myme.machine.user.profile.config.programs = {
  ssh = {
    enable = true;
    includes = [
      config.age.secrets.ssh.path
    ];
  };
};

The ssh.includes attribute then resolves to the following line in the ~/.ssh/config file:

Include /run/agenix/ssh

agenix will then ensure that files are decrypted using each machine’s sshd host key and the content made available through files under /run/agenix.

In the repl

One thing that I find quite useful (and also quite amazing) is using the nix repl to browse through nixpkgs and flakes. This is a great way to explore the properties of derivations and other nix expressions. I can’t get over how awesome it is to do exactly that with your NixOS configuration as well.

When loading the dotfiles flake into the nix repl I can browse the various nixosConfiguration and homeConfigurations as well as build individual apps and programs. This is great to explore a system configuration to learn or validate that it’s configured correctly. And it’s not restricted to the configuration for the current system, but all systems specified in the current flake.

As an example, I was debugging why the X server didn’t want to start properly in the qemu-vm example configuration. I realized I had to have a look at some of the generated files for the user’s home directory. Instead of tweaking configurations, rebuilding and booting the VM, I could simply load up the repl and build the user’s home-files, a part of the NixOS configuration provided by the Home Manager’s NixOS module. The built derivation contains all the dotfiles that will be symlinked into the user’s home directory, which can be easily inspected:

I should note that you don’t need the repl to build sub-parts of a NixOS like this. The tab completion and interactive navigation of the repl makes exploration a lot more seamless.

Controlled updates

One of the most empowering benefits of automatic version locking/pinning is how easy it makes working with unstable software “channels”². Personally the main reason it’s a hassle to base machine configurations on unstable upstreams is when things break exactly because of upstream updates. It can be immensely frustrating when anything from a simple configuration to the whole system stops working because of an uncontrolled software update.

Having controlled and atomic rollbacks fixes part of this problem as it gets you back to a working state and NixOS have been having this for a long time. Having complete control of when to upgrade fixes most of the remaining issues. An unstable upstream is only as unstable as the new features that come in, so by ensuring that upstream updates don’t automatically trickle down to your system the system state for a locked version should remain the same. For example if your system is working well under a specific version of nixos-unstable it is stable for you and should remain so until you decide to pull in new changes.

Whereas nix channels traditionally have been updated globally either on the system or user level, flake inputs are locked to specific versions for each individual project. Firstly, this means that no global command affect how individual nix flake projects manage their dependencies. Furthermore, a flake can easily track unstable upstreams as it simply locks down that upstream to a specific version ignoring new changes to the upstream and only on the user’s request will it update that lock information. This happens primarily due to two things:

1. The user changes a flake input url.
2. The user runs nix flake lock --update-input <input>.

Updating inputs

$ nix flake lock --update-input nixpkgs

And similarly for a non-flake input:

$ nix flake lock --update-input doomemacs

Remote updates

I mentioned I have both a NUC and a RaspberryPi, which aren’t the most powerful computers. I have experimented a bit with using my desktop computer to build the NixOS configuration for the NUC. It worked well enough once I started getting grips on the binary cache signature checks of nix. I did end up with some unusable derivations that were synced over and that I weren’t able to properly remove. One of the key parts (no pun intended) was to set the binaryCachePublicKeys to include that of the host that built the configuration:

nix.binaryCachePublicKeys = [
  "tuple:RLwVT0X7XUres7PkgkMLgsMfWhbHP0PYIfQmqJ2M6Ac="
];

I definitely see a lot of potential of doing remote builds using nix. Tools like NixOps have been around for a while, but I have yet to really give it a spin. Alternatively, there are other options like serokell/deploy-rs which seem to integrate even nicer with flakes.

Bootstrapping

The majority of time spent managing NixOS running on general purpose machines is done through changing configurations and then invoking nixos-rebuild to apply it. This, of course, depends on the system already running NixOS.

I had some big plans making sure that bootstrapping new machines with NixOS would be a breeze with proper automation. Due to a couple of reasons I haven’t been able to deliver on this promise to myself yet.

In the interim I’ve first of all not really (re-)installed machines all that much - and for the few times it’s happened I’ve used a plain NixOS installation medium and pretty much installed machines the way I’ve always done.

Then, once a machine is running NixOS I copy the flake template from whichever machine is the most similar and tweak it to my specific needs.

I have done some experimentation with scripting the installation process and managed to get it working quite well for any machine running a live-ISO version of NixOS. It requires a running system with an open SSH port. Then the bootstrap/copy.sh script uses rsync to copy over the configuration.

The scripts can be found under ./bootstrap in the dotfiles repo, but aren’t really very configurable making them neither very impressive nor useful.

There are obviously things to improve when it comes to the bootstrapping of new machines. It’s somewhat less rewarding work because there’s often quite a long time between each time I actually need to setup a new machine from scratch. However, with the structure of the dotfiles settling down for now I might find some motivation to eventually improve that too. Perhaps it could be fun to combine it with something like serokell/deploy-rs. I don’t know…

Rounding off

I’m very happy I started on this next chapter of the never-ending journey of configuration management. I had to make quite a few changes and updates to have things fit into this next-generation solution. Regardless of this, most of the actual configurations and setup have been preserved from the previous non-flakes NixOS setup and non-NixOS, standalone Home Manager approach. I believe it’s good to be making these kind of incremental changes, not worrying about getting everything “right” or “perfect” the first time around, but rather have most things working along the way.

As always, I don’t expect I’ll ever be done, but another configuration milestone reached. Fun was had, and so it was time to share my experience of migrating to a fully flakes based NixOS configuration across all machines.

Let me know if you also decide to take the plunge!

Thanks to @evenbrenden for proof-reading.

Footnotes

Much has changed since I made a brief post about my dotfiles and which tools I use, but my fondness for managed dotfiles has not changed. Being able to track the changes I make to setup on one machine greatly reduces the complexity of setting up new ones or apply the changes to the other machines that I use.

This post started as a sub-section of a more elaborate post describing to how to manage full system configurations using nix flakes. However, as I continued writing I realized that what was intended as a slight digressions grew out of proportions and would probably be better as an individual post entirely.

I respect that not everyone cares about how I managed dotfiles in the past, but I’m sure most people have a turbulent relationship with dotfiles in one way or another so the following post might be interesting to you. In any case, I write this mostly for myself to remember.

Primordial soup, aka rotfiles

In my early days of Linux I’m fairly certain I didn’t have any management of dotfiles at all. The first memory I have of any kind of “dotfile curation” was a .vimrc copied between myself and a few friends. I definitely did not use any form of version control. No application to manage or install configurations. No consideration regarding which version or even which applications were installed on a machine nor any means of upgrading existing applications and configurations.

Configuration files were left to rot or forgotten on decommissioned machines and through the means of “customization” Darwinism only the most fit¹ tweaks would stand a chance of surviving. While without sensible tracking and diffing, useful changes to configuration files on one system would easily be forgotten and eventually lost.

Missing links: git init $HOME

One of the first attempts I remember of keeping my dotfiles on a leash was to turn my user home directory into a git repository. By this I mean to simply git init inside of $HOME. At the time it must have felt like the greatest idea ever, but it quickly turns out there are a bunch of issues with this approach. Most of which I no longer remember, but I imagine just the fact that any git related tool (like a bash prompt status) used under $HOME will always detect that it’s running in some git worktree would suck quite bad. Similarly, without smart use of e.g. excludes there will always be a bunch of untracked files in the worktree.

It must have felt promising then to stumble over the Ask HN: What do you use to manage dotfiles? post over at the orange site. The top-rated comment details how to rather use a bare git repository somewhere under $HOME and defining a shell alias setting the --git-dir and --work-tree to point to the bare repo and $HOME respectively:

git init --bare $HOME/.myconf
alias config='/usr/bin/git --git-dir=$HOME/.myconf/ --work-tree=$HOME'
config config status.showUntrackedFiles no

The date of the Hacker News thread (February 10th, 2016) suspiciously coincides with the first commit of my dotfiles repo that I still have to this date²:

commit 61a3f80babec8c1339391462590dafe7ff30fe7f
Author: Martin Myrseth <mm@myme.no>
Date:   Wed Feb 10 11:59:23 2016 +0100

    Inital import of tuple

From the .zshrc of that commit:

alias conf="GIT_DIR=~/.dotfiles GIT_WORK_TREE=~ git"

I don’t think I was very impressed by this workflow because less than a year later I seem to have made the decision to switch setup again (without a mention as to why):

commit 6b0faf6ced6b20669fb3bab8b68617e94ea3ffb9
Author: Martin Myrseth <mm@myme.no>
Date:   Wed Dec 21 10:53:02 2016 +0100

    Switch to a GNU Stow based setup

Link farming: The (GNU) stown age

As the commit message above hints at I made the decision to stop having git mess about with files in my user home and use a more specialized tool the job. GNU stow is a general purpose “link farm” manager which merges and replicates distinct directory structures in a single location using symbolic links. In fact it’s actually quite similar to how complete nix profiles are built from merging together the build outputs of many smaller nix packages using symbolic links.

This is quite useful for dotfiles management as the symbolic links allow for all the configurations to be stored in a git repository in an arbitrary location on disk, but then installed into $HOME. At the same time the symbolic links ensure that changes made to files under $HOME are reflected back into the repository.

One thing that I think makes the Stow approach superior to the bare git repo approach is that Stow can manage sub-parts of the configurations independently. It’s relatively easy to exclude or remove unwanted sections of the configurations simply by not installing them or using Stow’s remove action. git on the other hand would see such a removal as a file deletion forcing either keeping a dirty local configuration state or having to create dedicated branches for host configurations³.

One of the downsides of Stow is that it forces the directory structure in the source location to match how it’s supposed to end up in the destination location. This can put unwanted restrictions on the structure of a configuration repository. Additionally, since Stow only manages files and directory structures it does not provide much help in building and managing applications (this is a major selling-point for e.g. Home Manager, as we’ll get to).

In order to ensure that the applications (and plugins) that I use were available I resorted to a combination of git submodules:

❯ git submodule
-21063bcd924bd8efb65eb36b2fe12ffd6ed1b6a5 bash/.bash/bash-git-prompt
+c7753adbb301dcb647dc96d182c28b228551890e emacs/.emacs.d (v2.0-14235-gc7753adbb)
-f0fe79dd3bb4b782ad6040c970b4bfc818729f05 fzf/apps/fzf
-d049fdfeef422933912c66245a50904ee98f86d0 haskell/apps/haskell-ide-engine
-2f9947b7b966a0da31528f987bee3bf274c4ae82 i3/apps/i3-gnome
-909900a553443beb75ee47f7354da26b43a2c1b6 shell/apps/gogh
 26d9ace1b47f4591b2afdf333442a498311b6ace tmux/.tmux/plugins/tpm (v3.0.0-45-g26d9ace)

and a Makefile:

APT := sudo apt

emacs:
 $(APT) install build-essential mu4e isync
 $(APT) build-dep emacs25
 (cd ~/apps/emacs && ./autogen.sh && ./configure && make -j)
 ./install emacs

# https://fontawesome.com/v4.7.0
fonts:
 wget https://fontawesome.com/v4.7.0/assets/font-awesome-4.7.0.zip -P /tmp
 xdg-open /tmp/font-awesome-4.7.0.zip

i3:
 $(APT) install i3 i3blocks compton rofi gnome-flashback gnome-power-manager gnome-screensaver feh session-shortcuts
 sudo make -C i3/apps/i3-gnome install

haskell:
 $(APT) install haskell-stack
 stack upgrade
 stack install hindent hlint

python:
 $(APT) install virtualenv virtualenvwrapper

term:
 wget -O gogh https://git.io/vQgMr && chmod +x gogh && ./gogh && rm gogh

.PHONY: all emacs fonts haskell i3 python term

I shudder looking back at this “crap”, but to be honest things could probably have been much worse. At least with a Makefile there is a place listing which packages and processes likely were run on a machine. I say “likely” because there’s no guarantee all commends were ever run.

Of course there are a number of obvious flaws here:

Building emacs from source this way requires system-wide installation of emacs’s build dependencies. There were a bunch of auxiliary applications to ensure i3 ran the way I wanted, kept “miles” away from the i3 configuration. Installing Haskell and Python tools system-wide with apt, while having a much more manually download process for fonts and theme manager applications. Perhaps most of all there seems to be very little cohesion between applications and their associated configurations, as well as no consistent tracking of installed content. Cleanup must have been a pain (or I never did).

The Age of Enlightenment: NixOS

Fate eventually lead me to install NixOS on my first couple of machines back in 2019. And for a while I stuck with my stow based dotfile setup.

There’s really not a whole lot of configuration necessary to have a fully functional NixOS installation. The defaults are often enough to get a machine running, with the exception of some hardware or network configuration. In fact, I found the NixOS text-based installation flow is so surprisingly simple I fear people run a risk of gaining unfounded⁴ confidence early on in their first NixOS encounter.

According to the NixOS options search NixOS has thousands of options. Not only has NixOS the appeal of reproducibility and declarative configuration, but every part of a system configuration can be tweaked and molded into whichever shape.

However, I suspect for the majority of people who venture into the world of NixOS and who do not have a pre-existing experience with nix start off by only managing system configurations declaratively using /etc/nixos/configuration.nix. There’s no obvious way for newcomers to include their user profiles and everything else under $HOME in the global system NixOS configuration.

Even when using NixOS it’s easy to miss out of the full advantage of using nix through and through. I assume many newbies (like myself at that point) start by managing $USER apps imperatively using nix-env. Then eventually thinking that this is weirdly inconsistent with the declarative nixos-rebuild workflow. I turned to the Declarative Package Management section of the nixpkgs manual:

# Put something like this into ~/.config/nixpkgs/config.nix
{
  packageOverrides = pkgs: with pkgs; {
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      paths = [
        aspell
        bc
        coreutils
        gdb
        ffmpeg
        nixUnstable
        emscripten
        jq
        nox
        silver-searcher
      ];
    };
  };
}

This would allow installing everything using nix-env -iA nixpkgs.myPackages, which surely felt a whole lot better. It’s a nice complement to the stow based approach providing a means of installing applications. This means I couldn’t yet purge Stow from my setup as it was responsible for providing the configs of all the apps I might choose to install. Perhaps worse is the fact that there are scary impurities of using packageOverrides (and global overlays) that I won’t elaborate on now.

The Industrial Age: Home manager

It’s inevitable for anybody who survive long enough in the NixOS world to encounter Home Manager. The README.md states:

This project provides a basic system for managing a user environment using the Nix package manager together with the Nix libraries found in Nixpkgs. It allows declarative configuration of user specific (non global) packages and dotfiles.

Home Manager reuses the same concepts and conventions found in the NixOS configurations to create a way to manage user profiles and dotfiles in pretty much the same declarative way. Not only does Home Manager provide a framework for building such configurations, but also curates and exposes modules for a multitude of options for many well-known applications and services.

One major shift of switching over to Home Manager is that not only does it generate configuration files, it also ensures all the required software for the given configuration is built. This is huge! With every approach I’ve used up to this point executables had to be installed separately from configurations. This could easily lead to inconsistencies between application versions and their respective configurations. Home Manager ensures that applications and configurations stay in sync through community effort.

I was skeptical at first to what I felt was handing over control of all configurations to pre-written “templates”, so I didn’t immediately jump on the band wagon. However, I eventually made the switch and quickly realized which superior solution to dotfile management it is. Furthermore, not only did it make my NixOS machines configuration better, it also works well on non-NixOS machines greatly simplifying the configuration management across all the machines I use.

commit 2ed03a3abd85fa35a2f9ab47ded79d1dce9b3826
Author: Martin Øinæs Myrseth <myrseth@gmail.com>
Date:   Fri Dec 6 15:32:20 2019 +0100

    nix: Add home.nix

    Initial commit of Home Manager

At this point I had a fairly automated and reproducible setup. My NixOS configuration was managing all system setup, then separately invoking Home Manager to generate my $USER configuration. Life was good, and has been good for a long time. In fact so good that I didn’t really see the flaw with this setup:

Hey yo! I know you like your config and all, but you know, it’s still not like managed as a single declarative unit.

Aw, poop… You’re right.

Re-provisioning a new or existing machine to match a certain configuration still requires a combination of commands for updates in addition to some first-time initial setup of e.g. Home Manager. Without up-to-date documentation this is bound to be non-reproducible and can easily lead to some pain.

The Next Ice Snowflake Age

What we want is a machine configuration where pretty much everything is declaratively defined in a single nix configuration hierarchy. Home Manager has a NixOS module, which allows it to be integrated into the build of NixOS generations. This removes the disconnect between system and user settings.

One remaining issue is that typically such a setup relies on nix channels to determine the specific version of nixpkgs being used to build the current NixOS generation. Channels are designed to be moving targets and which means that a configuration that built or ran successfully at some point can break in the future if the channel has been updated.

This non-determinism is one of the problems that flakes tries to address. For the last year or so I’ve increasingly adopted flakes into my workflow and have really begun to enjoy the benefits they bring. Locking down software project inputs is done by default and with less ceremony than earlier, which ensures I won’t forget to pin manually.

So…

By combining the Home Manager module for NixOS and integrating everything into a flake I’ve eventually reached the point where pretty much everything required to get my machines up and running is handled through a single build step. Not only that, by using git to track configuration history and flakes pinning dependencies I can be much more certain that configurations that have worked in the past will continue to work in the future. This is great for rolling back experimental updates. Furthermore, with an automatically pinned configuration there’s much less hassle to follow the rolling release of NixOS unstable. Should stuff break, it’s just a matter of rolling back and sitting quietly for a little while.

I feel it’s a very exciting setup and would like to dive into the details, but those are saved for another post… Stay tuned!

Footnotes

This post is another in an unstructured series of nix and NixOS related material. It hopefully provides a few new takes on what was already covered by For developers and Containers a la Nix. It is a case-study of a recent “full stack”¹ polyglot pet project of mine combining tools and languages that I enjoy: nix, Haskell and TypeScript. It’s a good candidate to serve as an example application of nix flakes and how they can be used to combine different language environments in a monorepo, while at the same time having a codebase small and unimpressive enough to not get in the way.

What got me motivated into writing this post is that once all the pieces came into place the resulting development workflow turned out to be quite amazing. Of course it might not be perfect or even suited to everyone else out there, but at the very least I hope it serves as inspiration to finding your own perfect setup. Also I should note that flakes are quite new to me and so I wouldn’t claim to be following best practice. I’d love to hear about improvements, misconceptions and remarks regarding the presented content.

The application is an ad-hoc copy/paste app allowing to share a piece of text or a secret between two computers using a Web Browser. To avoid having the shared data pass through the backend server it’s using WebRTC to establish a peer-to-peer data channel connection. In order for WebRTC to work a signaling server is required to help the peers connect to each other. This signaling server is implemented in Haskell.

The application is live at cp.myme.no/ and the code is hosted on github.com/myme/xchg (commit from the time of writing this article).

The elevator pitch

For developers did an attempt at selling the purpose of pursuing nix for software projects, but it doesn’t hurt to reiterate:

Arguably the sole purpose of using computers and building software for them is to automate tasks for humans. Computers tirelessly execute automated work to simplify and augment² anything from the lives of individuals to world-spanning business endeavors.

Computers don’t discriminate the task that they are given, yet it’s common for us humans to discriminate the tasks we want the computers to solve. It’s not uncommon to see effort directed primarily towards meeting metrics and measurables that directly affect the revenue bottom line, customer satisfaction or whatever good user-facing feature seemed important at the time. While it’s hard to disagree with such a sentiment (not making money is a sure way to kill any project), what’s hard to measure on the bottom line also becomes what’s hard to justify investing time in.

Software does not write itself, and computers do not write (amazing) software³. Humans write software – and I’m sorry to say we’re often not all that great at it. We need computers to help us maintain quality in the work that we do. Computers help us scale – because humans scale poorly. The output we produce is limited to the time we get to spend uninterrupted at our keyboards, while our minds cannot easily be shared with our peers who require assistance.

I believe the efforts we make to automate the setup and execution of deterministic development and build environments significantly improve the overall quality and velocity of software projects. A reproducible environment scales to any number of developer machines, continuous integration servers or production nodes. Although there are many great tools that do this in various ways no tool that I know has done it to the extent and with the level of success that nix has.

Hey, wait, you said this was an elevator pitch. Enough talking!

Right. Let’s measure success in lines of bash. Starting with launching a development server from scratch by running a single command:

❯ nix run

Preparing a deterministic editing environment for <insert-favorite-editor-here> with access to compilers, LSP servers, static analysis tools and whatever else you desire:

❯ nix develop

Building a complete application container image without worrying about sudo docker commands, pinning apt sources in a Dockerfile or having an otherwise inconsistent environment during the image creation:

❯ nix build

Coming from a npm, cargo or poetry-based project this might not be all that impressive to you. But contrary to language-specific package managers the nix commands above all ensure that every single required dependency is fetched, built⁴ and available before launching the respective executables and shells. Equally important, build recipes are composed and reused ensuring there’s no difference between the way something’s built for development as for production.

Nix Flakes

If you’ve been worried about “pinning” nixpkgs or using tools like niv to take back control of you nix inputs and so on then flakes might be for you. The seminal article that most seem to point to as an introduction to flakes is Eelco Dolstra’s Nix Flakes, Part 1: An introduction and tutorial. In my opinion it does give a digestible introduction to why flakes were needed and how to get started, so it’s definitely worth the read.

Do not expect this article to give a thorough introduction to the what’s and why’s of flakes. I’ve only recently started to use them more extensively myself and there are most likely more in-depth and precise writeups out there. Another caveat is the fact that the commands and APIs are still marked experimental and may experience some changes in the future.

Flakes are (still) an upcoming feature of the nix package manager. It was only in the fairly recent nix 2.4 release that it was made a generally available feature, yet behind an experimental toggle. In order to enable both the nix command⁵ and flakes support add the following to ~/.config/nix/nix.conf (or /etc/nix/nix.conf):

experimental-features = nix-command flakes

or for NixOS under nix.extraOptions in the system’s /etc/nixos/configuration.nix:

nix = {
  extraOptions = ''experimental-features = nix-command flakes'';

  # The following previously got a pre-release of 2.4 and may no longer be necessary.
  # If the system is stuck on an older then channel then this might still be useful.
  package = pkgs.nixFlakes;
};

If you just want to try it out it’s also possible to simply pass the following command line arguments to the nix commands intended to be used with a flake:

❯ nix --experimental-features "nix-command flakes" <...>

Dependency pinning

Traditional nix commands typically would map a reference to <nixpkgs> to a channel or pinned version of the github.com/NixOS/nixpkgs repository. This puts the responsibility directly on the user of nix to ensure a package stays fully reproducible. Outside factors like a rouge NIX_PATH environment variable can have devastating effects on the build result of a nix expression.

Flakes make a few changes to this model where first of all builds are even more hermetic than previously. Fixed output derivation fetchers like fetchGit and fetchTarball has allowed sources to be fetched from various locations, but require hashes to be considered pure. Manually calculating and updating revisions and hashes is tedious work. Impure inputs are disabled by default and flakes ensure inputs are consistent by introducing a perhaps familiar concept: lock files, named flake.lock. These lock files are only updated by running package update commands like nix flake update.

Here’s an example based off the project’s flake.lock:

{
  "nodes": {
    "nixpkgs": {
      "locked": {
        "lastModified": 1628427351,
        "narHash": "sha256-WuZUIQ07AvRw+T9wvQ3qFf8MXmKZ+ktZz9drNgWXDbs=",
        "path": "/nix/store/aqinic6h77nrsrzwdsq2mxihw0kd87ml-source",
        "rev": "348bc5de8bca09c624f5c4975f538684da4713d2",
        "type": "path"
      },
      "original": {
        "id": "nixpkgs",
        "type": "indirect"
      }
    },
    "root": {
      "inputs": {
        "nixpkgs": "nixpkgs"
      }
    }
  },
  "root": "root",
  "version": 7
}

Anatomy of a flake

The flake schema over on the NixOS wiki lists all of the attributes supported by flakes in more detail. Following is a quick run through.

Top-level attributes

Here’s an extract showing the 3 top-level attributes description, inputs and outputs:

{
  description = "xchg - Exchange stuff between things using WebRTC";

  inputs.nixpkgs.url = "nixpkgs";

  outputs = { self, nixpkgs }: {
    # ...
  };
}

The description should be self-explanatory. The inputs are either absolute URIs to flakes or symbolic identifies, again managed by the nix registry. The inputs are what nix will lock in the flake.lock file. Finally the outputs describe the various flake outputs, which aren’t necessarily just applications or libraries, but also development environments and NixOS modules.

Output schema:

Flake outputs describe all the possible outputs, results or build targets if you will, of a flake. One important thing to note is that each output attribute is actually an attribute set over the systems that the flake supports, like x86_64-linux, aarch64-linux, i686-linux, x86_64-darwin.

Here’s one version of the output part of the project’s flake.nix with some annotations added:

{
  # overlay are exposed in dependent derivations through callPackage
  # NB: Some care needs to be taken no to introduce infinite recursions.
  overlay = (final: prev: {
    xchg = (final.callPackage ./. { } // {
        server = final.callPackage ./server { };
        frontend = final.callPackage ./frontend { };
    });
  });

  # executed by `nix run`
  apps = forAllSystems (system: { dev = nixpkgsFor.${system}.xchg.dev; });

  # executed by `nix build .#name`
  packages = forAllSystems (system:
    let pkgs = nixpkgsFor.${system};
    in {
      image = pkgs.xchg.image;
      server = pkgs.xchg.server.server;
      static = pkgs.xchg.frontend.static;
    });

  # executed by `nix buil` (no name, aka default package)
  defaultPackage = forAllSystems (system: self.packages.${system}.image);

  # executed by `nix develop` (also used by nix-direnv's "use flake")
  devShell = forAllSystems (system:
    mergeEnvs nixpkgsFor.${system}
    (with self.devShells.${system}; [ frontend server ]));

  # executed by `nix develop .#name` (development shells)
  devShells = forAllSystems (system:
    let
      pkgs = nixpkgsFor.${system};
      haskellPackages = pkgs.haskellPackages;
    in {
      frontend = pkgs.xchg.frontend.shell;
      server = pkgs.xchg.server.shell;
    });
}

Home-grown utilities

The nix expressions above do get a bit convoluted due to a couple of functions called forAllSystems and nixpkgsFor. The reasoning behind these utilities is to generate nixpkgs and outputs for all valid systems. Using helper functions like these avoids some repetitive code where attribute sets for each supported system would have to be defined for each output. Here are their definitions:

let
  supportedSystems = [ "x86_64-linux" ];
  forAllSystems = f: nixpkgs.lib.genAttrs supportedSystems (system: f system);
  nixpkgsFor = forAllSystems (system:
    import nixpkgs {
      inherit system;
      overlays = [ self.overlay ];
    });

in {
  # ...
}

1. forAllSystems

   Building an attribute set by mapping a function over all the supported systems. Typically this is useful to create derivations for all supported systems by reusing a single derivation definition.

2. nixpkgsFor

   Instantiates the nixpkgs package set for all the supported systems as well as including the flake’s own overlay which contains the derivations being built.

Flake utils

There are tools for these kind of things if we don’t want to write these small utilities ourselves. Particularly eachDefaultSystem from github.com/numtide/flake-utils helps to reduce the complexity of a flake by allowing to specify an outputs set which is shared for all supported systems. To make use of it flake-utils can be added just as another input to the flake. It would turn the expression into something like the following, producing the exact same outputs as before while looking a bit cleaner:

{
  inputs.nixpkgs.url = "nixpkgs";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  # Note how `flake-utils` becomes an input to the outputs
  outputs = { self, nixpkgs, flake-utils }:
    # This creates equal attribute sets for each supported system
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ overlay ];
        };

        overlay = (final: prev: {
          xchg = (final.callPackage ./. { } // {
            server = final.callPackage ./server { };
            frontend = final.callPackage ./frontend { };
          });
        });

      # Instead of "rec" (which is susceptible to infinite recursion),
      # referring to sibling outputs can be done using `self.${system}.<output>`.
      in rec {
        inherit overlay;
        apps = { dev = pkgs.xchg.dev; };
        packages = {
          image = pkgs.xchg.image;
          server = pkgs.xchg.server.server;
          static = pkgs.xchg.frontend.static;
        };
        defaultPackage = packages.image;
        checks = packages;
        devShell = mergeEnvs pkgs (with devShells; [ frontend server ]);
        devShells = {
          frontend = pkgs.xchg.frontend.shell;
          server = pkgs.xchg.server.shell;
        };
      });
}

I’m not sure if it’s idiomatic, but it seems to make sense to me that flake.nix should primarily define the purpose of components without having too much additional nix “logic”. It’s a declarative definition of the various outputs that a package may have. Then by using regular default.nix files written in a callPackage design pattern users are still granted control over exactly how the derivations will be used. This makes the flake more of a progressive enhancement over “regular” nix and does not get in the way of backwards compatibility.

For instance, a non-flake user may use the default nixpkgs to build the server application and its shell using the following command line invocation:

❯ nix-build --expr '(with import <nixpkgs> {}; callPackage ./server {})'

Overlay

One thing that can be tricky to grok the first times around with nix are overlays. Specifically an overlay is a function applied to two different “views” of a base package set:

1. First the final set often called “self” is the resulting set of applying the overlay itself 🤯.
2. Secondly the previous set often called “super” is the set before applying the overlay.

In the flake presented above the overlay is a core piece defining a nested attribute set of all app derivations:

overlay = (final: prev: {
  xchg = (final.callPackage ./. { } // {
    server = final.callPackage ./server { };
    frontend = final.callPackage ./frontend { };
  });
});

This resulting structure looks something like the following simplified attribute set:

{
  xchg = {
    # Docker image
    image = ...;

    # Development server script
    dev = ...;

    frontend = {
      # Node.js + dependencies
      nodeDependencies = ...;

      # Fully built frontend as static files
      static = ...;

      # Development shell for the frontend
      shell = ...;
    };

    server = {
      # Fully built backend server
      server = ...;

      # Development shell for the server
      shell = ...;
    };
  };
}

Due to the laziness of nix the xchg packages are able to reference each other through the final package set. For instance since each of the sub-derivations are invoked using final.callPackage it allows the top level image target to reference both the frontend and server derivations and combine them into a resulting docker image.

File structure

The composability of nix is great. It’s not uncommon to see projects with several smaller .nix files rather than a few bigger ones. This make reuse and overriding easier. Here’s how the various .nix files are placed in the project and set up to match the overlay structure explained above:

❯ tree -P '*.nix' --prune
.
├── default.nix
├── flake.nix
├── frontend
│   ├── default.nix
│   └── nix
│       ├── default.nix
│       ├── node-env.nix
│       └── node-packages.nix
└── server
    └── default.nix

We’ve already seen the flake.nix file. On the top level there’s also a default.nix which contains the top level expressions for the app’s container image and development server. The default.nix files in frontend/ and server/ specify the build recipes for the frontend and backend server respectively, as well as the development shells for each of them.

The files under frontend/nix are managed by node2nix, a generator generating nix expressions from package.json and package-lock.json files. The resulting files should not be edited manually. The files are added to git even though the files are derived by node2nix. Similar to package-lock.json this helps validating updates when they are regenerated.

Tracking the node2nix files does have a humorous effect on the language distribution of the project:

Application components

Let’s step away from nix for a short while and look closer at which technologies are used to build and drive the application itself.

Frontend

The app is not intended to appear overly impressive. It’s a simple form-based UI that allows the user to input and existing “Share Id” or start a new “Share”:

An initiating client creates a new share which generates a Share Id which may be used by a second client to connect to the session. When the second client connects the signaling server lets the two clients know about each other and relays ICE information.

Once the two peers are connected the UI switches to a very simple form with a text input that lets users share plain text or hidden text. The main purpose of this app is to simplify copy/paste of e.g. a SSH public key or a WiFi passphrase to a fresh computer.

Dependencies

It’s a long time since I attempted to stay up to date with the wild pace of the JavaScript ecosystem. I’m sure React is by no means the trendiest framework for building Web UIs in 2022, but combined with great tools like vite and TypeScript it still feels fresh and ergonomic to me. There is also a great deal of comfort in the maturity of these tools. As a fan of functional programming I have yet to make more extensive use of elm, although I have been using PureScript a bit previously.

It’s still very convenient to use npm to manage a JavaScript app’s dependencies, even in a “nix first” project. When adding new dependencies, upgrading them or running npm audit to audit security issues it’s nice to be able to this with the npm commands. Here’s the package.json describing the development and runtime dependencies as well as the scripts to be used with npm run:

{
  "name": "xchg",
  "version": "0.0.0",
  "scripts": {
    "build": "tsc && vite build",
    "preview": "vite preview",
    "start": "vite"
  },
  "devDependencies": {
    "@types/node": "^17.0.8",
    "@types/react": "^17.0.38",
    "@types/react-dom": "^17.0.11",
    "@types/uuid": "^8.3.4",
    "@vitejs/plugin-react": "^1.1.3",
    "typescript": "^4.4.4",
    "vite": "^2.7.2"
  },
  "dependencies": {
    "react": "^17.0.2",
    "react-dom": "^17.0.2",
    "react-router-dom": "^6.2.1",
    "uuid": "^8.3.2"
  }
}

Production build

For the production build vite build is used which handles all bundling of dependencies and optimization. The result is a static build of the site:

❯ npm run build

> xchg@0.0.0 build /home/myme/src/xchg/frontend
> tsc && vite build

vite v2.7.7 building for production...
✓ 63 modules transformed.
dist/assets/favicon.7383218d.svg   1.82 KiB
dist/index.html                    0.54 KiB
dist/assets/index.cd59808c.css     1.99 KiB / gzip: 0.84 KiB
dist/assets/index.abe69b62.js      15.49 KiB / gzip: 5.45 KiB
dist/assets/vendor.f7ee7bb4.js     141.70 KiB / gzip: 46.21 KiB

Fast feedback

Regardless if I’m working in the frontend or backend, in a dynamic or static typed languages one thing is certain: I feel severely limited if I don’t have some kind of a near instant feedback loop. Typically this would come from a type-checker or static analysis tool like a linter which integrates with the Language Server Protocol (LSP) client of emacs, my editor. For more dynamic languages having keybindings to launch a test suite might provide a sufficiently short feedback loop.

In many ways I’m actually quite fond of JavaScript. It’s expressive, flexible and actually quite functional in nature. However, similarly to Python without type annotations I quickly get to a point where I start to lose track of data structures and function signatures. This is why I’m quick to introduce TypeScript into my JavaScript based projects. The main reason why I typically lean towards languages with stronger types is more about being “lazy” than anything else. I want the computer to take care of checking the things I don’t want to waste mental energy worrying about myself, or write tests for to simply validate that two pieces of code fit nicely together.

Stronger types also grant the computer more insight into the code. A type checker can provide richer feedback, autocompletions make more sense, and so on.

Fast feedback is not just something that happens within the editor. For this project I’m using vite, which provides builtin support for both automatic reload with page refresh and Hot Module Replacement (HRM) for both JavaScript sources and CSS. What HRM means is that whenever a change to code or styles are saved, the development server pushes this change to the client without having the browser reload the page:

Backend

The backend server is written in Haskell using warp: a fast, light-weight web server for WAI applications. It’s harder to show a screenshot of a backend, so here’s the middleware stack of the application to show off something:

main :: IO ()
main =
  Warp.run port $
    wsMiddleware getNextId state $
      routeMiddleware $
        staticMiddleware
          staticRoot
          fallbackApp

There’s a bit more to it of course, but digging into application details isn’t the point of this article. What is interesting though is that it’s a Haskell application and it has some dependencies.

Dependencies

Before getting into nix my goto tool for Haskell projects was stack. It’s a great tool, but since switching to nix simply using Cabal based projects seem sufficient. Since the main nixpkgs tool for building Cabal projects support hpack I kind of like using it for its brevity:

name: xchg-server
version: '0.1.0.0'
dependencies:
  - base == 4.*
  - aeson
  - http-types
  - optparse-applicative
  - random
  - text
  - wai
  - wai-extra
  - wai-middleware-static
  - wai-websockets
  - warp
  - websockets