nix-git/README.md

# nix-git

This repository contains my NixOS configuration files with some guidance on how to use them.

See also:
- [TODO.md](./TODO.md)
- [NixOS.md](./NixOS.md)

## Table of Contents

<!-- TOC -->
* [nix-git](#nix-git)
  * [Table of Contents](#table-of-contents)
  * [NixOS installation](#nixos-installation)
    * [Graphical installation](#graphical-installation)
    * [Remote installation: disko and nixos-anywhere](#remote-installation-disko-and-nixos-anywhere)
    * [ARM: SD image for the Raspberry Pi 3B+](#arm-sd-image-for-the-raspberry-pi-3b)
      * [ISO creation](#iso-creation)
      * [Default configuration](#default-configuration)
      * [Apply modified configuration](#apply-modified-configuration)
    * [ARM: Vanilla UEFI bootloader for the Raspberry Pi 3B+](#arm-vanilla-uefi-bootloader-for-the-raspberry-pi-3b)
  * [Update, build and switch](#update-build-and-switch)
    * [tldr](#tldr)
    * [Update](#update)
    * [Build and switch: Using Colmena](#build-and-switch-using-colmena)
    * [Build and switch: Manually](#build-and-switch-manually)
      * [Option I: Build new config and activate it:](#option-i-build-new-config-and-activate-it)
      * [Option II: Build new config and activate it during next boot:](#option-ii-build-new-config-and-activate-it-during-next-boot)
      * [Option III: Build config and view changes:](#option-iii-build-config-and-view-changes)
  * [Colmena: Deployment and secret management](#colmena-deployment-and-secret-management)
  * [BTRFS swap file](#btrfs-swap-file)
  * [BTRFS CoW](#btrfs-cow)
  * [Automount encrypted drive](#automount-encrypted-drive)
  * [List active and failed services](#list-active-and-failed-services)
  * [View system log](#view-system-log)
  * [Server maintenance with a smartphone](#server-maintenance-with-a-smartphone)
<!-- TOC -->

## NixOS installation

### Graphical installation

For beginners, NixOS can be installed with a graphical installer.

Getting the ISO:

* There is no official torrent as they are not needed due to CDN.
* ISO and checksum are available here: https://nixos.org/download#nixos-iso
* There are unofficial torrents. If the checksum is compared with the one from the official website, these can be used as well: https://github.com/AnimMouse/NixOS-ISO-Torrents/releases

During installation:

* If the installation target is a SSD
  * Trim the whole disk to mark all cells as unused and restore its initial performance
  * `sudo blkdiscard -f /dev/nvmeXXX`
* Select manual partitioning:
  * One 512MB (or larger) Fat32 partition, mounted at `/boot`, "boot" flag enabled
  * Another partition (e.g. BTRFS) covering the rest of the drive, mounted at `/`, encryption enabled

### Remote installation: disko and nixos-anywhere

Install NixOS via SSH everywhere.

There is a separate repository for these steps. Its README can be found here: https://codeberg.org/privacy1st/nixos-anywhere-example/src/template/README.md

### ARM: SD image for the Raspberry Pi 3B+

#### ISO creation

If you are on an architecture other than aarch64, enable emulation: `boot.binfmt.emulatedSystems = [ "aarch64-linux" ];`.

An ISO for the Raspberry Pi 3B+ can then be built with:

```shell
# If on aarch64
#nix-build '<nixpkgs/nixos>' -A config.system.build.sdImage -I nixos-config=./iso-aarch64.nix
# If not on aarch64
nix-build '<nixpkgs/nixos>' -A config.system.build.sdImage -I nixos-config=./iso-aarch64.nix --argstr system aarch64-linux

ls result/sd-image/*.img
```

**Note** about cross compilation

Alternatively to emulating the aarch64 architecture we could also cross compile from e.g. x86 to it. However, this has one big drawback: The binary cache (https://cache.nixos.org/) won't be used. The reason for this is that packages built with cross compilation are (slightly) different from native built ones. Their checksums don't match.

#### Default configuration

When the Raspberry Pi is booted, run `nixos-generate-config` to generate the default `configuration.nix` and `hardware-configuration.nix` files.

A slightly modified config can be found in [./hosts/pi3bplus](./hosts/pi3bplus)

#### Apply modified configuration

The Raspberry Pi 3B+ has only 1GB RAM, which is not enough for `nixos-rebuild`. It is recommended to create and activate a SWAP file first: https://wiki.archlinux.org/title/swap#Swap_file_creation

```shell
nix-channel --list
#=> nixos https://nixos.org/channels/nixos-23.05
nix-channel --update nixos
```

```shell
passwd
```

```shell
nixos-rebuild boot
reboot
```

### ARM: Vanilla UEFI bootloader for the Raspberry Pi 3B+

**TODO**: The EFI menu was working. But I got many squashfs errors when booting an USB stick with the NixOS installer.

* https://www.eisfunke.com/posts/2023/uefi-boot-on-raspberry-pi-3.html

Preparing a micro SD card with an UEFI bootloader:

```shell
nix-shell -p parted gptfdisk
```
```shell
sudo parted --script /dev/SDX -- \
	mklabel gpt \
	mkpart ESP fat32 2Mib 512MiB \
	set 1 esp on \
	mkpart primary 512MiB 100% \
	print

sudo wipefs --all /dev/SDX1
sudo wipefs --all /dev/SDX2

sudo mkfs.fat -F32 /dev/SDX1

sudo gdisk /dev/SDX
# r
# p
# h
# 1
# N
# 0c
# N
# N
# o
# w
# Y
```

In addition to the SD card, attach an USB stick with NixOS ARM (https://nixos.org/download.html#nixos-iso, https://hydra.nixos.org/job/nixos/release-23.05/nixos.iso_minimal.aarch64-linux, https://hydra.nixos.org/job/nixos/release-23.05/nixos.iso_minimal_new_kernel_no_zfs.aarch64-linux or https://hydra.nixos.org/job/nixos/trunk-combined/nixos.iso_minimal.aarch64-linux) to the Raspberry Pi.

```shell
sudo umount /dev/SDX
sudo dd if=nixos-minimal-XXX-aarch64-linux.iso of=/dev/SDX bs=4M conv=fsync
```

Then install NixOS the way you prefer, but don't touch/modify the partition table!

See https://nixos.wiki/wiki/NixOS_on_ARM/UEFI#Installing

## Update, build and switch

### tldr

Local node (e.g. yodaTux):

```shell
sudo nix-channel --update && niv update
colmena build --on $(hostname) -v --show-trace && colmena apply-local --sudo boot
```

Single remote node (e.g. deploy to yodaTab):

```shell
niv update
colmena build --on remoteTab -v --show-trace && colmena apply --on remoteTab boot
colmena exec --on remoteTab "nix-channel --update"
```

Multiple remote nodes (e.g. deploy to all nodes with tag @server):

```shell
niv update
colmena build --on @server -v --show-trace && colmena apply --on @server boot
colmena exec --on @server "nix-channel --update"
```

### Update

* Updating NixOS. https://superuser.com/a/1604695

Update channel and configuration:

```shell
sudo nix-channel --update && niv update
```

To apply the updates, continue with "Build and switch".

### Build and switch: Using Colmena

See section "Colmena: Deployment and secret management" to build and apply updates.

### Build and switch: Manually

* https://nixos.wiki/wiki/Nixos-rebuild
* https://discourse.nixos.org/t/how-to-get-this-pending-updates-notification-in-gnome/16344/3
* https://discourse.nixos.org/t/how-to-get-this-pending-updates-notification-in-gnome/16344/6

#### Option I: Build new config and activate it:

```shell
sudo nixos-rebuild -I nixos-config=hosts/$(hostname)/configuration.nix switch
```

To view changes, see section "Compare two versions of NixOS system profile".

#### Option II: Build new config and activate it during next boot:

```shell
sudo nixos-rebuild -I nixos-config=hosts/$(hostname)/configuration.nix boot
```

#### Option III: Build config and view changes:

```shell
# This leaves a symlink named `result` in the current directory.
sudo nixos-rebuild -I nixos-config=hosts/$(hostname)/configuration.nix build
nix --extra-experimental-features nix-command store diff-closures /run/current-system ./result
#=> ...
#=> linux: 6.1.47, 6.1.47-modules → 6.1.51, 6.1.51-modules, -11.8 KiB
```

Depending on if there are large kernel changes, either switch to it directly or wait until next boot:

```shell
sudo ./result/bin/switch-to-configuration switch
# or
sudo ./result/bin/switch-to-configuration boot
```

Note: The Perl script executed by both above commands _should_ create a new bootloader menu entry and mark it as default. However, if I remember correctly, the menu entry was once missing and the configuration change thus not permanent. This might need further testing. See also: https://nixos.wiki/wiki/Nixos-rebuild#Internals

## Colmena: Deployment and secret management

* https://github.com/zhaofengli/colmena#colmena

> Colmena is a simple, stateless NixOS deployment tool modeled after NixOps and morph, written in Rust.

Alternative: Deployment with Morph: https://xeiaso.net/blog/morph-setup-2021-04-25

Configuration is done inside [hive.nix](hive.nix).

Build config:

```shell
colmena build -v
```

Apply to all non-local nodes:

```shell
colmena apply --on @server switch
#colmena apply --on @server boot

#colmena apply --on remoteTab switch
#colmena apply --on remoteTab boot
```

Apply to local node:

```shell
colmena apply-local --sudo switch
#colmena apply-local --sudo boot
```

Filtering:
- You can filter hosts by tags with `--on @tag-a,@tag-b`.
- You can use globs in tag matching as well: `colmena apply --on '@infra-*'`

## BTRFS swap file

* Detailed instructions: https://nixos.wiki/wiki/Btrfs#Swap_file
* https://wiki.archlinux.org/title/btrfs#Swap_file

Summary:

* Create subvolume `@swap` directly below top-level subvolume.
* Mount at `/swap`
* Create swapfile: `sudo btrfs filesystem mkswapfile --size 8g --uuid clear /swap/swapfile`
* Regenerate hardware-configuration: `sudo nixos-generate-config --dir hosts/$(hostname)`
* Add `swapDevices = [ { device = "/swap/swapfile"; } ];` to hardware configuration and run `nixos-rebuild switch` (see above).

## BTRFS CoW

* https://wiki.archlinux.org/title/btrfs#Disabling_CoW

The Arch wiki has the following warning about disabling CoW:

> Disabling CoW in Btrfs also disables checksums. Btrfs will not be able to detect corrupted nodatacow files. When combined with RAID 1, power outages or other sources of corruption can cause the data to become out of sync.

Thus, we e.g. don't disable CoW for increased DB performance.

When systemd initially creates its journal file, it warns us about enabled CoW:

> Creating journal file /var/log/journal/.../system.journal on a btrfs file system, and copy-on-write is enabled. This is likely to slow down journal access substantially, please consider turning off the copy-on-write file attribute on the journal directory, using chattr +C.

As corruption of log files is not dramatic, we disable CoW for `/var/log` with

```shell
sudo chattr +C /var/log
```

Note from Arch wiki:
> If the 'C' flag is set on a directory, it will have no effect on the directory, but new files created in that directory will have the No_COW attribute.

## Automount encrypted drive

* Use the same password for all attached LUKS devices to be only prompted once while booting
  * Discussion: https://discourse.nixos.org/t/how-to-unlock-some-luks-devices-with-a-keyfile-on-a-first-luks-device/18949/11
  * Related NixOS config option: https://github.com/NixOS/nixpkgs/blob/ed2ccd4d1748e52d5d28c440d5be4b25a4f21c08/nixos/modules/system/boot/luksroot.nix#L570-L584
* luksOpen and mount drive, e.g. to `/mnt/data1`
* Re-generate hardware configuration:

```shell
sudo nixos-generate-config --dir hosts/$(hostname)
```

* If it is an SSD, enable `boot.initrd.luks.devices.<name>.allowDiscards`

## List active and failed services

If there are e.g. file conflicts due to preexisting dotfiles, the Home Manager user service might fail.
To list all failed services, run:

```shell
systemctl --failed
```

List all active services:

```shell
systemctl list-units --type=service --state=running
systemctl --user list-units --type=service --state=running
```

## View system log

```bash
# Kernel message log with timestamps
dmesg --ctime
# System journal: Errors since last boot.
journalctl -p err -b
# GNOME session logs since last boot. https://help.gnome.org/admin/system-admin-guide/stable/session-debug.html.en
journalctl _UID="${UID}" -b
```

## Server maintenance with a smartphone

While I was on a bike tour I did some config changes to update my NixOS servers using only my smartphone and Nitrokey (GPG smartcard):

- `home-manager.users.yoda.programs.git.signing.signByDefault = false;`
- Import `../../modules/git.nix` in `hosts/yodaNas/configuration.nix`
- Import `./../modules/docker-pushrm.nix` in `hosts/yodaNas/configuration.nix`
- In `hieve.nix`, set `allowLocalDeployment` to `true` for `yodaNas` and `yodaHedgehog`

This allowed me to connect with my servers via `ssh`, `git clone` this NixOS configuration, build and switch to it with `colmena` and `git commit` and `git push` any changes.

On my smartphone I was using a terminal emulator with `ssh` and GPG smartcard support.

Another approach would have been to run `colmena` directly on the phone.