nix-git/README.md

# NixOS

- [NixOS](#nixos)
  - [NixOS installation](#nixos-installation)
    - [Graphical installation](#graphical-installation)
  - [Remote installation: disko and nixos-anywhere](#remote-installation-disko-and-nixos-anywhere)
  - [ARM device: SD image for the Raspberry Pi 3B+](#arm-device-sd-image-for-the-raspberry-pi-3b)
    - [ISO](#iso)
    - [Default configuration](#default-configuration)
    - [Apply modified configuration](#apply-modified-configuration)
  - [ARM device: Vanilla UEFI boot loader for the Raspberry Pi 3B+](#arm-device-vanilla-uefi-boot-loader-for-the-raspberry-pi-3b)
  - [Update, build and switch](#update-build-and-switch)
    - [Update](#update)
    - [Build and switch: Using Colmena](#build-and-switch-using-colmena)
    - [Build and switch: Manually](#build-and-switch-manually)
      - [Option I: Build new config and activate it:](#option-i-build-new-config-and-activate-it)
      - [Option II: Build new config and activate it during next boot:](#option-ii-build-new-config-and-activate-it-during-next-boot)
      - [Option III: Build config and view changes:](#option-iii-build-config-and-view-changes)
  - [niv: Dependency management](#niv-dependency-management)
    - [Add Home Manager with niv](#add-home-manager-with-niv)
    - [Add NUR with niv](#add-nur-with-niv)
  - [Colmena: Deployment and secret management](#colmena-deployment-and-secret-management)
  - [BTRFS swap file](#btrfs-swap-file)
  - [LUKS Parameters](#luks-parameters)
  - [Automount encrypted drive](#automount-encrypted-drive)
  - [Failed services](#failed-services)
  - [Garbage collection](#garbage-collection)
  - [Run AppImages](#run-appimages)
  - [General Notes](#general-notes)
    - [Nix Pills](#nix-pills)
    - [Papers](#papers)
    - [System information](#system-information)
    - [Search for packages](#search-for-packages)
    - [Search for options](#search-for-options)
    - [Search wich package owns a file](#search-wich-package-owns-a-file)
    - [List files of package](#list-files-of-package)
    - [Compare two versions of NixOS system profile](#compare-two-versions-of-nixos-system-profile)
    - [NixOS configuration debugging](#nixos-configuration-debugging)
    - [Show Nix configuration](#show-nix-configuration)
    - [Evaluate NixOS configuration to JSON](#evaluate-nixos-configuration-to-json)
    - [References](#references)
  - [TODOs](#todos)


## NixOS installation

### Graphical installation

For beginners, NixOS can be installed with a graphical installer.

Getting the ISO:

* There is no official torrent as they are not needed due to CDN.
* ISO and checksum are available here: https://nixos.org/download#nixos-iso
* There are unofficial torrents. If the checksum is compared with the one from the official website, these can be used as well: https://github.com/AnimMouse/NixOS-ISO-Torrents/releases

During installation:

* If the installation target is a SSD
  * Trim the whole disk to mark all cells as unused and restore its initial performance
  * `sudo blkdiscard -f /dev/nvmeXXX`
* Select manual partitioning:
  * One 512MB (or larger) Fat32 partition, mounted at `/boot`, "boot" flag enabled
  * Another partition (e.g. BTRFS) covering the rest of the drive, mounted at `/`, encryption enabled

## Remote installation: disko and nixos-anywhere

Install NixOS via SSH everywhere.

There is a separate repository for these steps. Its README can be found here: https://codeberg.org/privacy1st/nixos-anywhere-example/src/template/README.md

## ARM device: SD image for the Raspberry Pi 3B+

### ISO

If you are on an architecture other than aarch64, enable emulation: `boot.binfmt.emulatedSystems = [ "aarch64-linux" ];`.

An ISO for the Raspberry Pi 3B+ can then be built with:

```shell
# If on aarch64
#nix-build '<nixpkgs/nixos>' -A config.system.build.sdImage -I nixos-config=./iso-aarch64.nix
# If not on aarch64
nix-build '<nixpkgs/nixos>' -A config.system.build.sdImage -I nixos-config=./iso-aarch64.nix --argstr system aarch64-linux

ls result/sd-image/*.img
```

**Note** about cross compilation

Alternatively to emulating the aarch64 architecture we could also cross compile from e.g. x86 to it. However, this has one big drawback: The binary cache (https://cache.nixos.org/) won't be used. The reason for this is that packages built with cross compilation are (slightly) different from native built ones. Their checksums don't match.

### Default configuration

When the Raspberry Pi is booted, run `nixos-generate-config` to generate the default `configuration.nix` and `hardware-configuration.nix` files.

A slightly modified config can be found in [./hosts/pi3bplus/](./hosts/pi3bplus/)

### Apply modified configuration

The Rapberry Pi 3B+ has only 1GB RAM, which is not enough for `nixos-rebuild`. It is recommended to create and activate a SWAP file first: https://wiki.archlinux.org/title/swap#Swap_file_creation

```shell
nix-channel --list
#=> nixos https://nixos.org/channels/nixos-23.05
nix-channel --update nixos
```

```shell
passwd
```

```shell
nixos-rebuild boot
reboot
```

## ARM device: Vanilla UEFI boot loader for the Raspberry Pi 3B+

**TODO**: The EFI menu was working. But I got many squashfs errors when booting an USB stick with the NixOS installer.

* https://www.eisfunke.com/posts/2023/uefi-boot-on-raspberry-pi-3.html

Preparing a micro SD card with an UEFI boot loader:

```shell
nix-shell -p parted gptfdisk
```
```shell
sudo parted --script /dev/SDX -- \
	mklabel gpt \
	mkpart ESP fat32 2Mib 512MiB \
	set 1 esp on \
	mkpart primary 512MiB 100% \
	print

sudo wipefs --all /dev/SDX1
sudo wipefs --all /dev/SDX2

sudo mkfs.fat -F32 /dev/SDX1

sudo gdisk /dev/SDX
# r
# p
# h
# 1
# N
# 0c
# N
# N
# o
# w
# Y
```

In addition to the SD card, attach an USB stick with NixOS ARM (https://nixos.org/download.html#nixos-iso, https://hydra.nixos.org/job/nixos/release-23.05/nixos.iso_minimal.aarch64-linux, https://hydra.nixos.org/job/nixos/release-23.05/nixos.iso_minimal_new_kernel_no_zfs.aarch64-linux or https://hydra.nixos.org/job/nixos/trunk-combined/nixos.iso_minimal.aarch64-linux) to the Raspberry Pi.

```shell
sudo umount /dev/SDX
sudo dd if=nixos-minimal-XXX-aarch64-linux.iso of=/dev/SDX bs=4M conv=fsync
```

Then install NixOS the way you prefer, but don't touch/modify the partition table!

See https://nixos.wiki/wiki/NixOS_on_ARM/UEFI#Installing

## Update, build and switch

Local (yodaTux):

```shell
niv update && colmena build --on yodaTux -v && colmena apply-local --sudo
```

Server:

```shell
niv update && colmena build --on @server -v && colmena apply --on @server switch
```

### Update

* Updating NixOS. https://superuser.com/a/1604695

Update channel and configuration:

```shell
sudo nix-channel --update && niv update
```

To apply the updates, continue with "Build and switch".

### Build and switch: Using Colmena

See section "Colmena: Deployment and secret management" to build and apply updates.

### Build and switch: Manually

* https://nixos.wiki/wiki/Nixos-rebuild
* https://discourse.nixos.org/t/how-to-get-this-pending-updates-notification-in-gnome/16344/3
* https://discourse.nixos.org/t/how-to-get-this-pending-updates-notification-in-gnome/16344/6

#### Option I: Build new config and activate it:

```shell
sudo nixos-rebuild -I nixos-config=hosts/$(hostname)/configuration.nix switch
```

To view changes, see section "Compare two versions of NixOS system profile".

#### Option II: Build new config and activate it during next boot:

```shell
sudo nixos-rebuild -I nixos-config=hosts/$(hostname)/configuration.nix boot
```

#### Option III: Build config and view changes:

```shell
# This leaves a symlink named `result` in the current directory.
sudo nixos-rebuild -I nixos-config=hosts/$(hostname)/configuration.nix build
nix --extra-experimental-features nix-command store diff-closures /run/current-system ./result
#=> ...
#=> linux: 6.1.47, 6.1.47-modules → 6.1.51, 6.1.51-modules, -11.8 KiB
```

Depending on if there are large kernel changes, either switch to it directly or wait until next boot:

```shell
sudo ./result/bin/switch-to-configuration switch
# or
sudo ./result/bin/switch-to-configuration boot
```

Note: The Perl script executed by both above commands _should_ create a new bootloader menu entry and mark it as default. However, if I remember correctly, the menu entry was once missing and the configuration change thus not permanent. This might need further testing. See also: https://nixos.wiki/wiki/Nixos-rebuild#Internals

## niv: Dependency management

* https://nix.dev/tutorials/first-steps/towards-reproducibility-pinning-nixpkgs#dependency-management-with-niv

niv:

> Easy dependency management for Nix projects.
>
> Nix is a very powerful tool for building code and setting up environments. niv complements it by making it easy to
> describe and update remote dependencies (URLs, GitHub repos, etc). It is a simple, practical alternative to Nix
> flakes.
>
> https://github.com/nmattia/niv

> Niv is an easy dependency management for Nix projects with package pinning.
>
> https://github.com/mikeroyal/NixOS-Guide

Initialize:

```shell
niv init
```

Change the tracking branch of nixpkgs from unstable to 23.05:

```shell
niv modify nixpkgs --branch nixos-23.05
```

Add nixpkgs unstable:

```shell
niv add NixOS/nixpkgs -n unstable -b nixpkgs-unstable
```

### Add Home Manager with niv

Home Manager:

> [Home Manager] allows declarative configuration of user specific (non-global) packages and dotfiles.
>
> To avoid breaking users' configurations, Home Manager is released in branches corresponding to NixOS releases (
> e.g. `release-23.05`).
>
> Home Manager provides both the channel-based setup and the flake-based one.
>
> https://github.com/nix-community/home-manager

Check your channel:

```shell
sudo nix-channel --list
#=> nixos https://nixos.org/channels/nixos-23.05
```

Use the corresponding branch:

```shell
niv add nix-community/home-manager -n home-manager -b release-23.05
```

### Add NUR with niv

> The Nix User Repository (NUR) is community-driven meta repository for Nix packages.
> 
> ... packages are built from source and are not reviewed by any Nixpkgs member.
> 
> https://github.com/nix-community/NUR

```shell
niv add nix-community/NUR -n NUR
```

## Colmena: Deployment and secret management

* https://github.com/zhaofengli/colmena#colmena

> Colmena is a simple, stateless NixOS deployment tool modeled after NixOps and morph, written in Rust.

Alternative: Deployment with Morph: https://xeiaso.net/blog/morph-setup-2021-04-25

Configuration is done inside [hive.nix](hive.nix).

Build config:

```shell
colmena build -v
```

Apply to all non-local nodes:

```shell
colmena apply --on @server switch
#colmena apply --on @server boot

#colmena apply --on @yodaTab switch
#colmena apply --on @yodaTab boot
```

Apply to local node:

```shell
colmena apply-local --sudo switch
#colmena apply-local --sudo boot
```

Filtering:
- You can filter hosts by tags with `--on @tag-a,@tag-b`.
- You can use globs in tag matching as well: `colmena apply --on '@infra-*'`

## BTRFS swap file

* Detailed instructions: https://nixos.wiki/wiki/Btrfs#Swap_file
* https://wiki.archlinux.org/title/btrfs#Swap_file

Summary:

* Create subvolume `@swap` directly below top-level subvolume.
* Mount at `/swap`
* Create swapfile: `sudo btrfs filesystem mkswapfile --size 8g --uuid clear /swap/swapfile`
* Regenerate hardware-configuration: `sudo nixos-generate-config --dir hosts/$(hostname)`
* Add `swapDevices = [ { device = "/swap/swapfile"; } ];` to hardware configuration and run `nixos-rebuild switch` (see above).

## LUKS Parameters

**Warning**: NixOS has a hardcoded timeout of 10 seconds when opening encrypted drives during boot. Please choose `--iter-time` <= `5000`.

* https://github.com/NixOS/nixpkgs/blob/ed2ccd4d1748e52d5d28c440d5be4b25a4f21c08/nixos/modules/system/boot/luksroot.nix#L498
* https://github.com/NixOS/nixpkgs/blob/ed2ccd4d1748e52d5d28c440d5be4b25a4f21c08/nixos/modules/system/boot/luksroot.nix#L30
* https://github.com/NixOS/nixpkgs/blob/ed2ccd4d1748e52d5d28c440d5be4b25a4f21c08/nixos/modules/system/boot/luksroot.nix#L36C7-L36C7

## Automount encrypted drive

~~* Generate and add keyfile to LUKS device~~
* Use the same password for all attached LUKS devices to be only prompted once while booting
  * Discussion: https://discourse.nixos.org/t/how-to-unlock-some-luks-devices-with-a-keyfile-on-a-first-luks-device/18949/11
  * Related NixOS config option: https://github.com/NixOS/nixpkgs/blob/ed2ccd4d1748e52d5d28c440d5be4b25a4f21c08/nixos/modules/system/boot/luksroot.nix#L570-L584
* luksOpen and mount drive, e.g. to `/mnt/data1`
* Re-generate hardware configuration:

```shell
sudo nixos-generate-config --dir hosts/$(hostname)
```

* If it is an SSD, enable `boot.initrd.luks.devices.<name>.allowDiscards`

## Failed services

If there are e.g. file conflicts due to preexisting dotfiles, the Home Manager user service might fail. To list all failed services, run:

```shell
systemctl --failed
```

## Garbage collection

* https://nixos.org/manual/nix/stable/package-management/garbage-collection.html
* https://discourse.nixos.org/t/why-doesnt-nix-collect-garbage-remove-old-generations-from-efi-menu/17592/4

This is automated in [base.nix](modules/base.nix) with the `nix.gc` option.

Run manually for all profiles:

```shell
sudo nix-collect-garbage --delete-older-than 14d
```

Remove leftover EFI entries of removed generations:

```shell
sudo /run/current-system/bin/switch-to-configuration boot
```

## Run AppImages

* https://nixos.wiki/wiki/Appimage

```shell
# Note how your shell prefix changes.
nix-shell -p appimage-run
```
```shell
# Inside the shell, you can run an AppImage:
appimage-run ~/Downloads/ubports-installer_0.10.0_linux_x86_64.AppImage
```

## General Notes

* There is controversy about flakes, rather use channels (e.g. with niv)
* Prins, P., Suresh, J. and Dolstra, E., "Nix fixes dependency hell on all Linux distributions," [Archived](https://web.archive.org/web/20081226010942/http://www.linux.com/feature/155922) December 26, 2008, at the Wayback Machine linux.com, December 22, 2008

### Nix Pills

* https://nixos.org/guides/nix-pills/pr01

It provides a tutorial introduction into the Nix package manager and Nixpkgs package collection, in the form of short
chapters called 'pills'.

### Papers

Papers about Nix:

* The Purely Functional Software Deployment Model (2006). http://nixos.org/~eelco/pubs/phd-thesis.pdf, https://edolstra.github.io/pubs/phd-thesis.pdf, https://github.com/edolstra/edolstra.github.io/blob/master/pubs/phd-thesis.pdf
  * May 2023: Some insights from the thesis in short form. https://jonathanlorimer.dev/posts/nix-thesis.html
* NixOS: A Purely Functional Linux Distribution (2008): https://github.com/edolstra/edolstra.github.io/blob/master/pubs/nixos-icfp2008-final.pdf, https://github.com/edolstra/edolstra.github.io/blob/master/pubs/nixos-jfp-final.pdf
* Nix: A Safe and Policy-Free System for Software Deployment (2004): https://edolstra.github.io/pubs/nspfssd-lisa2004-final.pdf

### System information

```shell
nix-info -m
```
```
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.51, NixOS, 23.05 (Stoat), 23.05.3242.da5adce0ffaf`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.13.5`
 - channels(root): `"nixos-23.05"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
```

### Search for packages

* https://search.nixos.org/packages?channel=23.05

### Search for options

* https://search.nixos.org/options?channel=23.05
    * Or `man configuration.nix`
* https://mipmip.github.io/home-manager-option-search
    * Or `man home-configuration.nix`

### Search wich package owns a file

```shell
# Note how your shell prefix changes.
nix-shell -p nix-index
```
```shell
# Either build the index manually (requires >12GB RAM):
nix-index
# Or download weekly build:
mkdir -p ~/.cache/nix-index/ && wget -q -N https://github.com/nix-community/nix-index-database/releases/latest/download/index-x86_64-linux -O ~/.cache/nix-index/files

# Then search for a file
nix-locate --whole-name '/bash'
```

### List files of package

* https://discourse.nixos.org/t/list-files-of-package/25830/2?u=langfingaz

Example for `nano`:

```shell
find $(nix-build '<nixpkgs>' -A nano --no-link)
```

Excerpt of the result:

```
/nix/store/jqvxmx65mfinbsm6db9kmcqmphl44xhp-nano-7.2/share/nano
/nix/store/jqvxmx65mfinbsm6db9kmcqmphl44xhp-nano-7.2/share/nano/asm.nanorc
/nix/store/jqvxmx65mfinbsm6db9kmcqmphl44xhp-nano-7.2/share/nano/autoconf.nanorc
```

### Compare two versions of NixOS system profile

Get latest system profile. This is the profile (usually) being active after booting the system:

```shell
ls -1 /nix/var/nix/profiles/ | sort -t'-' -n -k2 | tail -n 1
#=> 120
```

Compare current with previous profile:

```shell
# https://stackoverflow.com/a/36641298
prev="$(ls -1 /nix/var/nix/profiles/ | sort -t'-' -n -k2 | tail -n 2 | head -n 1)"
curr="$(ls -1 /nix/var/nix/profiles/ | sort -t'-' -n -k2 | tail -n 1)"
nix --extra-experimental-features nix-command store diff-closures /nix/var/nix/profiles/"${prev}" /nix/var/nix/profiles/"${curr}"
```

Compare two arbitrary system profiles:

```shell
nix --extra-experimental-features nix-command store diff-closures /nix/var/nix/profiles/system-110-link /nix/var/nix/profiles/system-116-link
```
```
cpupower: 6.1.47 → 6.1.51
element-desktop: 1.11.38 → 1.11.40, +2218.9 KiB
element-web: 1.11.38 → 1.11.40, -73.1 KiB
exempi: 2.6.3 → 2.6.4
firefox: 116.0.3 → 117.0
firefox-unwrapped: 116.0.3 → 117.0, -292.6 KiB
gnome-shell-extension-openweather: ∅ → 121, +590.5 KiB
hm_fontconfigconf.d10hmfonts.conf: ∅ → ε
initrd: ∅ → ε
initrd-linux: 6.1.47 → 6.1.51
libcap: 2.68 → 2.69
linux: 6.1.47, 6.1.47-modules → 6.1.51, 6.1.51-modules, -11.8 KiB
meld: ∅ → 3.22.0, +3858.5 KiB
net-snmp: 5.9.3 → 5.9.4
nixos-system-yodaTab: 23.05.3085.2ab91c8d65c0 → 23.05.3242.da5adce0ffaf
openjdk: +19.5 KiB
python3.10-pygobject: +27.0 KiB
stage: ∅ → 1-init.sh, +29.5 KiB
tor-browser-bundle-bin: 12.5.2 → 12.5.3, +18.1 KiB
user: +2885.0 KiB
```

### NixOS configuration debugging

* https://nixos.wiki/wiki/Nix_command/repl

Evaluating parts of the configuration.

First, start `nix repl`:

```shell
nix repl --file '<nixpkgs/nixos>' -I nixos-config=hosts/$(hostname)/configuration.nix
```

Example: `config.home-manager`

```shell
config.home-manager.
# Press `TAB`
#=> config.home-manager.backupFileExtension  config.home-manager.useUserPackages
#=> config.home-manager.extraSpecialArgs     config.home-manager.users
#=> config.home-manager.sharedModules        config.home-manager.verbose
#=> config.home-manager.useGlobalPkgs
```

Example: The `home` variable:

```shell
config.home-manager.users.yoda.home
```

Example: The value of one config option

```shell
# The following option is set to `"${config.xdg.dataHome}/.histfile";`
# where `config` is the Home Manager configuration.

config.home-manager.users.yoda.programs.zsh.history.path
#=> "/home/yoda/.local/share/.histfile"
```

### Show Nix configuration

* https://nixos.org/manual/nix/stable/command-ref/new-cli/nix3-show-config.html

```shell
nix --extra-experimental-features nix-command show-config
```

### Evaluate NixOS configuration to JSON

See also section "NixOS Configuration Debugging"!

* https://discourse.nixos.org/t/can-i-run-nix-instantiate-eval-strict-on-my-configuration-nix/7105/4

This evaluates `configuration.nix` (single module):

```shell
NIXPKGS_ALLOW_UNFREE=1 nix-instantiate --strict --json --eval -E '
import ./hosts/yodaTab/configuration.nix  {
  config = {};
  pkgs = import <nixpkgs> {};
  lib = import <nixpkgs/lib>;
}
' > evaluated-config.json
```

Then open `evaluated-config.json`.

### References

Some references to websites that helped me create this repository:

* https://github.com/Misterio77/nix-starter-configs
* https://github.com/mikeroyal/NixOS-Guide#getting-started

## TODOs

* Nitrokey LUKS unlock
  * Yubikey LUKS: https://nixos.wiki/wiki/Yubikey_based_Full_Disk_Encryption_(FDE)_on_NixOS
  * Yubikey LUKS: https://github.com/georgewhewell/nixos-host/blob/master/profiles/luks-yubi.nix
  * Old wiki entry, initramfs smartcard LUKS unlock: https://wiki.ubuntu.com/SmartCardLUKSDiskEncryption#SmartCard_Setup
* Nitrokey PAM log-in
  * https://docs.nitrokey.com/pro/linux/login-with-pam
    * You have two options: `pam_p11` or `PAM Poldi`.
    * The solution with pam_p11 is more difficult to achieve and is based on S/MIME certificates.
  * I could not find pam-poldi for NixOS :/

* Impermanence, opt-in to persistence:
  https://github.com/Misterio77/nix-starter-configs/tree/main#try-opt-in-persistance
* nix-shell / lorri
  * https://ghedam.at/15978/an-introduction-to-nix-shell
    * docker-compose.yml for services and nix-shell to run the code