Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Pablo Fraile Infrastructure docs

NixOS Configurations

Known issues:

  • We’re using autofirma-nix for signing files with my dni (spain identification certificate). It has some issues:
    • Opensc doesn’t allow to have the yubikey and a smartcard connected at the same time. You have to disconnect the yubikey and then add the smartcard with the dni-e before openning the program autofirma.
    • You must run autofirma-setup (with firefox opened) the first time you install firefox.

Other devices not managed by nixos

Updating the HP firmware

This guide updates the X89 system firmware on saruman, an HP ZBook Ultra G1a, from NixOS with Lanzaboote and Secure Boot enabled.

Check the current version

Connect the charger, then check the installed BIOS and the fwupd device:

cat /sys/class/dmi/id/bios_version
fwupdmgr get-devices

The system-firmware GUID should be 6b887c32-2e1f-462a-a1bc-2d08ce9ffac5.

Download and inspect the update

Use the official ftp.hp.com SoftPaq URL from the GitHub issue. Work in a new directory so an older firmware image cannot be selected accidentally:

softpaq_url='paste-the-softpaq-url-from-the-issue-here'
update_directory="$HOME/Downloads/hp-bios-update"

mkdir "$update_directory"
cd "$update_directory"
curl --fail --location --remote-name "$softpaq_url"
nix shell nixpkgs#p7zip -c 7z x "${softpaq_url##*/}"

Read the values that will be passed to fwupd:

rg 'FirmwareId|FirmwareVersion|FirmwareFilename' X89_*.inf

Confirm that FirmwareId matches the GUID above. Convert the hexadecimal FirmwareVersion from the INF file to decimal; for example, version 0x01060200 becomes 17170944:

printf '%d\n' "$((0x01060200))"

Sign the fwupd boot helper

NixOS supplies the EFI helper, but Secure Boot requires a copy signed with the same db key used by Lanzaboote:

sudo systemd-tmpfiles --create --prefix=/run/fwupd-efi

sbsign="$(nix build --no-link --print-out-paths nixpkgs#sbsigntool)/bin/sbsign"
sudo "$sbsign" \
  --key /etc/secureboot/keys/db/db.key \
  --cert /etc/secureboot/keys/db/db.pem \
  --output /run/fwupd-efi/fwupdx64.efi.signed \
  /run/fwupd-efi/fwupdx64.efi

This signs the fwupd EFI helper, not HP’s firmware .bin file. Do not reboot between this step and staging the update because /run is temporary.

Install and verify

Set the filename and hexadecimal version to the values from the new INF file:

firmware_bin='X89_replace_with_the_INF_value.bin'
firmware_version=0x_replace_with_the_INF_value

sudo fwupdtool install-blob \
  "$firmware_bin" \
  6b887c32-2e1f-462a-a1bc-2d08ce9ffac5 \
  "$((firmware_version))"

Reboot when prompted, keep the charger connected, and do not interrupt the firmware update. After NixOS starts again, verify the result:

cat /sys/class/dmi/id/bios_version
fwupdmgr get-devices

The System Firmware entry should show the new version and Update State: Success.

Remove a stale updater boot entry

Normally fwupd cleans up its temporary boot entry. If the computer instead keeps booting EFI/nixos/fwupdx64.efi, inspect the entries and find the number named Linux Firmware Updater:

efibootmgr_bin="$(nix build --no-link --print-out-paths nixpkgs#efibootmgr)/bin/efibootmgr"
sudo "$efibootmgr_bin" -v

Only after confirming that the firmware update succeeded, replace XXXX with that entry’s number and remove it before deleting the helper:

sudo "$efibootmgr_bin" -b XXXX -B
sudo rm -- /boot/EFI/nixos/fwupdx64.efi
sudo "$efibootmgr_bin" -v

Do not delete any other file under /boot/EFI/nixos/.

Observability module options

observability.enable

Whether to enable host observability.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.alloy.enable

Whether to enable Grafana Alloy journal collection.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.alloy.endpoint_url

Full Loki push endpoint used by Grafana Alloy.

Type: null or string

Default:

null

Example:

"https://loki.example.com/loki/api/v1/push"

Declared by:

observability.prometheus.alerts

Prometheus alert rules registered by service name.

Rules are evaluated on the host running the observability stack.

Type: attribute set of (submodule)

Default:

{ }

Declared by:

observability.prometheus.alerts.<name>.rules

Prometheus rules contributed by this service.

Type: list of strings concatenated with “\n”

Default:

[ ]

Declared by:

observability.prometheus.exporterListenAddress

Address used by the host Prometheus exporters.

Keep this on 127.0.0.1 when Prometheus runs on this host.

For remote scraping, bind it to a trusted LAN/VPN address or 0.0.0.0 and restrict access using the firewall.

Type: string

Default:

"127.0.0.1"

Declared by:

observability.prometheus.node.targets

node_exporter instances scraped by the local Prometheus server.

Type: list of string

Default:

[ ]

Example:

[
  "server2:9100"
  "server3:9100"
]

Declared by:

observability.prometheus.smartctl.devices

Disks monitored by smartctl-exporter.

An empty list lets smartctl-exporter auto-discover disks.

Type: list of string

Default:

[ ]

Example:

[
  "/dev/disk/by-id/ata-Samsung_SSD..."
  "/dev/disk/by-id/nvme-Samsung_SSD..."
]

Declared by:

observability.prometheus.smartctl.targets

smartctl exporters scraped by the local Prometheus server.

Type: list of string

Default:

[ ]

Example:

[
  "server2:9633"
  "server3:9633"
]

Declared by:

observability.prometheus.targets

Additional Prometheus scrape targets.

Individual service modules should append their metrics endpoints here.

Type: list of (submodule)

Default:

[ ]

Example:

[
  {
    name = "immich-api";
    targets = [
      "localhost:8081"
    ];
  }
  {
    name = "immich-microservices";
    targets = [
      "localhost:8082"
    ];
  }
]

Declared by:

observability.prometheus.targets.*.metricsPath

Optional metrics path for this Prometheus job.

Type: null or string

Default:

null

Example:

"/ups_metrics"

Declared by:

observability.prometheus.targets.*.name

Prometheus job name.

Type: string

Example:

"immich-api"

Declared by:

observability.prometheus.targets.*.params

Optional query parameters for this Prometheus job.

Type: attribute set of list of string

Default:

{ }

Example:

{
  ups = [
    "ups-eaton"
  ];
}

Declared by:

observability.prometheus.targets.*.targets

Targets belonging to this Prometheus job.

Type: list of string

Default:

[ ]

Example:

[
  "localhost:8081"
  "server2:8081"
]

Declared by:

observability.prometheus_tailscale.enable

Whether to enable Prometheus Tailscale exporter.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.prometheus_tailscale.environmentFile

Environment file containing TAILSCALE_TAILNET, TAILSCALE_OAUTH_CLIENT_ID, and TAILSCALE_OAUTH_CLIENT_SECRET.

Type: absolute path

Declared by:

observability.stack.alertmanager.enable

Whether to enable Prometheus alert notifications.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.stack.alertmanager.mail.enable

Whether to enable email alert notifications.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.stack.alertmanager.mail.environmentFile

Environment file containing SMTP_PASSWORD.

Type: absolute path

Declared by:

observability.stack.alertmanager.mail.from

Sender address. Defaults to the SMTP username.

Type: null or non-empty string

Default:

null

Example:

"alerts@mailbox.org"

Declared by:

observability.stack.alertmanager.mail.server

SMTP server and port.

Type: non-empty string

Default:

"smtp.mailbox.org:587"

Declared by:

observability.stack.alertmanager.mail.to

Addresses that receive alert notifications.

Type: non-empty (list of non-empty string)

Example:

[
  "admin@mailbox.org"
]

Declared by:

observability.stack.alertmanager.mail.username

SMTP authentication username.

Type: non-empty string

Example:

"alerts@mailbox.org"

Declared by:

observability.stack.alertmanager.ntfy.enable

Whether to enable ntfy alert notifications.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.stack.alertmanager.ntfy.secretConfigFile

YAML file merged into alertmanager-ntfy’s configuration. It should contain the ntfy authentication token under ntfy.auth.token.

Type: absolute path

Declared by:

observability.stack.alertmanager.ntfy.server

Base URL of the ntfy server.

Type: string

Example:

"https://ntfy.sh"

Declared by:

observability.stack.alertmanager.ntfy.topic

ntfy topic to which alerts are published.

Type: string

Declared by:

observability.stack.alertmanager.public_url

Public URL used by Alertmanager when generating links.

Type: string

Default:

"http://no.url.org"

Declared by:

observability.stack.grafana.enable

Whether to enable Grafana.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.stack.grafana.adminPasswordFile

File containing the initial Grafana administrator password.

The secret itself is not copied into the Nix store.

Type: null or string

Default:

null

Example:

"/run/agenix/grafana-admin-password"

Declared by:

observability.stack.grafana.oidc.enable

Whether to enable Pocket ID login for Grafana.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.stack.grafana.oidc.clientId

Pocket ID client ID for Grafana.

Type: string

Declared by:

observability.stack.grafana.oidc.clientSecretFile

File containing the Pocket ID client secret for Grafana.

Type: absolute path

Declared by:

observability.stack.grafana.secretKeyFile

File containing Grafana’s secret key.

Type: null or string

Default:

null

Example:

"/run/agenix/grafana-secret-key"

Declared by:

observability.stack.loki.enable

Whether to enable Loki.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.stack.prometheus.enable

Whether to enable Prometheus.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.stack.prometheus.oidc.enable

Whether to enable Pocket ID authentication for the Prometheus UI.

Type: boolean

Default:

false

Example:

true

Declared by:

observability.stack.prometheus.oidc.clientId

Pocket ID client ID for Prometheus.

Type: string

Declared by:

observability.stack.prometheus.oidc.environmentFile

Environment file containing OAUTH2_PROXY_CLIENT_SECRET and OAUTH2_PROXY_COOKIE_SECRET.

Type: absolute path

Declared by: