Server infrastructure — l.asturias · l.jezerca · c.h.main · c.h.apps

Written on 2026-09-23 from a read-only inspection. Sources: both git repos (home-server/,
main-server/), a live SSH inventory of the hosts (docker ps, ss, systemctl,
journalctl, disk and memory stats, the Pangolin and Coolify databases), and DNS and HTTP
probes from the LAN. Nothing was changed on any host.

Updated on 2026-09-30 after Immich, Jellyfin, notes and Paperless moved, together with the
DAS, from l.asturias (called l.macmachine until then) to the new l.jezerca. Source:
, which also holds the rollback steps. Only what the
migration touched was re-checked. Everything else is still the 23 Sep snapshot. Points that
follow from the move but were not checked on a host are marked inferred.

Contents

1. The short version
2. Findings, by priority
3. Topology
4. l.asturias — the core home server
5. l.jezerca — the media and data server
6. c.h.main — the edge / ingress VPS
7. c.h.apps — the Coolify app VPS
8. Cross-cutting concerns
   • 8.1 Domain map · 8.2 Request paths · 8.3 DNS & certificates · 8.4 Deployment & updates ·
     8.5 Observability · 8.6 Secrets · 8.7 Backups · 8.8 Dependencies & failure modes
9. Suggested action list
10. Appendix — operating cheat-sheet

1. The short version

How it fits together: c.h.main is the front door. Public DNS for *.max-richter.dev
points at it. Pangolin's Traefik terminates TLS and forwards most hostnames through WireGuard
tunnels (gerbil ↔ newt) to containers at home. Each home server runs its own newt with its
own Pangolin site. On the LAN, AdGuard on l.asturias answers the same hostnames with
l.asturias's IP, so local traffic goes to a LAN-only Traefik there that serves a wildcard
certificate copied over from c.h.main. For the stacks on l.jezerca, that Traefik forwards to
published ports on 192.168.178.92 through a file-provider config. c.h.apps runs separately:
Coolify builds apps from the home Gitea, and it also hosts the Grafana LGTM stack
(observe.max-richter.dev) that all four OpenTelemetry collectors ship to.

2. Findings, by priority

Everything below was observed directly unless it is marked unverified or inferred. Status
notes dated 30 Sep come from the migration.

🔴 High — fix soon

🟠 Medium

🟡 Low / tidy-ups

• c.h.main dead config:
  • The Traefik labels in mail/compose.yaml (certresolver=cloudflare, traefik-forward-auth@docker) do nothing: Pangolin's Traefik has providers.docker: null, and Roundcube is actually routed by a Pangolin resource.
  • auto-sync.sh maps mail/roundcube-config/*, which doesn't exist (the config is mail/roundcube/config/, gitignored).
  • update.sh at the repo root is the pre-Renovate "pull everything and up -d" script. Running it now would fight the pinned-version workflow. Delete it.
  • certbot.timer runs with no certificates in /etc/letsencrypt/live.
  • fstab lists HC_Volume_8007847, which isn't attached (harmless thanks to nofail).
  • Stale UFW rules for 7890, 27896, 8080.
  • The openobserve-agent user (also on c.h.apps) has a login shell. OpenObserve has been replaced by LGTM.
• c.h.main nginx writes a large header dump to /var/log/nginx/test.log for every request (incoming_headers log format). It looks like a debugging leftover. It also counts against the 88%-full root disk.
• Pangolin config:
  • api.insecure: true on Traefik. Port 8080 isn't published, so this is internal only. The mm config at pangolin/mm/ expects it on host.docker.internal:8080.
  • serversTransport.insecureSkipVerify: true.
  • The otel exporter uses tls.insecure: true towards an HTTPS endpoint.
  • On 23 Sep the qbit Pangolin resource pointed at 0.0.0.0:9999 on c.h.main (a placeholder). qbit was among the hostnames checked from outside on 30 Sep, so it may have a real target now. The minecraft resource has no target.
• home-server stale docs:
  • renovate.json still says "watchtower deploys those".
  • filebrowser/compose.yaml mentions "the deploy timer".
  • .gitignore says "ten" SOPS files; there were twelve on 23 Sep, and the per-host newt/otel env files came since.
  • doco-cd still had LOG_LEVEL: debug on 23 Sep ("until the migration has settled", meaning the earlier move to doco-cd, which has settled).
  • README.md's install step cd /mnt/ssd && git pull doesn't work on l.asturias: that checkout is stale and dirty, and has an unreadable immich/pgdata.stale-2026-08-21/.
• Unpinned third-party images on l.jezerca: flaresolverr:latest, immich-pet-tagger:cpu, and bazarr's DOCKER_MODS=…:latest (the bazarr image itself is pinned to v1.6.2-ls366 since 30 Sep). The Immich server (v3.2.2) and ML (v3.1.0) versions also differed on 23 Sep. Immich expects them to match.
• gluetun publishes 51820/tcp+udp on the LAN for no clear reason (it's a WireGuard client).
• jellyseerr keeps its config under ${COMMON_PATH}/configs/ (the movies disk), while every other *arr keeps it under /mnt/ssd/jellyfin/configs/.
• c.h.apps Coolify clutter:
  • datasette runs twice (datasette and a4gswc8…-223935664408, same Host rule).
  • The nodarium PR previews pr-27 and pr-31 have been up for two weeks.
  • Three apps are exited (cryptpad, the old hn, invoice).
  • The app named hn.max-richter.dev serves hn2.… and vice versa.
• Unused / leftover services on l.asturias:
  • iwd/wpa_supplicant and check_ping_and_restart_iwd.sh (the box is on wired enp3s0f0).
  • virtlogd/virtlockd (libvirt).
  • podman-auto-update.timer (inactive).
  • /etc/exports exports /mnt/projects, which isn't a mount (and /mnt/media, see M12).
  • The 23 stopped pre-migration containers (restart=no) and the old copies under /mnt/ssd/{immich,jellyfin,notes,paperless}. Kept for rollback until 2026-10-07.
• Privacy note: ~/bin/save_notes.sh (cron on l.asturias, every 5 min) sends each notes diff, up to 8 000 chars, to OpenAI (gpt-4o-mini) to write the commit message.

3. Topology

flowchart LR
  subgraph Internet
    U[Browser / git client]
    CF[Cloudflare DNS<br/>apex + pass.* proxied]
  end

  subgraph MAIN["c.h.main · 78.47.148.93 · Hetzner"]
    G[gerbil<br/>:80 :443 :22 :51820/udp]
    T[traefik v3<br/>in gerbil netns]
    P[pangolin<br/>control plane]
    A[anubis<br/>PoW gate]
    NG[nginx<br/>static sites]
    SF[sftp :2221]
    MS[docker-mailserver<br/>25 143 465 587 993 4190]
    RC[roundcube]
    OM[otel-collector]
  end

  subgraph AST["l.asturias · 192.168.178.48 · LAN"]
    NW[newt<br/>WG client]
    LT[traefik LAN<br/>wildcard cert]
    AG[AdGuard DNS<br/>split horizon, whole LAN]
    GI[gitea<br/>+ registry, runner]
    SVC[(HA, vaultwarden,<br/>mcp, wg, noise, …)]
    DC[doco-cd]
    OH[otel-collector]
  end

  subgraph JEZ["l.jezerca · 192.168.178.92 · LAN"]
    NW2[newt<br/>own Pangolin site]
    SVC2[(immich, jellyfin + *arr,<br/>silverbullet, paperless)]
    DAS[(DAS: hdd, movies,<br/>kemper, media)]
    DC2[doco-cd]
    OJ[otel-collector]
  end

  subgraph APPS["c.h.apps · 167.235.31.193 · Hetzner"]
    CP[coolify-proxy traefik]
    CO[coolify]
    APPX[(15 apps / services)]
    LGTM[grafana otel-lgtm<br/>observe.max-richter.dev]
    OA[otel-collector]
  end

  U --> CF --> G
  G --- T
  T -->|pangolin HTTP provider| P
  T -->|git.*, pangolin.*| A --> T
  T -->|max-richter.dev, nodes.*, splitnir.app| NG
  T -->|mail.*| RC --> MS
  G <==>|WireGuard tunnel| NW
  G <==>|WireGuard tunnel| NW2
  NW -->|by container name on 'newt' network| SVC
  NW --> GI
  NW2 -->|'newt' network| SVC2
  SVC2 --- DAS
  LAN[LAN clients] -->|DNS| AG
  AG -->|rewrite → 192.168.178.48| LT --> SVC
  LT -->|file provider → published ports| SVC2
  GI -.->|push webhook /v1/webhook/asturias| DC
  GI -.->|push webhook /v1/webhook/jezerca| DC2
  DC & DC2 -->|git over SSH via public IP| G
  CO -->|git clone| T
  U --> CP --> APPX
  OM & OH & OJ & OA -->|OTLP/HTTP| LGTM

The Hetzner private network 10.10.0.0/16 links c.h.main (.2) and c.h.apps (.3), but
nothing in the configs uses it. Telemetry between them goes over the public
observe.max-richter.dev.

4. l.asturias — the core home server

Formerly l.macmachine. Since 30 Sep it keeps the LAN ingress, Gitea and the smaller
services; the media and data stacks run on l.jezerca.

4.1 Hardware & OS

• Apple Mac mini 7,1 (Late 2014), Intel i5-4278U (Haswell, 2 cores / 4 threads), 7.6 GiB RAM. Swap is 9.5 GiB in total: a 976 MiB partition, /swapfile (its fstab line was re-added on 30 Sep), and 4.6 GiB of zram.
• Haswell iGPU at /dev/dri. VAAPI works only through the i965 driver, with no HEVC/VP9/AV1 hardware decode. It used to transcode for Jellyfin; that now runs on l.jezerca (5.5).
• Debian GNU/Linux forky/sid, kernel 7.1.7+deb14 (23 Sep). Docker 29.8.0, Compose v5.5.1. Rootless Podman runs for user max (used by the AI reviewer).
• Wired: enp3s0f0, 192.168.178.48/24, default gateway FritzBox 192.168.178.1, plus a global IPv6 address.
  • During the migration a bad cable dropped it off the LAN, and two DHCP clients fought: systemd-networkd (metric 10) and dhcpcd (metric 1002), started by the udev unit [email protected]. Fixed with systemctl mask [email protected].
  • Once it had no IPv6 default route (RA routes only). ping -6 worked after a reboot. Keep an eye on it.
• Host services: docker, ssh, fail2ban, smartmontools, nfs-server (+ rpcbind), exim4 (localhost only), avahi, bluetooth, iwd, libvirt log/lock daemons, sysstat, lynis.timer (daily audit).
• /etc/docker/daemon.json sets the log tag {{.Name}}|{{.ImageName}}|{{.ID}} (which the otel collector parses into Loki's service_name) and container DNS 1.1.1.1 / 8.8.8.8 / 8.8.4.4. The public resolvers are also what lets containers reach Gitea's SSH (8.2). No max-size is set for json logs, unlike both VPSes.

4.2 Storage

The four DAS disks (sdc–sdf on 23 Sep) moved to l.jezerca (5.2). Their fstab lines were
removed on 30 Sep. The lines to restore them are in the migration note's rollback section.

NFS exports (/etc/exports, 23 Sep): /mnt/media and /mnt/projects → 192.168.178.0/24 (rw,sync). Neither is a mount any more (M12).

4.3 Deployment model — doco-cd GitOps (both home servers)

• The repo lives at git.max-richter.dev/max/home-server (on l.asturias's Gitea).
• doco-cd 0.110.1 (_doco/doco-cd.compose.yaml, deployed by hand) runs on both home servers and clones the repo into /opt/doco-cd/data (mode 700). On l.asturias it was recreated from git show origin/main:_doco/doco-cd.compose.yaml, because the /mnt/ssd checkout there is stale.
• Per-host deploy lists, selected by the webhook path: Gitea's push webhook calls /v1/webhook/<host> on port 9080, and doco-cd reads .doco-cd.<host>.yaml. A stack belongs in exactly one list, except newt and otel, which run on both. Auto-discovery is off on purpose so doco-cd never redeploys itself. doco-cd never stops a stack that leaves a list: stop the old containers by hand.
• No polling (it polled every 5 min until the migration). This means:
  • a restart of doco-cd deploys nothing;
  • a missed webhook stays missed until the next push. Redeliver it from Gitea's webhook log;
  • force_image_pull: true (noise-analyzer, mcp, wg, gitea, wordcloud; it replaces watchtower) only runs when a push touches that stack. A new build of an own :latest image isn't picked up on its own;
  • the first deploy on a host needs a webhook delivery. A signed local POST to http://127.0.0.1:9080/v1/webhook/<host>?wait=true works when Gitea's delivery failed.
• Git access: doco-cd clones over SSH through the public address (8.2). l.jezerca's deploy key is read-only and lives in /opt/doco-cd/secrets (the secrets dir must be owned by max).
• Secrets: the .env files are SOPS/age-encrypted and committed. The age key is at /opt/doco-cd/secrets/sops_age_key.txt on each host. doco-cd decrypts them in place inside its clone. That's why running docker compose up -d by hand in a checkout starts services with ENC[…] values. The README warns about this.
• Renovate runs in Gitea Actions every Monday at 04:00 UTC and opens one PR per service directory. Database majors are excluded.
• doco-cd errors seen on 22/23 Sep, all transient:
  • 22 Sep: Docker Hub pulls failed with "x509: certificate is valid for …fritz.box…". The FritzBox briefly answered DNS for registry-1.docker.io. The daemon's own lookups ignore daemon.json dns, which applies only to containers.
  • 23 Sep: ssh: handshake failed: EOF and 502 pulls while Traefik and Gitea restarted. This is the known self-dependency: the registry sits behind the thing being redeployed.

4.4 Stacks (.doco-cd.asturias.yaml)

4.5 Host scripts (~/bin)

sync_acme.sh is not on this host. It runs from the workstation (~/bin/sync_acme.sh, see 8.3).

5. l.jezerca — the media and data server

In service since 30 Sep. Everything here comes from ; the
OS, kernel and Docker versions weren't recorded.

5.1 Hardware & OS

• AMD Ryzen 5 7530U (6 cores / 12 threads), 15 GiB RAM, 1 TB NVMe. Radeon iGPU (amdgpu, Mesa radeonsi), used by Jellyfin for VAAPI (5.5).
• Network: wired 2.5 GbE at 192.168.178.92, with a FritzBox DHCP reservation ("Persistent IPv4"). It first linked at 100 Mbit/s because of a cable; now 1000 Mbit/s. No throughput measurement yet (no iperf3).
  • Wi-Fi wlp3s0 (192.168.178.90) is still up as a fallback. ~/.ssh/config and possibly the Gitea webhook still use it. Switching both to .92 and turning Wi-Fi off is open.
• ufw is active. Docker-published ports bypass it, but a container that talks to the host's own IP goes through ufw's INPUT chain and gets dropped (UFW BLOCK IN=br-… SRC=172.22.x DST=192.168.178.92 DPT=8085). Containers must reach each other by service name (gluetun), never via the host IP. The temporary 8080,8085/tcp from 172.16.0.0/12 rule is gone again.
• /etc/docker/daemon.json: same as l.asturias (public dns 1.1.1.1 / 8.8.8.8 / 8.8.4.4, log tag {{.Name}}|{{.ImageName}}|{{.ID}}). Containers only pick up the tag when they are created.
• Name resolution for the host comes from AdGuard on l.asturias. While asturias is down, l.jezerca has no DNS.

5.2 Storage

The DAS moved physically, so nothing on it was copied. Only app state from asturias's
/mnt/ssd was copied (tar through a root container) and compared by entries, files, bytes
and an owner/mode fingerprint. The Immich model cache was skipped and gets rebuilt. No SMART
monitoring covers any of these disks (H8).

5.3 Deployment

• doco-cd as in 4.3, reading .doco-cd.jezerca.yaml on webhook path /v1/webhook/jezerca.
• Read-only deploy key in /opt/doco-cd/secrets.
• Don't run docker compose by hand where env files are SOPS-encrypted. To stop a stack, docker stop its containers.

5.4 Stacks (.doco-cd.jezerca.yaml, 26 containers on 30 Sep)

Image versions are from the 23 Sep inventory unless noted.

5.5 Jellyfin hardware transcoding

• VAAPI via Mesa radeonsi. Decode: H.264, HEVC Main/Main10, VP9 (incl. 10-bit), MPEG-2, VC-1. Encode: H.264, HEVC. Not supported: AV1, HEVC 12-bit/4:4:4, VP8.
• encoding.xml: HardwareDecodingCodecs is h264, hevc, mpeg2video, vc1, vp9 (backup encoding.xml.bak-2026-09-30, which had asturias's Haswell list). AllowHevcEncoding is off; tone mapping is off (no HDR sources).
• Same 4K HEVC 10-bit file: 1.3× realtime with CPU decode on asturias, 3.45× GPU end to end here. 1080p HEVC 10-bit: 9.1×. The ffmpeg log shows -hwaccel vaapi, scale_vaapi, h264_vaapi, and no hwupload or libx264.

6. c.h.main — the edge / ingress VPS

6.1 Machine

• Hetzner Cloud vServer (main-01), 1 vCPU Skylake, 1.9 GiB RAM, no swap. Debian 13, kernel 6.12.107.
• Disks: sda 19 G root (88%). sdb is a Hetzner volume, HC_Volume_102570017, mounted at /mnt/static-content: 100 GiB device, 50 GiB filesystem (M1). It holds Docker's data-root (13 G), containerd (5.1 G), nginx_data (4.7 G), an old Mail (580 M), notes, sync_data.
• Docker 29.7.2, Compose v5.4.0. daemon.json: log rotation 10 M × 3, the otel tag, data-root=/mnt/static-content/docker.
• sshd on port 2222, key-only. Host port 22 belongs to gerbil (Gitea SSH).
• UFW active (allow 22, 2222, 80, 443, mail ports, 2221, 51820/udp; stale 7890, 27896, 8080; DENY 3923, which Docker bypasses). fail2ban active. CrowdSec bouncer active without its agent (M9). unattended-upgrades on.

6.2 Deployment model — auto-sync.sh

• The repo is checked out at /srv (root-owned) from [email protected]:max/main-server.
• main-server-sync.timer runs every 60 s (OnUnitActiveSec). auto-sync.sh:
  1. timeout 60 git fetch. Gitea is reached through this box's own gerbil/Traefik, so a restart can leave a half-open connection.
  2. Fast-forward only, because commits are sometimes made on the server.
  3. For each changed top-level dir: docker compose up -d --remove-orphans if the compose file changed, then docker compose restart <svc> for config files, using a path → service table (pangolin/config/traefik/* → traefik, pangolin/config/* → pangolin, nginx/*.conf → nginx, …). The Traefik file provider has no watch, so a restart is needed.
• The weak spot (H1): the apply step is keyed on git diff old..new. Any out-of-band git pull moves HEAD without applying anything, and nothing ever reconciles it.
• Renovate works like the home server's (weekly PRs, DB majors off, Traefik majors need dashboard approval because the badger plugin is pinned in traefik_config.yml).
• _archive/ holds retired stacks (drone-runner, excalidraw, headscale, minio, notes, portainer, prometheus, syncthing, old traefik+CrowdSec, wireguard).

6.3 Stacks

6.4 Static Traefik rules (pangolin/config/traefik/rules/)

• anubis.yml: pangolin.max-richter.dev. The UI goes through Anubis (priority 200). /api/v1 bypasses it (priority 201). Anubis sends traffic back to entrypoint anubis (:3923), where the UI and API routers pick it up.
• gitea.yml: git.max-richter.dev on 443 → Anubis → :3923 → http://100.89.128.4:55792, which is l.asturias's newt site address, i.e. through the tunnel to Gitea. HTTP on 80 redirects to HTTPS. secure-headers forces X-Forwarded-Proto: https so the registry generates correct URLs.
• dynamic_config.yml: HTTP→HTTPS redirect for Pangolin, and the *.nodes.max-richter.dev wildcard (its own nodes.max-richter.dev + SAN cert) → nginx.
• Git SSH: the Pangolin raw TCP resource git.max-richter.dev | SSH on :22 → gitea:2222 on l.asturias's site. gerbil sends it with the PROXY protocol, which Gitea's SSH requires (8.2).

7. c.h.apps — the Coolify app VPS

7.1 Machine

• Hetzner Cloud vServer (apps-01), 4 vCPU, 7.6 GiB RAM (5.5 G used), no swap. Debian 13, running kernel 6.12.90, 6.12.107 installed (69 days up).
• Disks: sdb 38 G root (37%). A 100 G Hetzner volume HC_Volume_103853122 at /mnt/docker-overlay is Docker's data-root (47%).
• Docker 29.8.0. daemon.json: log rotation, the otel tag, default-address-pools 10.0.0.0/8 /24.
• No firewall, no fail2ban, PasswordAuthentication yes, no unattended-upgrades (H6, M6).
• Only root and openobserve-agent have login shells.

7.2 Coolify

• Coolify 4.3.23 (+ coolify-db postgres 15, coolify-redis, coolify-realtime 1.0.19, coolify-sentinel). Its proxy is Traefik v3.6 (coolify-proxy, :80/:443/:443-udp/:8080), with ACME at /data/coolify/proxy/acme.json.
• UI published on 0.0.0.0:8000 over HTTP, realtime on :6001–6002.
• One server registered (localhost). 0 scheduled database backups.
• Git sources are [email protected]:max/*, so every build depends on the home Gitea being reachable through c.h.main.

7.3 Applications (from Coolify's DB + container labels)

Outside Coolify: /srv/compose.yaml runs an otel-collector:latest on the coolify
network (host metrics + docker logs → LGTM). It was placed by hand in Dec 2025 and isn't in
any repo.

8. Cross-cutting concerns

8.1 Domain map

8.2 Request paths

From the internet to a home service (e.g. photos.max-richter.dev):
client → 78.47.148.93:443 (gerbil) → Traefik (router from Pangolin, wildcard cert) → WireGuard (gerbil ⇄ newt) → newt on the target host → container on that host's 'newt' network. Each home server has its own newt and Pangolin site. For services on l.asturias
the LAN Traefik is not involved. For the moved stacks, a resource can also point at Traefik
on asturias, which then forwards to l.jezerca.

From the LAN (e.g. the same hostname): AdGuard answers 192.168.178.48 → Traefik on
l.asturias (wildcard cert). For services on asturias, a Docker-label router sends it to the
container. For services on l.jezerca, a file-provider router (dynamic/jezerca.yaml) sends it
to http://192.168.178.92:<published port>. Only hostnames with both an AdGuard rewrite
and a Traefik route work this way.

• Rewritten to the LAN (23 Sep): git, movies, photos, docs, notes, home, noise, movie-request, qbit.
• Resolve to the public IP (hairpin through c.h.main): wg, gitai, notizen, wordcloud, files, marka, mcp, books, audiobooks.
• Through Cloudflare: pass and the apex.
• 30 Sep, all Traefik routes to l.jezerca answer: movies, photos, notes, docs, qbit, and movie-request including /bazarr, /radarr, /prowlarr, /sabnzbd. /sonarr returns 401 by design.

wg has a Traefik label but no rewrite. The others have no LAN Traefik route at all (the LAN
Traefik returns 404 for them).

Git from containers:

• Git over SSH only works via the public address. On the LAN, git.max-richter.dev:22 is l.asturias's own sshd, and Gitea's SSH (host port 2222) only accepts PROXY-protocol connections, which only gerbil sends. Containers resolve the public IP 78.47.148.93 because Docker's dns points at public resolvers. doco-cd on both hosts and SilverBullet on l.jezerca use this path; SilverBullet pins HostName 78.47.148.93 in its SSH config.
• Over HTTPS, Traefik on l.asturias holds the alias git.max-richter.dev on the newt network, so containers there (runner, reviewer) reach Gitea without leaving the host.
• doco-cd deliberately stays off newt: via AdGuard, that name resolves to a newt-local address that refuses port 22.

8.3 DNS & certificates

• Public DNS: Cloudflare. Most records point directly at 78.47.148.93. max-richter.dev and pass. are proxied (orange cloud).
• Public certs: Pangolin's Traefik, Let's Encrypt, Cloudflare DNS-01, *.max-richter.dev wildcard preferred (stored in /srv/pangolin/config/letsencrypt/acme.json). docker-mailserver reads the same acme.json. Coolify's Traefik on c.h.apps has its own ACME store.
• LAN DNS: AdGuard on l.asturias serves the whole LAN, l.jezerca included. While asturias is down, nothing on the LAN resolves names. Containers on both home servers use public resolvers instead (daemon.json dns); the Docker daemon's own lookups (image pulls) still go to the host resolver.
• LAN certs: the LAN Traefik has no resolver. ~/bin/sync_acme.sh on the workstation pipes acme.json from c.h.main to l.asturias over two SSH hops, extracts the wildcard with jq, validates it (not expired, key matches), installs it and restarts Traefik. It runs by hand only. The current cert (installed 23 Sep 13:41) expires 2026-12-22. Pangolin renews around day 60, so re-run it in late October / November. The LAN falls back to Traefik's self-signed cert if this is forgotten.
• AdGuard rewrites must use $dnsrewrite syntax. Hosts-style entries return :: for AAAA.

8.4 Deployment & updates

Both repos share a near-identical renovate.json and .gitea/workflows/renovate.yaml. The
workflows run on the home Gitea runner, as Renovate <[email protected]>.

8.5 Observability

• Backend: grafana/otel-lgtm:latest on c.h.apps (observe.max-richter.dev): Loki, Prometheus/Mimir, Tempo and Grafana in one container, with a single Grafana data volume and no backup. Ingest is protected with HTTP Basic auth.
• Collectors (all otelcol-contrib, all using the Docker log tag name|image|id to derive service.name):
  • l.asturias: docker logs, host/docker metrics, Prometheus scrapes (speedtest), OTLP in on 127.0.0.1:4318 (restic metrics).
  • l.jezerca: docker logs (host_name=l.jezerca), host/docker metrics, jellyfin_exporter (jellyfin_up 1).
  • Both home collectors come from one otel/ folder in home-server (compose.asturias.yml / compose.jezerca.yml, a collector config per host, project name otel on both).
  • c.h.main: docker logs, host/docker metrics, plus the Traefik access-log pipeline with GeoIP. Traefik also exports metrics and traces straight to LGTM.
  • c.h.apps: :latest, hand-deployed in /srv.
• Component names differed between hosts on 23 Sep (file_log/host_metrics/otlp_http on the Mac mini vs filelog/hostmetrics/otlphttp on c.h.main), and the versions differ too (0.161 pinned, 0.159 running, latest). Both parse today. Aligning them avoids surprises on the next upgrade.
• Queryable from Claude: grafana-mcp (read-only Prometheus + Loki) in the mcp stack on l.asturias.
• Gaps: there is no SMART data at all (H8). Nothing alerts on backups (the restic.backup.last_run.timestamp_seconds metric exists), disk fill or cert expiry (unverified: I checked for alert rules in the repos only, not in Grafana).

8.6 Secrets inventory (locations only)

8.7 Backups — current vs. needed

The only backup job (restic → GCS) runs on l.asturias. Anything that now lives on
l.jezerca is out of its reach (H4).

For databases, a pg_dump/sqlite3 .backup pre-step before restic gives consistent copies.
The simplest broad fix is now one job per home server: all of /mnt/ssd on both (after
dumping the DBs), since the repo keeps all state there, plus the DAS data on l.jezerca
(/mnt/hdd, /mnt/kemper/photos).

8.8 Dependencies & failure modes

flowchart TB
  DNS[public DNS] --> MAIN
  MAIN["c.h.main<br/>gerbil/traefik · pangolin · mail · nginx"]
  AST["l.asturias<br/>gitea + registry + runner/renovate · AdGuard (LAN DNS) · LAN traefik · HA · doco-cd"]
  JEZ["l.jezerca<br/>immich · jellyfin · notes · paperless · the DAS · doco-cd"]
  APPS["c.h.apps<br/>coolify · LGTM"]
  MAIN <-->|WireGuard| AST
  MAIN <-->|WireGuard, own site| JEZ
  MAIN -->|auto-sync git fetch| AST
  APPS -->|coolify git clone| AST
  AST -->|push webhooks · LAN traefik → published ports| JEZ
  JEZ -->|DNS · git and registry pulls via c.h.main| AST
  MAIN & AST & JEZ -->|OTLP| APPS

9. Suggested action list

Ordered roughly by value per minute of effort. The migration's own open items are tracked in
.

1. Backups (H4): before the next restic run on l.asturias, check what it does with the missing DAS paths and stop it from pruning good snapshots. Then add a job on l.jezerca for /mnt/ssd (after DB dumps) and the DAS (/mnt/hdd, /mnt/kemper/photos), fix asturias's job to cover its own /mnt/ssd (Gitea from /mnt/ssd/gitea/gitea_data, plus DB dumps), add c.h.main (/srv) and Coolify's DB, and alert on restic.backup.last_run.timestamp_seconds.
2. SMART (H8): deploy smartctl-exporter on l.jezerca (DAS disk IDs + NVMe), optionally a second instance on l.asturias, re-enable the receivers in both collector configs, and add an alert.
3. c.h.main: apply the pending images: cd /srv/nginx && docker compose up -d, cd /srv/pangolin && docker compose up -d, then mail after reading the docker-mailserver v16 notes. Add a 2 GB swapfile first (H1, H7).
4. c.h.main: resize2fs /dev/sdb (M1). Remove 3923:3923 from gerbil's ports: (M2).
5. l.asturias: chmod 600 ~/bin/backup/* (H5). From 2026-10-07: remove the 23 stopped containers and their images, the old copies under /mnt/ssd/{immich,jellyfin,notes,paperless}, and the idle buildx builders, then prune again. Keep house-analysis:local and notes-bridge:latest (built locally) (H3).
6. l.asturias: move the host scripts and the NFS export that point at the DAS to l.jezerca, or retire them (M12).
7. l.jezerca: turn Wi-Fi off, point ~/.ssh/config and the Gitea webhook at 192.168.178.92, and measure throughput with iperf3. Watch /mnt/hdd capacity (~166 GB free) (M11). Confirm Immich's asset count and face/search jobs once the models are downloaded.
8. c.h.apps: Hetzner Cloud Firewall (22/80/443 only), PasswordAuthentication no, apt install unattended-upgrades, reboot into the new kernel (H6, M6).
9. Secrets: move the marka key into a SOPS env file and the LGTM Basic auth out of git, then rotate both (M3).
10. auto-sync: add drift reconciliation (compare docker compose config --images with running containers, or keep an "applied" marker) so a manual pull can't strand an update again.
11. l.asturias: apt full-upgrade + reboot (openssh, kernel). Consider moving from sid to trixie (M5). Clean up the stale /mnt/ssd git checkout and fix the README install step.
12. Tidy-ups from the Low list: stale comments, dead labels, update.sh, UFW rules, CrowdSec, the duplicate datasette, the PR previews, the Jellyfin Renovate regex (M8), doco-cd LOG_LEVEL.

10. Appendix — operating cheat-sheet

# ---- l.asturias / l.jezerca (doco-cd works the same on both) ----
ssh l.asturias                                          # 192.168.178.48
ssh l.jezerca                                           # alias still points at Wi-Fi .90; wired is .92
docker logs -f doco-cd                                  # deploy log
docker ps --filter label=com.docker.compose.project=<stack>
# No polling: restarting doco-cd deploys nothing. Redeliver the push from Gitea's webhook
# log, or send a signed POST locally to http://127.0.0.1:9080/v1/webhook/<host>?wait=true
SOPS_AGE_KEY_FILE=/opt/doco-cd/secrets/sops_age_key.txt sops edit <stack>/.env
# NEVER: docker compose up -d by hand where env files are SOPS-encrypted (starts with ENC[...] secrets)
tail -f /var/log/backup.log                             # l.asturias only

# ---- c.h.main ----
ssh c.h.main                                            # port 2222
journalctl -u main-server-sync -f
systemctl start main-server-sync                        # sync now
cd /srv/<dir> && docker compose config --images         # what SHOULD run
docker ps --format '{{.Names}} {{.Image}}'              # what DOES run
docker exec mailserver setup email list

# ---- c.h.apps ----
ssh c.h.apps
# Coolify UI: use its HTTPS domain, not :8000
docker exec coolify-db psql -U coolify -d coolify -c 'select name,fqdn,status from applications'

# ---- workstation ----
~/bin/sync_acme.sh                                      # refresh LAN wildcard cert (before 2026-12-22)

Repo references

• home-server/README.md: doco-cd rules, Gitea-down recovery, SOPS usage (its install step doesn't work on l.asturias)
• home-server/.doco-cd.asturias.yaml, home-server/.doco-cd.jezerca.yaml: the per-host lists of deployed stacks
• home-server/traefik/config/traefik/dynamic/jezerca.yaml: LAN routes from Traefik on asturias to l.jezerca
• home-server/newt/compose.jezerca.yaml, home-server/otel/compose.{asturias,jezerca}.yml: per-host overrides
• home-server/home-assistant/analysis/README.md: the house-analysis report
• main-server/_deploy/README.md: auto-sync behaviour and the restart table
• main-server/pangolin/config/traefik/: static ingress config
• : migration log, lessons learned and rollback