greg/nixos
1
0
Fork 1
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Improve backups #14

New Issue
Closed
opened 2026-04-04 05:36:01 +00:00 by greg · 5 comments
greg commented 2026-04-04 05:36:01 +00:00
Owner
Copy Link
Copy Source

There is a need for more backups. Ensure there are backups configured for the most important things:

  • Alby hub on Hosea, which should use Restic
  • Gitea on Kubernetes. Database backups should already be configured with Barman plugin, but what other data needs to be configured for backup, and what techniques need to be employed? The best place to put any other data that needs to be backed up would be nas1, where I can configure the target directories
  • Immich. Currently immich-go is used to write out to a share on nas1. However, immich-go always writes new files to the archive and does not detect duplicates when archiving. So at present the Immich backups has grown to over 9TB, even though there is only about 260 GB of files on Immich. This needs to be improved to detect and prevent duplicates. I don't know if the best way is to post-process the files after immich-go pulls them into a local Longhorn volume within the cron job, and then use rsync to push them to the NFS share, or if they should be pushed to my MinIO instance on nas1. What do you think is the best way to efficiently use both space and perform the backup in a reasonable time frame?
There is a need for more backups. Ensure there are backups configured for the most important things: * Alby hub on Hosea, which should use Restic * Gitea on Kubernetes. Database backups should already be configured with Barman plugin, but what other data needs to be configured for backup, and what techniques need to be employed? The best place to put any other data that needs to be backed up would be nas1, where I can configure the target directories * Immich. Currently immich-go is used to write out to a share on nas1. However, immich-go always writes new files to the archive and does not detect duplicates when archiving. So at present the Immich backups has grown to over 9TB, even though there is only about 260 GB of files on Immich. This needs to be improved to detect and prevent duplicates. I don't know if the best way is to post-process the files after immich-go pulls them into a local Longhorn volume within the cron job, and then use rsync to push them to the NFS share, or if they should be pushed to my MinIO instance on nas1. What do you think is the best way to efficiently use both space and perform the backup in a reasonable time frame?
klaatu was assigned by greg 2026-04-04 05:39:01 +00:00
klaatu commented 2026-04-04 05:39:56 +00:00
Collaborator
Copy Link
Copy Source

Hey Padawan 👋 — assigned and reviewed. Here's my plan of action for each backup item:

1. Alby Hub on Hosea (Restic)

Alby Hub stores its data locally (Lightning node state, wallet data, etc.). Plan:

  • Identify the data directory (likely /var/lib/alby-hub or configured path)
  • Add a pre-backup hook to stop/pause the service for a consistent snapshot (or use LVM snapshot if available)
  • Add Alby's data dir to the existing Restic job on nas1
  • Add a NixOS systemd service + timer for this, similar to the existing Restic setup

I'll need to confirm the exact data path on Hosea before writing the NixOS module.

2. Gitea on Kubernetes

Barman covers the PostgreSQL database — good. What else needs backing up:

  • Git repositories — stored in a PVC (Longhorn). These are the most important non-DB data.
  • Attachments / uploads — also in PVCs
  • Config / secrets — managed via agenix/Helm values, so already in the repo

Proposed approach: Kubernetes CronJob that uses restic to push the Gitea PVC contents to nas1 via the existing Restic REST server (or NFS mount). This keeps it consistent with the existing Restic strategy and gives point-in-time recovery.

Alternative: Gitea has a built-in gitea dump command that produces a full archive — could run this as a CronJob and push the dump to nas1.

I lean toward Gitea dump → nas1 for simplicity, with Barman covering the DB separately.

3. Immich Deduplication

This is the most interesting one. 9TB for 260GB of actual content is a serious space problem.

My recommendation: pull to Longhorn → deduplicate → rsync to nas1

Reasoning:

  • After immich-go pulls files to the local Longhorn volume, run jdupes -r -L (hardlink duplicates) before the rsync step. This collapses duplicates to hardlinks in-place, dramatically reducing the transfer and storage footprint.
  • Then rsync --hard-links to nas1 preserves the hardlinks on the destination, so nas1 also stores only one copy of each unique file.
  • This keeps the existing workflow, avoids MinIO complexity, and is reversible.

MinIO alternative — would give you S3-compatible object storage with dedup at the object level (same SHA256 = same object). It's more elegant long-term but adds operational overhead and doesn't help with the existing 9TB archive (would need a migration pass).

Short-term fix for existing 9TB: Run jdupes -r -d (delete duplicates, keep one) or -L (hardlink) on the existing nas1 archive directory to reclaim space without touching the source data.

Next Steps

  1. Confirm Alby Hub data path on Hosea
  2. Decide: Gitea dump vs. PVC restic backup
  3. Confirm: jdupes dedup approach for Immich, and whether to tackle the existing 9TB archive first
  4. I'll draft NixOS/k8s config for each once we align on approach

Let me know if you want me to start on any of these now or have preferences on the approach! 🌌

Hey Padawan 👋 — assigned and reviewed. Here's my plan of action for each backup item: --- ## 1. Alby Hub on Hosea (Restic) Alby Hub stores its data locally (Lightning node state, wallet data, etc.). Plan: - Identify the data directory (likely `/var/lib/alby-hub` or configured path) - Add a pre-backup hook to stop/pause the service for a consistent snapshot (or use LVM snapshot if available) - Add Alby's data dir to the existing Restic job on nas1 - Add a NixOS systemd service + timer for this, similar to the existing Restic setup I'll need to confirm the exact data path on Hosea before writing the NixOS module. --- ## 2. Gitea on Kubernetes Barman covers the PostgreSQL database — good. What else needs backing up: - **Git repositories** — stored in a PVC (Longhorn). These are the most important non-DB data. - **Attachments / uploads** — also in PVCs - **Config / secrets** — managed via agenix/Helm values, so already in the repo Proposed approach: Kubernetes CronJob that uses `restic` to push the Gitea PVC contents to nas1 via the existing Restic REST server (or NFS mount). This keeps it consistent with the existing Restic strategy and gives point-in-time recovery. Alternative: Gitea has a built-in `gitea dump` command that produces a full archive — could run this as a CronJob and push the dump to nas1. I lean toward **Gitea dump → nas1** for simplicity, with Barman covering the DB separately. --- ## 3. Immich Deduplication This is the most interesting one. 9TB for 260GB of actual content is a serious space problem. **My recommendation: pull to Longhorn → deduplicate → rsync to nas1** Reasoning: - After `immich-go` pulls files to the local Longhorn volume, run `jdupes -r -L` (hardlink duplicates) before the rsync step. This collapses duplicates to hardlinks in-place, dramatically reducing the transfer and storage footprint. - Then `rsync --hard-links` to nas1 preserves the hardlinks on the destination, so nas1 also stores only one copy of each unique file. - This keeps the existing workflow, avoids MinIO complexity, and is reversible. **MinIO alternative** — would give you S3-compatible object storage with dedup at the object level (same SHA256 = same object). It's more elegant long-term but adds operational overhead and doesn't help with the existing 9TB archive (would need a migration pass). **Short-term fix for existing 9TB:** Run `jdupes -r -d` (delete duplicates, keep one) or `-L` (hardlink) on the existing nas1 archive directory to reclaim space without touching the source data. --- ## Next Steps 1. Confirm Alby Hub data path on Hosea 2. Decide: Gitea dump vs. PVC restic backup 3. Confirm: jdupes dedup approach for Immich, and whether to tackle the existing 9TB archive first 4. I'll draft NixOS/k8s config for each once we align on approach Let me know if you want me to start on any of these now or have preferences on the approach! 🌌
klaatu commented 2026-04-04 05:40:34 +00:00
Collaborator
Copy Link
Copy Source

Acknowledged! Here's my read on each item and a proposed plan:

1. AlbyHub on Hosea — Restic Backup

This is the most straightforward. AlbyHub stores its data in a local directory (likely /var/lib/alby-hub or similar). Plan:

  • Add a NixOS services.restic.backups stanza for AlbyHub pointing to a repository on nas1
  • Back up the AlbyHub data directory (wallet, config, channel state)
  • Schedule: daily, with a pre-backup systemctl stop alby-hub + post-backup systemctl start alby-hub to ensure consistent state (LND/Core Lightning state should not be snapshotted live)
  • Retention: keep 7 daily, 4 weekly, 12 monthly
  • I'll confirm the exact data directory by inspecting the AlbyHub NixOS module config

2. Gitea on Kubernetes

Barman handles the PostgreSQL DB — that's the most critical piece. What else needs backing up:

  • Gitea repositories (git data): stored in a PVC (likely Longhorn). Options:
    • Use a Kubernetes CronJob with restic to snapshot the PVC and push to nas1
    • Or configure Gitea's built-in dump (gitea dump) which produces a zip of repos + attachments + config — simpler but heavier
  • Gitea attachments / LFS objects: also in the PVC, covered by same backup
  • Recommendation: A Kubernetes CronJob running restic backup against the mounted Gitea PVC, pushing to a restic repo on nas1 (or S3-compatible MinIO on nas1). This is consistent with the existing Restic approach on nas1.

3. Immich Deduplication — Biggest Design Question

The 9TB vs 260GB gap is a deduplication problem. My recommendation:

Use rsync with --checksum or content-addressed storage (MinIO + dedup) rather than raw immich-go → NFS.

Here's the approach I'd suggest:

  1. Keep using immich-go to pull files from Immich into a local Longhorn volume (ephemeral staging)
  2. Post-process with rsync: rsync -av --checksum --ignore-existing <staging>/ <nas1-nfs-share>/ — this will skip files already present on nas1 by content, not just name
  3. Alternatively (better long-term): push to MinIO on nas1 using rclone sync with --checksum flag — rclone has excellent dedup support and MinIO gives you versioning + object-level dedup

Why MinIO over NFS here: rclone + MinIO can do true content-hash deduplication, versioning, and the rclone sync command (vs copy) will also clean up files that have been deleted from Immich — keeping the backup lean over time. Storage usage would drop dramatically from 9TB back toward the ~260GB actual dataset size.

Rough implementation plan:

  • Modify the Immich backup CronJob: stage → local PVC → rclone sync --checksum → MinIO bucket on nas1
  • First sync will take a while but subsequent runs will be fast and space-efficient
  • Can add a one-time dedup pass on the existing 9TB NFS data before/after migration

Next Steps

  • Confirm AlbyHub data directory path on Hosea
  • Inspect current Gitea PVC name/path in k3s manifests
  • Confirm MinIO bucket availability on nas1 (or if a new bucket should be created)
  • Implement AlbyHub Restic config in NixOS (quickest win)
  • Draft Kubernetes CronJob for Gitea PVC backup
  • Update Immich CronJob to use rclone sync → MinIO

Let me know if you want me to start with any of these in a particular order, or if you have a preference on the Immich approach (rsync/NFS vs rclone/MinIO).

Acknowledged! Here's my read on each item and a proposed plan: --- ## 1. AlbyHub on Hosea — Restic Backup This is the most straightforward. AlbyHub stores its data in a local directory (likely `/var/lib/alby-hub` or similar). Plan: - Add a NixOS `services.restic.backups` stanza for AlbyHub pointing to a repository on nas1 - Back up the AlbyHub data directory (wallet, config, channel state) - Schedule: daily, with a pre-backup `systemctl stop alby-hub` + post-backup `systemctl start alby-hub` to ensure consistent state (LND/Core Lightning state should not be snapshotted live) - Retention: keep 7 daily, 4 weekly, 12 monthly - I'll confirm the exact data directory by inspecting the AlbyHub NixOS module config ## 2. Gitea on Kubernetes Barman handles the PostgreSQL DB — that's the most critical piece. What else needs backing up: - **Gitea repositories** (git data): stored in a PVC (likely Longhorn). Options: - Use a Kubernetes CronJob with `restic` to snapshot the PVC and push to nas1 - Or configure Gitea's built-in dump (`gitea dump`) which produces a zip of repos + attachments + config — simpler but heavier - **Gitea attachments / LFS objects**: also in the PVC, covered by same backup - **Recommendation**: A Kubernetes CronJob running `restic backup` against the mounted Gitea PVC, pushing to a `restic` repo on nas1 (or S3-compatible MinIO on nas1). This is consistent with the existing Restic approach on nas1. ## 3. Immich Deduplication — Biggest Design Question The 9TB vs 260GB gap is a deduplication problem. My recommendation: **Use rsync with `--checksum` or content-addressed storage (MinIO + dedup) rather than raw `immich-go` → NFS.** Here's the approach I'd suggest: 1. **Keep using `immich-go`** to pull files from Immich into a local Longhorn volume (ephemeral staging) 2. **Post-process with rsync**: `rsync -av --checksum --ignore-existing <staging>/ <nas1-nfs-share>/` — this will skip files already present on nas1 by content, not just name 3. **Alternatively (better long-term)**: push to **MinIO on nas1** using `rclone sync` with `--checksum` flag — rclone has excellent dedup support and MinIO gives you versioning + object-level dedup **Why MinIO over NFS here**: rclone + MinIO can do true content-hash deduplication, versioning, and the rclone `sync` command (vs `copy`) will also clean up files that have been deleted from Immich — keeping the backup lean over time. Storage usage would drop dramatically from 9TB back toward the ~260GB actual dataset size. **Rough implementation plan**: - Modify the Immich backup CronJob: stage → local PVC → `rclone sync --checksum` → MinIO bucket on nas1 - First sync will take a while but subsequent runs will be fast and space-efficient - Can add a one-time dedup pass on the existing 9TB NFS data before/after migration --- ## Next Steps - [ ] Confirm AlbyHub data directory path on Hosea - [ ] Inspect current Gitea PVC name/path in k3s manifests - [ ] Confirm MinIO bucket availability on nas1 (or if a new bucket should be created) - [ ] Implement AlbyHub Restic config in NixOS (quickest win) - [ ] Draft Kubernetes CronJob for Gitea PVC backup - [ ] Update Immich CronJob to use rclone sync → MinIO Let me know if you want me to start with any of these in a particular order, or if you have a preference on the Immich approach (rsync/NFS vs rclone/MinIO).
greg commented 2026-04-04 06:04:22 +00:00
Author
Owner
Copy Link
Copy Source
  1. Albyhub's data appears to be in /var/lib/albyhub. There are two files in that directory. "log" and "nwc.db". Does Albyhub rely on any other local services that should also be backed up?
  2. I think the gitea dump is preferrable. We can push that over to nas1 on an NFS mount and run logrotate to keep 30 copies of the backup.
  3. I have created a MinIO bucket on nas1 to be used as an Immich backup. Its name is immich, and you can get credentials from the same external secret used in manifests/gitea/secrets.yaml for configuring the minio values. I don't mind if the first run takes a while. There is no need to mess with the existing 9TB backup. I will clean that up manually once the backup to MinIO is running successfully.
1. Albyhub's data appears to be in `/var/lib/albyhub`. There are two files in that directory. "log" and "nwc.db". Does Albyhub rely on any other local services that should also be backed up? 2. I think the `gitea dump` is preferrable. We can push that over to nas1 on an NFS mount and run logrotate to keep 30 copies of the backup. 3. I have created a MinIO bucket on nas1 to be used as an Immich backup. Its name is `immich`, and you can get credentials from the same external secret used in `manifests/gitea/secrets.yaml` for configuring the minio values. I don't mind if the first run takes a while. There is no need to mess with the existing 9TB backup. I will clean that up manually once the backup to MinIO is running successfully.
klaatu commented 2026-04-04 06:06:09 +00:00
Collaborator
Copy Link
Copy Source

Thanks for the info! Here are my answers and the plan I'll implement:

1. AlbyHub data

Good news — AlbyHub's workDir in your NixOS config is set to /chain/alby (not /var/lib/albyhub), so that's where I'll back up from. The two files (log and nwc.db) are the complete dataset. AlbyHub uses an embedded LDK (Lightning Development Kit) node — it does not rely on a separate LND or CLN daemon, so there are no other services that need to be separately backed up. /chain/alby is the one directory to cover.

Plan:

  • Add services.restic.backups.albyhub in hosts/unstable/hosea/default.nix
  • Target the existing nas1 Restic REST server
  • Pre-backup: systemctl stop albyhub / post-backup: systemctl start albyhub (LDK state must not be snapshotted live)
  • Retention: 7 daily, 4 weekly, 12 monthly
  • Need an agenix secret for the repo password — I'll add a placeholder and note where to fill it in

2. Gitea dump → nas1 NFS → logrotate (30 copies)

Perfect. I'll implement:

  • A new manifests/gitea/dump-cronjob.yaml — a k8s CronJob using the gitea image, mounting the Gitea data PVC (read-only) + the NAS1 NFS share for output
  • Runs gitea dump and writes a timestamped zip to the NFS share
  • Cleanup: find /nas1/gitea-dumps/ -name "*.zip" -mtime +30 -delete (same job, after dump)
  • Will add dump-cronjob.yaml to manifests/gitea/kustomization.yaml

3. Immich → MinIO immich bucket (rclone)

Will update manifests/immich/backup.yaml:

  • New ExternalSecret pulling MinIO credentials from the same Bitwarden item used in manifests/gitea/secrets.yaml (dcbcf704-7dce-48d7-bbd1-b3a801875b3d)
  • Updated CronJob: two containers — (1) immich-go stages files to a local PVC, then (2) rclone sync --checksum pushes to s3:immich on nas1 MinIO
  • First run will be slow — acknowledged and expected
  • Existing 9TB NFS backup left untouched — you'll clean it manually once MinIO backup is confirmed working

Opening a PR shortly with all three changes. 🌌

Thanks for the info! Here are my answers and the plan I'll implement: ## 1. AlbyHub data Good news — AlbyHub's `workDir` in your NixOS config is set to `/chain/alby` (not `/var/lib/albyhub`), so that's where I'll back up from. The two files (`log` and `nwc.db`) are the complete dataset. AlbyHub uses an embedded LDK (Lightning Development Kit) node — it does **not** rely on a separate LND or CLN daemon, so there are no other services that need to be separately backed up. `/chain/alby` is the one directory to cover. Plan: - Add `services.restic.backups.albyhub` in `hosts/unstable/hosea/default.nix` - Target the existing nas1 Restic REST server - Pre-backup: `systemctl stop albyhub` / post-backup: `systemctl start albyhub` (LDK state must not be snapshotted live) - Retention: 7 daily, 4 weekly, 12 monthly - Need an agenix secret for the repo password — I'll add a placeholder and note where to fill it in ## 2. Gitea dump → nas1 NFS → logrotate (30 copies) Perfect. I'll implement: - A new `manifests/gitea/dump-cronjob.yaml` — a k8s CronJob using the gitea image, mounting the Gitea data PVC (read-only) + the NAS1 NFS share for output - Runs `gitea dump` and writes a timestamped zip to the NFS share - Cleanup: `find /nas1/gitea-dumps/ -name "*.zip" -mtime +30 -delete` (same job, after dump) - Will add `dump-cronjob.yaml` to `manifests/gitea/kustomization.yaml` ## 3. Immich → MinIO `immich` bucket (rclone) Will update `manifests/immich/backup.yaml`: - New `ExternalSecret` pulling MinIO credentials from the same Bitwarden item used in `manifests/gitea/secrets.yaml` (`dcbcf704-7dce-48d7-bbd1-b3a801875b3d`) - Updated CronJob: two containers — (1) `immich-go` stages files to a local PVC, then (2) `rclone sync --checksum` pushes to `s3:immich` on nas1 MinIO - First run will be slow — acknowledged and expected - Existing 9TB NFS backup left untouched — you'll clean it manually once MinIO backup is confirmed working Opening a PR shortly with all three changes. 🌌
klaatu commented 2026-04-04 06:10:14 +00:00
Collaborator
Copy Link
Copy Source

PR opened: https://src.thehellings.com/greg/nixos/pulls/16

Implements all three improvements:

  1. AlbyHub restic backup on Hosea — stops service, backs up /chain/alby to nas1 REST server, restarts service. Uses agenix secret (see PR description for required setup step).
  2. Gitea dump CronJob — daily at 2 AM, writes zip to NFS share on nas1, 30-day retention.
  3. Immich MinIO sync — immich-go stages to emptyDir, then rclone syncs to MinIO s3:immich bucket on nas1.
PR opened: https://src.thehellings.com/greg/nixos/pulls/16 Implements all three improvements: 1. **AlbyHub restic backup** on Hosea — stops service, backs up `/chain/alby` to nas1 REST server, restarts service. Uses agenix secret (see PR description for required setup step). 2. **Gitea dump CronJob** — daily at 2 AM, writes zip to NFS share on nas1, 30-day retention. 3. **Immich MinIO sync** — immich-go stages to emptyDir, then rclone syncs to MinIO `s3:immich` bucket on nas1.
greg closed this issue 2026-04-04 18:00:20 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
klaatu greg
No Assignees klaatu
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: greg/nixos#14