Implements three backup improvements from issue #14:
1. AlbyHub Restic Backup (Hosea)
Adds services.restic.backups.albyhub to hosts/unstable/hosea/default.nix
Backs up /chain/alby to rest:https://nas1.shire-zebra.ts.net:30248/albyhub
Stops AlbyHub before backup and restarts after for LDK state consistency
Uses agenix secret albyhub-restic-password for repo password
Prunes snapshots: 7 daily, 4 weekly, 3 monthly
Action required: Run agenix -e secrets/albyhub-restic-password.age before deploying
2. Gitea Dump CronJob (Kubernetes)
Adds manifests/gitea/dump-cronjob.yaml
Runs daily at 2 AM (offset from 4 AM Immich backup)
Mounts Gitea data PVC (read-only) and NFS share nas1.shire-zebra.ts.net:/mnt/all/backups/gitea-dumps
Runs gitea dump as zip, then cleans up dumps older than 30 days
Added to manifests/gitea/kustomization.yaml
3. Immich MinIO Sync (Kubernetes)
Updates manifests/immich/backup.yaml
Adds ExternalSecret immich-minio-creds pulling from Bitwarden item dcbcf704-7dce-48d7-bbd1-b3a801875b3d
CronJob now uses init container (immich-go) to stage files into emptyDir, then main container (rclone/rclone:latest) syncs staged files to s3:immich on nas1 MinIO (http://nas1.shire-zebra.ts.net:9000)
Removes NFS write dependency for the photo archive
Implements three backup improvements from issue #14:
## 1. AlbyHub Restic Backup (Hosea)
- Adds `services.restic.backups.albyhub` to `hosts/unstable/hosea/default.nix`
- Backs up `/chain/alby` to `rest:https://nas1.shire-zebra.ts.net:30248/albyhub`
- Stops AlbyHub before backup and restarts after for LDK state consistency
- Uses agenix secret `albyhub-restic-password` for repo password
- Prunes snapshots: 7 daily, 4 weekly, 3 monthly
- **Action required:** Run `agenix -e secrets/albyhub-restic-password.age` before deploying
## 2. Gitea Dump CronJob (Kubernetes)
- Adds `manifests/gitea/dump-cronjob.yaml`
- Runs daily at 2 AM (offset from 4 AM Immich backup)
- Mounts Gitea data PVC (read-only) and NFS share `nas1.shire-zebra.ts.net:/mnt/all/backups/gitea-dumps`
- Runs `gitea dump` as zip, then cleans up dumps older than 30 days
- Added to `manifests/gitea/kustomization.yaml`
## 3. Immich MinIO Sync (Kubernetes)
- Updates `manifests/immich/backup.yaml`
- Adds ExternalSecret `immich-minio-creds` pulling from Bitwarden item `dcbcf704-7dce-48d7-bbd1-b3a801875b3d`
- CronJob now uses init container (`immich-go`) to stage files into emptyDir, then main container (`rclone/rclone:latest`) syncs staged files to `s3:immich` on nas1 MinIO (`http://nas1.shire-zebra.ts.net:9000`)
- Removes NFS write dependency for the photo archive
Closes #14
✅ Stop/start wrapper for LDK consistency is the right call
✅rest:https:// with the Tailscale hostname is correct
⚠️ The backupPrepareCommand / backupCleanupCommand run as root — systemctl stop/start will work, but if the backup itself fails mid-run, backupCleanupCommand still fires (that is the intended NixOS restic behavior), so AlbyHub will come back up regardless. That is fine.
⚠️Action item before deploy:agenix -e secrets/albyhub-restic-password.age — noted in the PR body, just confirming I see it
Minor: RandomizedDelaySec = "1h" means the backup could run anywhere from 00:00–01:00. That is fine for daily pruning cadence.
Gitea dump CronJob
✅ Read-only PVC mount is the right safety choice
✅find -mtime +30 -delete cleanup is clean
⚠️ The image is pinned to gitea/gitea:1.25.4 — worth keeping this in sync with whatever version the Helm chart deploys, otherwise gitea dump -c /data/gitea/conf/app.ini may complain about config schema mismatches. Consider using the same image tag variable or a latest-tracking tag if you do not pin the chart version.
⚠️securityContext.runAsUser: 1000 — verify that uid 1000 in the gitea container has read access to /data/gitea/conf/app.ini and write access to the NFS share. The Gitea Helm chart typically runs as uid 1000 (git user), so this should be fine, but worth a quick check on the NFS export permissions.
Immich → MinIO sync
✅ ExternalSecret pulling from the same Bitwarden item as Gitea MinIO creds — clean reuse
✅ init container staging pattern avoids writing to NFS, good
⚠️rclone/rclone:latest — unpinned image in production. Not a blocker, but consider pinning to a digest or tag for reproducibility
⚠️ First run will pull all photos into the emptyDir stage volume before rclone syncs. Depending on your photo library size, the pod may hit default ephemeral storage limits. If Immich has a lot of data, you may want to add resources.limits.ephemeral-storage or use a larger PVC instead of emptyDir for staging.
ℹ️--s3-no-check-bucket skips bucket existence check — fine if the immich bucket already exists on nas1 MinIO.
Overall: LGTM with the caveats above. The agenix secret creation is the only hard blocker before deploying to Hosea. Everything else is advisory. Merging when you are ready. 🌌
Reviewed — looks solid. A few notes:
**AlbyHub Restic (hosea)**
- ✅ Stop/start wrapper for LDK consistency is the right call
- ✅ `rest:https://` with the Tailscale hostname is correct
- ⚠️ The `backupPrepareCommand` / `backupCleanupCommand` run as root — `systemctl stop/start` will work, but if the backup itself fails mid-run, `backupCleanupCommand` still fires (that is the intended NixOS restic behavior), so AlbyHub will come back up regardless. That is fine.
- ⚠️ **Action item before deploy:** `agenix -e secrets/albyhub-restic-password.age` — noted in the PR body, just confirming I see it
- Minor: `RandomizedDelaySec = "1h"` means the backup could run anywhere from 00:00–01:00. That is fine for daily pruning cadence.
**Gitea dump CronJob**
- ✅ Read-only PVC mount is the right safety choice
- ✅ `find -mtime +30 -delete` cleanup is clean
- ⚠️ The image is pinned to `gitea/gitea:1.25.4` — worth keeping this in sync with whatever version the Helm chart deploys, otherwise `gitea dump -c /data/gitea/conf/app.ini` may complain about config schema mismatches. Consider using the same image tag variable or a `latest`-tracking tag if you do not pin the chart version.
- ⚠️ `securityContext.runAsUser: 1000` — verify that uid 1000 in the gitea container has read access to `/data/gitea/conf/app.ini` and write access to the NFS share. The Gitea Helm chart typically runs as uid 1000 (git user), so this should be fine, but worth a quick check on the NFS export permissions.
**Immich → MinIO sync**
- ✅ ExternalSecret pulling from the same Bitwarden item as Gitea MinIO creds — clean reuse
- ✅ init container staging pattern avoids writing to NFS, good
- ⚠️ `rclone/rclone:latest` — unpinned image in production. Not a blocker, but consider pinning to a digest or tag for reproducibility
- ⚠️ First run will pull all photos into the `emptyDir` stage volume before rclone syncs. Depending on your photo library size, the pod may hit default ephemeral storage limits. If Immich has a lot of data, you may want to add `resources.limits.ephemeral-storage` or use a larger PVC instead of emptyDir for staging.
- ℹ️ `--s3-no-check-bucket` skips bucket existence check — fine if the `immich` bucket already exists on nas1 MinIO.
**Overall:** LGTM with the caveats above. The agenix secret creation is the only hard blocker before deploying to Hosea. Everything else is advisory. Merging when you are ready. 🌌
Closing in favor of PR #17, which is the stronger implementation. Key reasons:
Restic secrets:#17 reuses the existing restic-env.age / restic-pw.age secrets (no new secret to create before deploying). #16 required a brand-new albyhub-restic-password.age that does not exist yet — a hard blocker I flagged in my earlier review.
Gitea dump CronJob:#17 separates the dump into an init container and cleanup into a main container (cleaner separation of concerns), uses --skip-log --skip-custom-dir --skip-db flags for a lighter repo-only dump, and retains the 30 most recent dumps by count rather than by mtime (more predictable). Also includes successfulJobsHistoryLimit / failedJobsHistoryLimit.
Immich backup:#17 adds a proper 400Gi Longhorn PVC for staging (vs emptyDir in #16 which could hit ephemeral storage limits), bakes the rclone config into the Secret as rclone.conf (cleaner than passing credentials as env vars), pins rclone to 1.68 (vs latest), and adds --transfers 4 --checkers 8 for reasonable parallelism.
Overall #17 is more production-ready. Closing this one.
Closing in favor of PR #17, which is the stronger implementation. Key reasons:
**Restic secrets:** #17 reuses the existing `restic-env.age` / `restic-pw.age` secrets (no new secret to create before deploying). #16 required a brand-new `albyhub-restic-password.age` that does not exist yet — a hard blocker I flagged in my earlier review.
**Gitea dump CronJob:** #17 separates the dump into an init container and cleanup into a main container (cleaner separation of concerns), uses `--skip-log --skip-custom-dir --skip-db` flags for a lighter repo-only dump, and retains the 30 most recent dumps by count rather than by mtime (more predictable). Also includes `successfulJobsHistoryLimit` / `failedJobsHistoryLimit`.
**Immich backup:** #17 adds a proper 400Gi Longhorn PVC for staging (vs emptyDir in #16 which could hit ephemeral storage limits), bakes the rclone config into the Secret as `rclone.conf` (cleaner than passing credentials as env vars), pins rclone to `1.68` (vs `latest`), and adds `--transfers 4 --checkers 8` for reasonable parallelism.
Overall #17 is more production-ready. Closing this one.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Implements three backup improvements from issue #14:
1. AlbyHub Restic Backup (Hosea)
services.restic.backups.albyhubtohosts/unstable/hosea/default.nix/chain/albytorest:https://nas1.shire-zebra.ts.net:30248/albyhubalbyhub-restic-passwordfor repo passwordagenix -e secrets/albyhub-restic-password.agebefore deploying2. Gitea Dump CronJob (Kubernetes)
manifests/gitea/dump-cronjob.yamlnas1.shire-zebra.ts.net:/mnt/all/backups/gitea-dumpsgitea dumpas zip, then cleans up dumps older than 30 daysmanifests/gitea/kustomization.yaml3. Immich MinIO Sync (Kubernetes)
manifests/immich/backup.yamlimmich-minio-credspulling from Bitwarden itemdcbcf704-7dce-48d7-bbd1-b3a801875b3dimmich-go) to stage files into emptyDir, then main container (rclone/rclone:latest) syncs staged files tos3:immichon nas1 MinIO (http://nas1.shire-zebra.ts.net:9000)Closes #14
Reviewed — looks solid. A few notes:
AlbyHub Restic (hosea)
rest:https://with the Tailscale hostname is correctbackupPrepareCommand/backupCleanupCommandrun as root —systemctl stop/startwill work, but if the backup itself fails mid-run,backupCleanupCommandstill fires (that is the intended NixOS restic behavior), so AlbyHub will come back up regardless. That is fine.agenix -e secrets/albyhub-restic-password.age— noted in the PR body, just confirming I see itRandomizedDelaySec = "1h"means the backup could run anywhere from 00:00–01:00. That is fine for daily pruning cadence.Gitea dump CronJob
find -mtime +30 -deletecleanup is cleangitea/gitea:1.25.4— worth keeping this in sync with whatever version the Helm chart deploys, otherwisegitea dump -c /data/gitea/conf/app.inimay complain about config schema mismatches. Consider using the same image tag variable or alatest-tracking tag if you do not pin the chart version.securityContext.runAsUser: 1000— verify that uid 1000 in the gitea container has read access to/data/gitea/conf/app.iniand write access to the NFS share. The Gitea Helm chart typically runs as uid 1000 (git user), so this should be fine, but worth a quick check on the NFS export permissions.Immich → MinIO sync
rclone/rclone:latest— unpinned image in production. Not a blocker, but consider pinning to a digest or tag for reproducibilityemptyDirstage volume before rclone syncs. Depending on your photo library size, the pod may hit default ephemeral storage limits. If Immich has a lot of data, you may want to addresources.limits.ephemeral-storageor use a larger PVC instead of emptyDir for staging.--s3-no-check-bucketskips bucket existence check — fine if theimmichbucket already exists on nas1 MinIO.Overall: LGTM with the caveats above. The agenix secret creation is the only hard blocker before deploying to Hosea. Everything else is advisory. Merging when you are ready. 🌌
Closing in favor of PR #17, which is the stronger implementation. Key reasons:
Restic secrets: #17 reuses the existing
restic-env.age/restic-pw.agesecrets (no new secret to create before deploying). #16 required a brand-newalbyhub-restic-password.agethat does not exist yet — a hard blocker I flagged in my earlier review.Gitea dump CronJob: #17 separates the dump into an init container and cleanup into a main container (cleaner separation of concerns), uses
--skip-log --skip-custom-dir --skip-dbflags for a lighter repo-only dump, and retains the 30 most recent dumps by count rather than by mtime (more predictable). Also includessuccessfulJobsHistoryLimit/failedJobsHistoryLimit.Immich backup: #17 adds a proper 400Gi Longhorn PVC for staging (vs emptyDir in #16 which could hit ephemeral storage limits), bakes the rclone config into the Secret as
rclone.conf(cleaner than passing credentials as env vars), pins rclone to1.68(vslatest), and adds--transfers 4 --checkers 8for reasonable parallelism.Overall #17 is more production-ready. Closing this one.
Pull request closed