Adds services.restic.backups.albyhub to hosts/unstable/hosea/default.nix
Backs up /chain/alby (LDK node data: log + nwc.db) using existing restic-env.age / restic-pw.age secrets
Service is stopped before snapshot, restarted after (LDK state must not be captured live)
Runs at 02:30 daily with 7-daily / 4-weekly / 12-monthly retention
2. Gitea dump CronJob
New manifests/gitea/dump-cronjob.yaml
Runs gitea dump at 3am daily, writing a timestamped .zip to NFS at nas1:/mnt/all/backups/gitea-dumps/
Cleanup container keeps the 30 most recent dumps
Added to manifests/gitea/kustomization.yaml
3. Immich backup → MinIO
Replaces NFS-based backup with rclone sync to MinIO immich bucket on nas1
New ExternalSecret (immich-minio-creds) pulls MinIO credentials from Bitwarden item dcbcf704-7dce-48d7-bbd1-b3a801875b3d (same item as Gitea's MinIO config)
New 400Gi staging PVC for immich-go to write to before rclone syncs
rclone uses --checksum for true deduplication — only new/changed files transferred
Existing 9TB NFS backup untouched; Greg will clean it up manually once MinIO backup is confirmed working
Action required after merge
Run colmena apply --on hosea to deploy the AlbyHub Restic config
Ensure NFS path /mnt/all/backups/gitea-dumps exists on nas1
The immich-backup-staging PVC will be created automatically by Kubernetes
Closes #14
## Changes
### 1. AlbyHub Restic backup (Hosea)
- Adds `services.restic.backups.albyhub` to `hosts/unstable/hosea/default.nix`
- Backs up `/chain/alby` (LDK node data: `log` + `nwc.db`) using existing `restic-env.age` / `restic-pw.age` secrets
- Service is stopped before snapshot, restarted after (LDK state must not be captured live)
- Runs at 02:30 daily with 7-daily / 4-weekly / 12-monthly retention
### 2. Gitea dump CronJob
- New `manifests/gitea/dump-cronjob.yaml`
- Runs `gitea dump` at 3am daily, writing a timestamped `.zip` to NFS at `nas1:/mnt/all/backups/gitea-dumps/`
- Cleanup container keeps the 30 most recent dumps
- Added to `manifests/gitea/kustomization.yaml`
### 3. Immich backup → MinIO
- Replaces NFS-based backup with rclone sync to MinIO `immich` bucket on nas1
- New `ExternalSecret` (`immich-minio-creds`) pulls MinIO credentials from Bitwarden item `dcbcf704-7dce-48d7-bbd1-b3a801875b3d` (same item as Gitea's MinIO config)
- New 400Gi staging PVC for immich-go to write to before rclone syncs
- rclone uses `--checksum` for true deduplication — only new/changed files transferred
- Existing 9TB NFS backup untouched; Greg will clean it up manually once MinIO backup is confirmed working
## Action required after merge
- Run `colmena apply --on hosea` to deploy the AlbyHub Restic config
- Ensure NFS path `/mnt/all/backups/gitea-dumps` exists on nas1
- The immich-backup-staging PVC will be created automatically by Kubernetes
successfulJobsHistoryLimit / failedJobsHistoryLimit set on both CronJobs ✅
One thing to verify before deploying:
The PVC claim name in dump-cronjob.yaml is gitea-shared-storage — confirm that matches what the Gitea Helm chart actually creates (the other PR used gitea-release-gitea). Check with kubectl get pvc -n gitea.
Action required after merge:
colmena apply --on hosea for the AlbyHub Restic config
Ensure nas1:/mnt/all/backups/gitea-dumps exists (create if not)
The immich-backup-staging PVC will be created automatically
LGTM otherwise — merging when you give the word. 🌌
Reviewed both PRs — closing #16 in favor of this one. This is the stronger implementation.
**What makes this better than #16:**
- Reuses existing `restic-env.age` / `restic-pw.age` — no new secret to create before deploying ✅
- 400Gi Longhorn PVC for Immich staging instead of `emptyDir` (no ephemeral storage limit risk) ✅
- rclone pinned to `1.68` (not `latest`) ✅
- rclone config baked into the Secret as `rclone.conf` (cleaner than env-var credential injection) ✅
- Gitea dump uses init+cleanup container split, lighter `--skip-log --skip-custom-dir --skip-db` flags, count-based retention ✅
- `successfulJobsHistoryLimit` / `failedJobsHistoryLimit` set on both CronJobs ✅
**One thing to verify before deploying:**
- The PVC claim name in `dump-cronjob.yaml` is `gitea-shared-storage` — confirm that matches what the Gitea Helm chart actually creates (the other PR used `gitea-release-gitea`). Check with `kubectl get pvc -n gitea`.
**Action required after merge:**
- `colmena apply --on hosea` for the AlbyHub Restic config
- Ensure `nas1:/mnt/all/backups/gitea-dumps` exists (create if not)
- The `immich-backup-staging` PVC will be created automatically
LGTM otherwise — merging when you give the word. 🌌
All of the changes in this file can be simplified to use the greg.backup module. It has support for pre and post commands, includes configured pruneOpts that I like, and configures other settings that I need for accessing my Restic server, such as disabling TLS checking.
All of the changes in this file can be simplified to use the `greg.backup` module. It has support for pre and post commands, includes configured pruneOpts that I like, and configures other settings that I need for accessing my Restic server, such as disabling TLS checking.
Let's change this to use the S3 backups as well, where we can set the items with a 30 day expiration time. I've created the bucket backup-gitea for holding them. No need for the NFS loading.
Let's change this to use the S3 backups as well, where we can set the items with a 30 day expiration time. I've created the bucket `backup-gitea` for holding them. No need for the NFS loading.
- manifests/gitea/dump-cronjob.yaml: Remove --skip-log, --skip-custom-dir,
--skip-db flags to make backup complete
- manifests/gitea/dump-cronjob.yaml: Replace NFS volume + cleanup container
with S3 upload to backup-gitea bucket using MinIO client (mc).
30-day lifecycle set via mc ilm. Uses minio_key/minio_secret from
existing gitea-config secret.
- hosts/unstable/hosea/default.nix: Replace raw services.restic.backups.albyhub
block with greg.backup.jobs.albyhub using the greg.backup module.
Remove manual age.secrets.restic-env and age.secrets.restic-pw entries
since the greg.backup module declares them.
1. Gitea dump flags removed (manifests/gitea/dump-cronjob.yaml)
Removed --skip-log, --skip-custom-dir, and --skip-db from the gitea dump command. The dump is now complete.
2. NFS → S3 via MinIO (manifests/gitea/dump-cronjob.yaml)
Replaced the NFS volume and busybox cleanup container with an upload-to-s3 container using minio/mc. After the dump, it:
Configures an mc alias pointing at nas1.shire-zebra.ts.net:9000
Uploads the dump zip to the backup-gitea bucket
Sets a 30-day lifecycle expiration via mc ilm rule add --expire-days 30 (idempotent)
Credentials are pulled from the existing gitea-config secret (keys minio_key / minio_secret), which is already populated by the ExternalSecret in manifests/gitea/secrets.yaml.
3. greg.backup module (hosts/unstable/hosea/default.nix)
Replaced the raw services.restic.backups.albyhub block with:
Also removed the manual age.secrets.restic-pw and age.secrets.restic-env entries since the greg.backup module declares them.
Addressed all three review items:
**1. Gitea dump flags removed** (`manifests/gitea/dump-cronjob.yaml`)
Removed `--skip-log`, `--skip-custom-dir`, and `--skip-db` from the `gitea dump` command. The dump is now complete.
**2. NFS → S3 via MinIO** (`manifests/gitea/dump-cronjob.yaml`)
Replaced the NFS volume and busybox cleanup container with an `upload-to-s3` container using `minio/mc`. After the dump, it:
- Configures an `mc` alias pointing at `nas1.shire-zebra.ts.net:9000`
- Uploads the dump zip to the `backup-gitea` bucket
- Sets a 30-day lifecycle expiration via `mc ilm rule add --expire-days 30` (idempotent)
Credentials are pulled from the existing `gitea-config` secret (keys `minio_key` / `minio_secret`), which is already populated by the ExternalSecret in `manifests/gitea/secrets.yaml`.
**3. greg.backup module** (`hosts/unstable/hosea/default.nix`)
Replaced the raw `services.restic.backups.albyhub` block with:
```nix
greg.backup.jobs.albyhub = {
src = "/chain/alby";
dest = "albyhub";
pre = "systemctl stop albyhub || true";
post = "systemctl start albyhub";
};
```
Also removed the manual `age.secrets.restic-pw` and `age.secrets.restic-env` entries since the `greg.backup` module declares them.
greg
approved these changes 2026-04-04 17:59:47 +00:00
greg
merged commit 828d61d818 into main2026-04-04 18:00:19 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
greg
Closes #14
Changes
1. AlbyHub Restic backup (Hosea)
services.restic.backups.albyhubtohosts/unstable/hosea/default.nix/chain/alby(LDK node data:log+nwc.db) using existingrestic-env.age/restic-pw.agesecrets2. Gitea dump CronJob
manifests/gitea/dump-cronjob.yamlgitea dumpat 3am daily, writing a timestamped.zipto NFS atnas1:/mnt/all/backups/gitea-dumps/manifests/gitea/kustomization.yaml3. Immich backup → MinIO
immichbucket on nas1ExternalSecret(immich-minio-creds) pulls MinIO credentials from Bitwarden itemdcbcf704-7dce-48d7-bbd1-b3a801875b3d(same item as Gitea's MinIO config)--checksumfor true deduplication — only new/changed files transferredAction required after merge
colmena apply --on hoseato deploy the AlbyHub Restic config/mnt/all/backups/gitea-dumpsexists on nas1Reviewed both PRs — closing #16 in favor of this one. This is the stronger implementation.
What makes this better than #16:
restic-env.age/restic-pw.age— no new secret to create before deploying ✅emptyDir(no ephemeral storage limit risk) ✅1.68(notlatest) ✅rclone.conf(cleaner than env-var credential injection) ✅--skip-log --skip-custom-dir --skip-dbflags, count-based retention ✅successfulJobsHistoryLimit/failedJobsHistoryLimitset on both CronJobs ✅One thing to verify before deploying:
dump-cronjob.yamlisgitea-shared-storage— confirm that matches what the Gitea Helm chart actually creates (the other PR usedgitea-release-gitea). Check withkubectl get pvc -n gitea.Action required after merge:
colmena apply --on hoseafor the AlbyHub Restic confignas1:/mnt/all/backups/gitea-dumpsexists (create if not)immich-backup-stagingPVC will be created automaticallyLGTM otherwise — merging when you give the word. 🌌
@@ -145,6 +145,24 @@ in};};prometheus.exporters.graphite.enable = true;restic.backups.albyhub = {All of the changes in this file can be simplified to use the
greg.backupmodule. It has support for pre and post commands, includes configured pruneOpts that I like, and configures other settings that I need for accessing my Restic server, such as disabling TLS checking.@@ -0,0 +39,4 @@--type zip \--skip-log \--skip-custom-dir \--skip-dbThere is no need to skip items. Let's make the backup complete.
@@ -0,0 +54,4 @@- /bin/sh- "-c"- |ls -t /dump-output/gitea-dump-*.zip 2>/dev/null | tail -n +31 | xargs rm -fLet's change this to use the S3 backups as well, where we can set the items with a 30 day expiration time. I've created the bucket
backup-giteafor holding them. No need for the NFS loading.Addressed all three review items:
1. Gitea dump flags removed (
manifests/gitea/dump-cronjob.yaml)Removed
--skip-log,--skip-custom-dir, and--skip-dbfrom thegitea dumpcommand. The dump is now complete.2. NFS → S3 via MinIO (
manifests/gitea/dump-cronjob.yaml)Replaced the NFS volume and busybox cleanup container with an
upload-to-s3container usingminio/mc. After the dump, it:mcalias pointing atnas1.shire-zebra.ts.net:9000backup-giteabucketmc ilm rule add --expire-days 30(idempotent)Credentials are pulled from the existing
gitea-configsecret (keysminio_key/minio_secret), which is already populated by the ExternalSecret inmanifests/gitea/secrets.yaml.3. greg.backup module (
hosts/unstable/hosea/default.nix)Replaced the raw
services.restic.backups.albyhubblock with:Also removed the manual
age.secrets.restic-pwandage.secrets.restic-enventries since thegreg.backupmodule declares them.