Removed vm-gitlab.shire-zebra.ts.net from the node, systemd, and ping scrape jobs. That VM is gone and was generating constant scrape errors. linode was already present in all jobs.
Issue 3 — Prometheus had no alerting rules
Added manifests/monitoring/alerts.yaml with a new prometheus-alerts ConfigMap containing rules for:
Backup health: ResticBackupStaleness (alerts if a backup hasn't succeeded in >26h)
Kubernetes: PodCrashLooping, PodNotReady
Wired rule_files into prometheus.yml and updated the Prometheus deployment to mount the alerts ConfigMap as a separate volume subPath (previously the whole ConfigMap was mounted as a directory, which would have shadowed individual files — fixed that too).
Opened by Klaatu 🌌
## Summary
Three security and monitoring fixes identified during infrastructure audit.
---
### Issue 1 — Grafana secret key hardcoded as `123456789`
Replaced with an agenix-managed secret using Grafana's `$__file{}` interpolation. The `.age` file needs to be created before deploying:
```bash
pwgen -s 64 1 # generate the secret
agenix -e secrets/grafana-secret-key.age
```
All existing Grafana sessions will be invalidated on next deploy (users will need to log in again — expected behavior).
---
### Issue 2 — Prometheus scraping retired `vm-gitlab` host
Removed `vm-gitlab.shire-zebra.ts.net` from the `node`, `systemd`, and `ping` scrape jobs. That VM is gone and was generating constant scrape errors. `linode` was already present in all jobs.
---
### Issue 3 — Prometheus had no alerting rules
Added `manifests/monitoring/alerts.yaml` with a new `prometheus-alerts` ConfigMap containing rules for:
- **Host health:** HostDown, HighCPULoad, LowDiskSpace, CriticalDiskSpace, HighMemoryUsage
- **Backup health:** ResticBackupStaleness (alerts if a backup hasn't succeeded in >26h)
- **Kubernetes:** PodCrashLooping, PodNotReady
Wired `rule_files` into `prometheus.yml` and updated the Prometheus deployment to mount the alerts ConfigMap as a separate volume subPath (previously the whole ConfigMap was mounted as a directory, which would have shadowed individual files — fixed that too).
---
> Opened by Klaatu 🌌
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
agenix -e secrets/grafana-secret-key.age
and entering a strong random secret (e.g. from 'pwgen -s 64 1')
Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
(node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs
Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
* HostDown (node unreachable > 2m) - critical
* HighCPULoad (>90% for 10m) - warning
* LowDiskSpace (<10% free) - warning
* CriticalDiskSpace (<5% free) - critical
* HighMemoryUsage (>90% for 10m) - warning
* ResticBackupStaleness (no success > 26h) - warning
* PodCrashLooping - warning
* PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
so prometheus.yml and alerts.yml are both accessible at
/etc/prometheus/ without conflicts
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Three security and monitoring fixes identified during infrastructure audit.
Issue 1 — Grafana secret key hardcoded as
123456789Replaced with an agenix-managed secret using Grafana's
$__file{}interpolation. The.agefile needs to be created before deploying:All existing Grafana sessions will be invalidated on next deploy (users will need to log in again — expected behavior).
Issue 2 — Prometheus scraping retired
vm-gitlabhostRemoved
vm-gitlab.shire-zebra.ts.netfrom thenode,systemd, andpingscrape jobs. That VM is gone and was generating constant scrape errors.linodewas already present in all jobs.Issue 3 — Prometheus had no alerting rules
Added
manifests/monitoring/alerts.yamlwith a newprometheus-alertsConfigMap containing rules for:Wired
rule_filesintoprometheus.ymland updated the Prometheus deployment to mount the alerts ConfigMap as a separate volume subPath (previously the whole ConfigMap was mounted as a directory, which would have shadowed individual files — fixed that too).Issue 1 - Grafana secret key: - Replace hardcoded '123456789' secret_key in hosea Grafana config with an agenix-managed secret ($__file interpolation) - Add age.secrets.grafana-secret-key declaration to hosea config - Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone) - NOTE: The .age file still needs to be created by running: agenix -e secrets/grafana-secret-key.age and entering a strong random secret (e.g. from 'pwgen -s 64 1') Issue 2 - Prometheus stale targets: - Remove retired vm-gitlab host from all three scrape jobs (node, systemd, ping) in manifests/monitoring/config.yaml - linode was already present in all scrape jobs Issue 3 - Prometheus alerting gaps: - Wire up rule_files in prometheus.yml pointing at alerts.yml - Add manifests/monitoring/alerts.yaml with alerting rules: * HostDown (node unreachable > 2m) - critical * HighCPULoad (>90% for 10m) - warning * LowDiskSpace (<10% free) - warning * CriticalDiskSpace (<5% free) - critical * HighMemoryUsage (>90% for 10m) - warning * ResticBackupStaleness (no success > 26h) - warning * PodCrashLooping - warning * PodNotReady (>10m) - warning - Mount alerts ConfigMap as separate volume subPath in deployment so prometheus.yml and alerts.yml are both accessible at /etc/prometheus/ without conflicts