root
7fee882488
buildbot/nix-eval Build done.
Issue 1 - Grafana secret key:
- Replace hardcoded '123456789' secret_key in hosea Grafana config
with an agenix-managed secret ($__file interpolation)
- Add age.secrets.grafana-secret-key declaration to hosea config
- Register grafana-secret-key.age in secrets/secrets.nix (publicKeys=everyone)
- NOTE: The .age file still needs to be created by running:
agenix -e secrets/grafana-secret-key.age
and entering a strong random secret (e.g. from 'pwgen -s 64 1')
Issue 2 - Prometheus stale targets:
- Remove retired vm-gitlab host from all three scrape jobs
(node, systemd, ping) in manifests/monitoring/config.yaml
- linode was already present in all scrape jobs
Issue 3 - Prometheus alerting gaps:
- Wire up rule_files in prometheus.yml pointing at alerts.yml
- Add manifests/monitoring/alerts.yaml with alerting rules:
* HostDown (node unreachable > 2m) - critical
* HighCPULoad (>90% for 10m) - warning
* LowDiskSpace (<10% free) - warning
* CriticalDiskSpace (<5% free) - critical
* HighMemoryUsage (>90% for 10m) - warning
* ResticBackupStaleness (no success > 26h) - warning
* PodCrashLooping - warning
* PodNotReady (>10m) - warning
- Mount alerts ConfigMap as separate volume subPath in deployment
so prometheus.yml and alerts.yml are both accessible at
/etc/prometheus/ without conflicts
57 lines
1.5 KiB
YAML
57 lines
1.5 KiB
YAML
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: prometheus
|
|
labels:
|
|
app: prometheus
|
|
spec:
|
|
replicas: 1
|
|
strategy:
|
|
rollingUpdate:
|
|
maxSurge: 1
|
|
maxUnavailable: 1
|
|
type: RollingUpdate
|
|
selector:
|
|
matchLabels:
|
|
app: prometheus
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: prometheus
|
|
annotations:
|
|
prometheus.io/scrape: "true"
|
|
prometheus.io/port: "9090"
|
|
spec:
|
|
serviceAccountName: prometheus
|
|
containers:
|
|
- name: prometheus
|
|
image: prom/prometheus
|
|
args:
|
|
- "--storage.tsdb.path=/prometheus"
|
|
- "--config.file=/etc/prometheus/prometheus.yml"
|
|
ports:
|
|
- name: http
|
|
containerPort: 9090
|
|
volumeMounts:
|
|
- name: prometheus-config-values
|
|
mountPath: /etc/prometheus/prometheus.yml
|
|
subPath: prometheus.yml
|
|
- name: prometheus-alerts
|
|
mountPath: /etc/prometheus/alerts.yml
|
|
subPath: alerts.yml
|
|
- name: prometheus-storage-volume
|
|
mountPath: /prometheus
|
|
restartPolicy: Always
|
|
volumes:
|
|
- name: prometheus-config-values
|
|
configMap:
|
|
defaultMode: 420
|
|
name: prometheus-config
|
|
- name: prometheus-alerts
|
|
configMap:
|
|
defaultMode: 420
|
|
name: prometheus-alerts
|
|
- name: prometheus-storage-volume
|
|
persistentVolumeClaim:
|
|
claimName: prometheus-data
|