next.thehellings.com (Nextcloud) was returning 503 from haproxy. Root cause: nginx.service was crash-looping and had hit systemd's start-limit-hit, so nothing was listening on 127.0.0.1:8080 (the port haproxy's next backend proxies to).
nginx's syslog sink only allows alphanumeric characters and underscores in the tag field - hyphens are rejected. This failed nginx's config test:
nginx: [emerg] syslog "tag" only allows alphanumeric characters and underscore in .../nginx.conf:114
nginx-pre-start failing meant nginx.service never started, which crash-looped until systemd gave up (start-limit-hit). haproxy, phpfpm-nextcloud, postgresql, and redis were all healthy throughout - this was purely an nginx config parse failure introduced by the syslog tag change.
Fix
Change the tag from nginx-access to nginx_access (underscore). Journalctl callers should use journalctl -t nginx_access going forward.
Verification
Diagnosed live via SSH as the emily monitoring account (journalctl -u nginx showed the exact syslog emerg line and timestamp of the config regression). Could not run nix flake check/nginx -t locally in the agent sandbox (no local Nix/nginx toolchain) - please confirm CI/garnix validates the build, and I'd recommend a manual systemctl restart nginx + curl check on next.thehellings.com after this merges since nginx is currently stuck in start-limit-hit and will need an explicit restart even after the config fix lands.
## What broke
next.thehellings.com (Nextcloud) was returning 503 from haproxy. Root cause: nginx.service was crash-looping and had hit systemd's start-limit-hit, so nothing was listening on 127.0.0.1:8080 (the port haproxy's `next` backend proxies to).
## Root cause
#37 (feat/emily-incident-logging) added:
```
access_log syslog:server=unix:/dev/log,tag=nginx-access combined;
```
nginx's syslog sink only allows alphanumeric characters and underscores in the `tag` field - hyphens are rejected. This failed nginx's config test:
```
nginx: [emerg] syslog "tag" only allows alphanumeric characters and underscore in .../nginx.conf:114
```
nginx-pre-start failing meant nginx.service never started, which crash-looped until systemd gave up (`start-limit-hit`). haproxy, phpfpm-nextcloud, postgresql, and redis were all healthy throughout - this was purely an nginx config parse failure introduced by the syslog tag change.
## Fix
Change the tag from `nginx-access` to `nginx_access` (underscore). Journalctl callers should use `journalctl -t nginx_access` going forward.
## Verification
Diagnosed live via SSH as the `emily` monitoring account (journalctl -u nginx showed the exact syslog emerg line and timestamp of the config regression). Could not run `nix flake check`/`nginx -t` locally in the agent sandbox (no local Nix/nginx toolchain) - please confirm CI/garnix validates the build, and I'd recommend a manual `systemctl restart nginx` + curl check on next.thehellings.com after this merges since nginx is currently stuck in start-limit-hit and will need an explicit restart even after the config fix lands.
The nginx_access syslog tag added in #37 (feat/emily-incident-logging)
used tag=nginx-access. nginx's syslog sink only accepts alphanumeric
characters and underscores in the tag field, so the generated
nginx.conf failed its config test on linode:
nginx: [emerg] syslog "tag" only allows alphanumeric characters
and underscore in .../nginx.conf:114
Because nginx-pre-start failed, nginx.service crash-looped until it
hit systemd's start-limit-hit and gave up entirely. Since Nextcloud is
proxied through nginx (127.0.0.1:8080, fronted by haproxy's 'next'
backend), this took next.thehellings.com down with a 503 from haproxy
(phpfpm-nextcloud/postgresql/redis backends were all healthy and
unaffected - purely an nginx config parse failure).
Fix: use an underscore (nginx_access) instead of a hyphen.
greg
approved these changes 2026-08-10 02:42:49 +00:00
greg
merged commit daa33daa0c into main2026-08-10 02:42:56 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
greg
What broke
next.thehellings.com (Nextcloud) was returning 503 from haproxy. Root cause: nginx.service was crash-looping and had hit systemd's start-limit-hit, so nothing was listening on 127.0.0.1:8080 (the port haproxy's
nextbackend proxies to).Root cause
#37 (feat/emily-incident-logging) added:
nginx's syslog sink only allows alphanumeric characters and underscores in the
tagfield - hyphens are rejected. This failed nginx's config test:nginx-pre-start failing meant nginx.service never started, which crash-looped until systemd gave up (
start-limit-hit). haproxy, phpfpm-nextcloud, postgresql, and redis were all healthy throughout - this was purely an nginx config parse failure introduced by the syslog tag change.Fix
Change the tag from
nginx-accesstonginx_access(underscore). Journalctl callers should usejournalctl -t nginx_accessgoing forward.Verification
Diagnosed live via SSH as the
emilymonitoring account (journalctl -u nginx showed the exact syslog emerg line and timestamp of the config regression). Could not runnix flake check/nginx -tlocally in the agent sandbox (no local Nix/nginx toolchain) - please confirm CI/garnix validates the build, and I'd recommend a manualsystemctl restart nginx+ curl check on next.thehellings.com after this merges since nginx is currently stuck in start-limit-hit and will need an explicit restart even after the config fix lands.