Prometheus runs as a pod and only has Tailscale routes for the k8s nodes it runs on (isaiah, jeremiah, zeke) and hosea. Scraping genesis, exodus, and linode via *.shire-zebra.ts.net fails instantly (~1ms, connection refused) because those Tailscale routes do not exist inside the cluster network namespace.
This was visible in the metrics: working hosts showed 100-230ms scrape durations (real network round-trip); failing hosts showed ~1ms (instant refusal).
Fix
Host
Old
New
Reason
genesis
genesis.shire-zebra.ts.net
10.42.1.5
LAN host, no k8s, LAN IP reachable
zeke
zeke.shire-zebra.ts.net
keep
k8s node, Tailscale route exists
exodus
exodus.shire-zebra.ts.net
100.80.99.48
Laptop, no fixed LAN IP, use TS IP directly
linode
linode.shire-zebra.ts.net
100.109.86.8
VPS, no LAN IP, use TS IP directly
Also fixes kea and dnsmasq scrape jobs (genesis-only) to use the LAN IP.
Opened by Klaatu 🌌
## Root cause
Prometheus runs as a pod and only has Tailscale routes for the k8s nodes it runs on (isaiah, jeremiah, zeke) and hosea. Scraping genesis, exodus, and linode via `*.shire-zebra.ts.net` fails instantly (~1ms, connection refused) because those Tailscale routes do not exist inside the cluster network namespace.
This was visible in the metrics: working hosts showed 100-230ms scrape durations (real network round-trip); failing hosts showed ~1ms (instant refusal).
## Fix
| Host | Old | New | Reason |
|------|-----|-----|--------|
| genesis | `genesis.shire-zebra.ts.net` | `10.42.1.5` | LAN host, no k8s, LAN IP reachable |
| zeke | `zeke.shire-zebra.ts.net` | keep | k8s node, Tailscale route exists |
| exodus | `exodus.shire-zebra.ts.net` | `100.80.99.48` | Laptop, no fixed LAN IP, use TS IP directly |
| linode | `linode.shire-zebra.ts.net` | `100.109.86.8` | VPS, no LAN IP, use TS IP directly |
Also fixes kea and dnsmasq scrape jobs (genesis-only) to use the LAN IP.
> Opened by Klaatu 🌌
Prometheus runs as a pod and only has Tailscale routes to the three
k8s nodes it runs on (isaiah, jeremiah, zeke) plus hosea. Scraping
genesis, exodus, and linode via *.shire-zebra.ts.net fails instantly
(~1ms, connection refused) because those Tailscale routes don't exist
inside the cluster network.
Fix by host type:
- k8s nodes (isaiah, jeremiah, zeke, hosea): keep Tailscale hostnames
- LAN hosts (genesis): use LAN IP 10.42.1.5
- Roaming/VPS (exodus, linode): use Tailscale IPs directly (100.x.x.x)
since they have no fixed LAN address but the k8s nodes do route
to other Tailscale IPs via the subnet router
Apply same logic to kea, dnsmasq (genesis-only jobs) and the
systemd/ping exporter jobs.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Root cause
Prometheus runs as a pod and only has Tailscale routes for the k8s nodes it runs on (isaiah, jeremiah, zeke) and hosea. Scraping genesis, exodus, and linode via
*.shire-zebra.ts.netfails instantly (~1ms, connection refused) because those Tailscale routes do not exist inside the cluster network namespace.This was visible in the metrics: working hosts showed 100-230ms scrape durations (real network round-trip); failing hosts showed ~1ms (instant refusal).
Fix
genesis.shire-zebra.ts.net10.42.1.5zeke.shire-zebra.ts.netexodus.shire-zebra.ts.net100.80.99.48linode.shire-zebra.ts.net100.109.86.8Also fixes kea and dnsmasq scrape jobs (genesis-only) to use the LAN IP.
Pull request closed