node-healthcheck: One Script, Three Exit Codes
Every host eventually gets a health script. Almost none of them can be trusted by another program. node-healthcheck is the script finished properly: eighteen checks, JSON on request, an exit code that means something, and a fleet mode that needs nothing installed on the far side.
What happened
The internal version was seventy-three lines of bash. It printed uptime, load, memory, disk, the addresses on each interface, whether two VPN tunnels were up, which peers answered a ping, and which services were running. Every host and address it cared about was written into the script. It was useful to a person reading a terminal and useless to anything else: it exited zero no matter what it found, and its output was prose.
That is the normal life cycle of a health script. It grows by accretion, it hard-codes the environment it was born in, and it is never wired into anything because nothing can consume it.
Why it mattered
A check that cannot fail is a log line, not a check. The moment a host script has a status contract (an exit code, a structured record per check, a numeric metric where one exists), it can sit in a cron job that alerts on non-zero, in a CI step that refuses to deploy to a sick host, in a wrapper that decides whether a maintenance window is safe. The same script that helped a person becomes a component.
The other cost of the hard-coded version was that it could not be published. A tool that names your hosts, your subnets, your tunnels, and your services is a map of your network. Rewriting it as a tool that takes those things as configuration is not just a scrub; it is the difference between a script and a product.
What it does
node-healthcheck is a single bash file. It runs eighteen checks: system, load, memory, swap, disk, inodes, network, gateway, dns, services, user_services, ports, peers, failed_units, time_sync, reboot_required, sessions, and zombies. Each check records a status (ok, warn, crit, info, or skip), a one-line summary, and numeric metrics.
$ node-healthcheck --services ssh,cron --ports 22 --peers 203.0.113.1 --quiet --no-color
node-healthcheck 1.0.0 - node-a - 2026-09-06T21:02:11Z
[WARN] swap 62.0% used (2540 MiB of 4096 MiB)
[CRIT] disk /data 91%
[WARN] reboot_required reboot required (linux-image-6.8.0-46-generic)
Overall: CRIT (exit 2)| Exit code | Meaning |
|---|---|
0 | every selected check is ok, info, or skip |
1 | at least one warning, no critical |
2 | at least one critical, or a host in fleet mode could not be reached |
3 | usage error, invalid threshold, unreadable config, unknown check |
Thresholds for load per core, memory, swap, disk, inodes, and zombies are flags or config-file keys. The things that must be true on a given host (which units must be active, which ports must listen, which peers must answer, which name must resolve) are declared the same way. --json turns the report into one document. --host runs the check on other machines over SSH and aggregates the results.
How it works
Status as a contract. All output goes through one function, record, which takes a check name, a status, a summary, and key=value metrics. The text renderer and the JSON renderer are two consumers of the same call, so they cannot disagree, and the overall exit code is derived from the worst status recorded. Adding a check means writing one function and adding its name to a list.
JSON without jq. Requiring a JSON tool would have made the script depend on something that is not on every host. Instead the script escapes strings itself (backslashes, quotes, newlines, tabs, and control characters) and emits numbers unquoted only when they match a strict numeric pattern. The test suite feeds it a hostname containing quotes, a backslash, and a tab and asserts that a real JSON parser reads the document back unchanged.
Configuration that cannot run code. The obvious way to read a shell config file is to source it, which means anyone who can write the file can execute commands as whoever runs the check. node-healthcheck parses key=value lines by hand, rejects unknown keys, and never evaluates a value. A test writes a command substitution into a config file and asserts that it did not run.
Fleet mode with nothing installed. --host user@node sends the script to the remote host on standard input and reads back its JSON. The remote side needs bash and a login. Flags and config-derived target lists are forwarded, output-mode flags apply to the aggregate, and a host that cannot be reached becomes a critical node record rather than a crash.
node-healthcheck --host admin@node-a --host admin@node-b --host admin@node-c --json \
| jq '.nodes[] | {host, status}'Tests without root. The check logic reads /proc through an overridable root and probes the system through commands on PATH. The test suite builds a fake proc tree and shims df, ip, ping, systemctl, ss, getent, timedatectl, who, and ssh, then drives ninety-four assertions through the real script: thresholds, list parsing, config handling, escaping, text-mode controls, and multi-node aggregation. The harness is plain bash; python3 is used only to parse JSON in assertions. CI runs shellcheck at style level, the suite, and a live smoke run on the runner.
What changed
The lab's health script became a public tool with a contract, and the lab now runs the public tool with a config file instead of the hard-coded original. Nothing about the original environment survived the rewrite: no host, address, interface, tunnel, or service name appears in the repository, and the example configuration uses documentation address ranges.
Version 1.0.0 is on GitHub under AGPL-3.0 with a README, a STARTHERE bootstrap for coding assistants, an example config, an idempotent setup script that links the tool into a user bin directory, and a release workflow that publishes the script with a checksum.
What comes next
The obvious extensions are more checks (certificate expiry, SMART status, container runtimes) and a Prometheus text-format output. Each will be added only if it keeps the single-file, zero-dependency property. A check that needs a library belongs in a different tool.
Why open source
Host health is a solved problem for large fleets with agents and time-series databases, and an unsolved one for the single box, the two-node lab, and the small cluster where an agent per host is not worth it. This fills that gap with something small enough to read in one sitting. It sits beside service-cartographer, which maps what is installed on a host, and cooldown-guard, which meters how often recurring work may run, in the OpenForge catalog. The public source note lists the claims and the validation behind them.
Get started
curl -fsSL https://raw.githubusercontent.com/GreyforgeLabs/node-healthcheck/main/bin/node-healthcheck -o node-healthcheck
chmod +x node-healthcheck && ./node-healthcheck --jsonThe repository, the v1.0.0 release, and the STARTHERE guide are the public entry points.