Skip to main content

​
Why health is separate from deploy status

A component’s deploy status answers “did the last apply succeed?”. That is a statement about the past. Health answers “is it running right now?”. Those two diverge constantly. Terraform and Helm skip when state already matches, so a component can report Active while its pods are stuck. helm upgrade can exit 0 while the release sits in pending-install and nothing serves traffic. A component fixed by hand out of band keeps showing yesterday’s error. Health is therefore a separate axis. Nuon never overwrites deploy status with health or the other way round, and it never guesses: if we cannot see a component, its health is unknown, never healthy and never unhealthy.

​
Verdicts

A component that has never deployed reports not-applicable and shows no resources — probes and pushed checks describe a workload, and until one exists there is nothing for them to describe. A rollout that cannot converge is degraded, not progressing. If a pod is stuck pulling a bad image tag, crash-looping, or unschedulable, that surfaces on its workload immediately rather than waiting out the Kubernetes progress deadline. Verdicts are debounced: it takes three consecutive bad observations to flip a component bad and two good ones to bring it back, so a single restart or a controller backing off does not page anyone. Observations stop arriving for five minutes and the verdict becomes unknown, which is visually distinct from a failure and never alerts. Nuon also computes an install-level rollup, so an install reads “all components healthy” or “2 components unhealthy” at a glance.

​
What gets checked, with no configuration

Helm chart and Kubernetes manifest components are checked automatically. The runner lists the resources each component owns roughly every minute, assesses each one, and reports. Deployments, StatefulSets, DaemonSets, Jobs, PVCs, Ingresses and Pods are all covered, plus the Helm release’s own status. Controller-side failures that never appear in an object’s own status — an Ingress whose load balancer rejected a certificate, for example — are picked up from Kubernetes warning events. Terraform components have their cloud resources listed so you can see what a component owns. Those rows are inventory, not assessment, so they do not affect the verdict. Everything the runner sees is browsable per install under Resources, filterable by component, kind, namespace and health.

​
Tuning it

Configuration is optional. It only tunes or extends what already happens.

​
Probes

Probes assert that your app is actually serving, above the level of infrastructure. A Deployment can be perfectly ready while the thing inside it returns 500.
An http probe passes on 2xx/3xx and never follows redirects. A tcp probe passes if the port accepts a connection. An exec probe passes on exit code 0 and reports the command’s output when it fails; the command is an argv, never a shell string, and runs with a minimal environment (PATH, HOME, TMPDIR) — it does not inherit the runner’s credentials. Every probe is bounded by a short timeout, and probes run once per report cycle. Probe results are ordinary health observations: they feed the same verdict, the same debounce, the same alerts and the same history as everything else.

​
Custom checks

Probes cover what the runner can reach. For anything else — a business metric, a queue depth, a result your CI already computed, a Datadog monitor — push the check to Nuon and it becomes part of the component’s health:
status is one of healthy, degraded, unhealthy, unknown. Check names are 1-100 characters of letters, digits, dots, dashes, or underscores. A custom check can make a component worse but never better — reporting healthy will not paper over crash-looping pods. Checks are merged in on the runner’s report cadence, so the normal debounce applies to them unchanged. A pushed check’s last report stands for 5 minutes by default. If yours reports less often than that, say so with stale_after (up to 60m) or it will read as unknown between reports:
Past its window the check reads unknown rather than keeping its last verdict — it stays visible and counted, but stops voting, because a stale answer is not an answer. unknown never alerts and never outranks a check that did report. Custom checks work on any component type, including Terraform components that have no Kubernetes footprint.

​
Alerting

Health transitions fan out through webhooks and Slack like any other Nuon event. Subscribe with a single per-resource flag:
  • component_health on components — delivers both directions, so a channel that hears about a failure always hears the recovery.
  • install_degraded on installs — the install-level rollup crossing.
Two things are deliberately quiet. A verdict of unknown never alerts, because a runner going offline is already reported as runner inactivity and would otherwise page you once per component. And when a component fails because something it depends on failed, the dependents are labelled downstream of <component> and only the root cause alerts — one outage, one alert. See Webhooks for the full subscription model.

​
Verified deploys

By default a deploy finishes when the apply succeeds. Set block_deploy and it finishes only when the component has held healthy for stabilization_window afterwards:
The gate appears as its own verify health step on the deploy workflow, narrating as it waits — how much of the window is left, what it is watching, and which check is holding things up. It judges the runner’s raw observations directly: everything observed inside the window is post-apply evidence, so a failing observation fails the step immediately, and an all-healthy window passes the moment the window ends. The wait is measured from the moment the apply finished, so a component that was already healthy beforehand still has to prove it survived the change — and a deploy that fixes an unhealthy component passes as soon as its own observations come back healthy. This is off by default. Turning it on is the only way health can affect whether a deploy passes, and a component Nuon cannot observe never blocks a deploy — absence of data is not treated as failure. Gate probes must assert what the component itself provides. A probe that checks the public endpoint belongs on the load-balancer component that exposes it — not on the app behind it. If a component’s gate asserts something a downstream component provides, a first install deadlocks: the app’s gate can never pass before the load balancer deploys, and the load balancer (which depends on the app) never deploys because the gate is holding it back.

​
Gating a rollout: canary and bake periods

The pieces above compose into progressive delivery. The primitive is a fleet health summary, scoped by install label:
The CLI wraps the same thing, which is usually easier to drop into CI:
A rollout then looks like: deploy to the installs labelled tier=canary, poll until all_healthy is true and has stayed true for your bake period, and only then continue to the rest of the fleet.
all_healthy is never true unless at least one install was actually evaluated, and installs whose health has never been evaluated are counted separately in unset. A rollout must never read “no data” as a pass, so the endpoint will not let it. Combine this with custom checks and the gate becomes whatever you need it to be — error rate, latency, or a business metric your own systems already track.

​
Proving it stayed up

Every debounced transition is recorded for 90 days, which gives each component and install a health timeline and an uptime percentage, rendered as status-page-style daily bars in the dashboard. Time spent in unknown is excluded from uptime rather than counted as either up or down: we will not claim availability we did not observe. An install with no observations reports zero observed time instead of a misleading 100%. There is also an incident bundle per component, pulling together the failing transition, the captured diagnosis (Kubernetes events, restart counts, termination reasons such as OOMKilled) and the deploy it followed — useful as input to a runbook or an agent.

​
Enabling it

Component health is currently behind the component-health org feature. Ask Nuon to switch it on for your organization.