Machine Health Check

A Machine Health Check automatically detects unhealthy worker nodes in your cluster and replaces them. Instead of waiting for someone to notice a broken node and fix it by hand, the platform deletes the faulty machine and the machine deployment creates a new one with the same configuration (flavor, availability zone, root disk, labels, and taints).

Machine Health Checks are configured per machine deployment and only apply to worker nodes. The control plane is monitored and repaired by us separately.

How a Machine Health Check Works

A Machine Health Check is active for every machine deployment. It does not need to be enabled.

Detection: Without further configuration, a machine is considered unhealthy if:

A newly created machine does not join the cluster as a node within 10 minutes, for example because the OpenStack instance cannot be created or the node fails to register with the cluster. The 10-minute timeout prevents nodes that are slower than usual during startup from being replaced unnecessarily.
The node of the machine has been deleted from the cluster.

Both checks are always active and cannot be changed or disabled. To also replace nodes that become unhealthy later during operation, for example when a node reports Ready=Unknown, you can add your own checks (see Custom Configuration).

Remediation: When a machine is marked as unhealthy, it is deleted and the machine deployment creates a replacement machine. The new node joins the cluster and becomes available for scheduling. Pods that were running on the unhealthy node are rescheduled by Kubernetes onto the remaining healthy nodes.

Safeguard: By default, only one machine at a time is replaced per machine deployment. If many nodes become unhealthy at once, for example during a short network disruption, this prevents all machines from being recreated at the same time.

Default Configuration

Without further settings, the following remediation configuration is applied to every machine deployment. No additional checks are configured:

workers:
- name: <machine-deployment-name>
  healthCheck:
    remediation:
      maxInFlight: 1

Custom Configuration

You can add your own checks and change the remediation settings. Your checks are added to the checks described in How a Machine Health Check Works; they never replace or disable them. Your remediation settings replace the default maxInFlight: 1 (see Combining Custom Settings with the Defaults). To do so, please contact us using the instructions provided here.

When reaching out, include the following information:

WIIT Resource Name
The name(s) of the machine deployment(s) the settings should apply to
The checks and remediation settings you need

Choose the timeouts with care: values that are too short can cause nodes to be replaced during short, self-recovering disruptions, while values that are too long leave broken nodes in the cluster for longer.

Checks

The checks block defines additional conditions under which a machine is considered unhealthy. Only the conditions you list are checked; no other conditions are added. The checks described in How a Machine Health Check Works stay active in any case:

Option	Controls
`unhealthyNodeConditions`	List of conditions on the Kubernetes node, for example `Ready`, `MemoryPressure`, `DiskPressure`, or `PIDPressure`.
`unhealthyMachineConditions`	List of conditions on the machine, for example `NodeHealthy` or `InfrastructureReady`.

Each condition consists of a type, a status (True, False, or Unknown), and a timeoutSeconds. A machine is considered unhealthy as soon as any of the conditions has been in the given status for at least its timeout. Each condition has its own timeout, so you can, for example, react faster to Ready=Unknown than to DiskPressure=True.

Available Node Conditions

These conditions are reported by the kubelet on each node. You can see them with kubectl describe node <node-name>.

Type	Healthy status	Unhealthy status	Meaning
`Ready`	`True`	`False`, `Unknown`	The node is healthy and ready to accept pods. `Unknown` means the node stopped reporting its status, for example because the kubelet is down or the node lost its network connection.
`MemoryPressure`	`False`	`True`	The node is running low on memory.
`DiskPressure`	`False`	`True`	The node is running low on disk space.
`PIDPressure`	`False`	`True`	Too many processes are running on the node.
`NetworkUnavailable`	`False`	`True`	The network of the node is not configured correctly.

Available Machine Conditions

These conditions are reported by the platform for each machine. They are not visible from inside your cluster.

Type	Healthy status	Unhealthy status	Meaning
`InfrastructureReady`	`True`	`False`	The OpenStack instance of the machine was created successfully and is running. `False` means, for example, that the instance went into an error state. This condition can detect problems before the node joins the cluster.
`BootstrapConfigReady`	`True`	`False`	The configuration used to set up the machine as a Kubernetes node was generated successfully.
`NodeReady`	`True`	`False`, `Unknown`	The node of the machine is ready. This mirrors the node’s `Ready` condition.
`NodeHealthy`	`True`	`False`, `Unknown`	The node of the machine is ready and reports no `MemoryPressure`, `DiskPressure`, or `PIDPressure`.

Other machine conditions, such as Ready, Available, or UpToDate, describe the lifecycle of a machine rather than its health. For example, UpToDate is False during every Node Rotation. Do not use them for health checks, as this can cause healthy machines to be replaced.

Example: Replace nodes that are not ready for 5 minutes, or that report disk pressure for 30 minutes:

workers:
- name: <machine-deployment-name>
  healthCheck:
    checks:
      unhealthyNodeConditions:
      - type: Ready
        status: "False"
        timeoutSeconds: 300
      - type: Ready
        status: Unknown
        timeoutSeconds: 300
      - type: DiskPressure
        status: "True"
        timeoutSeconds: 1800

Remediation

The remediation block defines how many machines are replaced and under which circumstances:

Option	Controls
`maxInFlight`	How many machines can be replaced at the same time, as an absolute number (e.g. `2`) or a percentage of the desired replicas (e.g. `20%`). If not set, the number of parallel replacements is only limited by the number of replicas.
`triggerIf.unhealthyLessThanOrEqualTo`	Machines are only replaced if the number of unhealthy machines is less than or equal to this value, as an absolute number (e.g. `2`) or a percentage (e.g. `40%`). If not set, machines are replaced regardless of how many are unhealthy.
`triggerIf.unhealthyInRange`	Machines are only replaced if the number of unhealthy machines is within the given range, e.g. `[3-5]`. Takes precedence over `unhealthyLessThanOrEqualTo` if both are set.

The triggerIf options act as a circuit breaker. If too many machines are unhealthy at the same time, this usually indicates a wider problem, such as a network or availability zone outage, where replacing machines would not help. In this case, no machines are replaced until the number of unhealthy machines is back within the configured limit.

How the limits are evaluated:

Absolute number: With unhealthyLessThanOrEqualTo: 2, up to 2 unhealthy machines are replaced. If 3 or more machines are unhealthy, none are replaced, regardless of the size of the machine deployment.
Percentage: With unhealthyLessThanOrEqualTo: 40% and 25 machines, up to 10 unhealthy machines are replaced; with 6 machines, up to 2. Non-whole results are rounded down. In machine deployments with only a few replicas, a single unhealthy machine already represents a large percentage.
Range: With unhealthyInRange: "[3-5]", machines are only replaced if 3, 4, or 5 machines are unhealthy. With 2 or fewer, or 6 or more unhealthy machines, none are replaced.

Example: Replace nodes that are not ready for 10 minutes, replace at most one machine at a time, and stop replacing machines if more than 40% of the machine deployment is unhealthy:

workers:
- name: <machine-deployment-name>
  healthCheck:
    checks:
      # Node conditions, combined with a logical OR
      unhealthyNodeConditions:
      - type: Ready
        status: "False"
        timeoutSeconds: 600
      - type: Ready
        status: Unknown
        timeoutSeconds: 600
      # Machine conditions, combined with a logical OR
      unhealthyMachineConditions:
      - type: NodeHealthy
        status: "False"
        timeoutSeconds: 600
    remediation:
      # Replace at most one machine at a time
      maxInFlight: 1
      triggerIf:
        # Only replace machines if at most 40% of the machine deployment is unhealthy
        unhealthyLessThanOrEqualTo: 40%
        # Alternatively, only replace machines if 1 to 3 machines are unhealthy.
        # Takes precedence over unhealthyLessThanOrEqualTo if both are set.
        # unhealthyInRange: "[1-3]"

Combining Custom Settings with the Defaults

Custom remediation settings are not merged with the default. If you specify remediation, even partially, the default maxInFlight: 1 is replaced completely. For example, if the remediation example above did not set maxInFlight: 1, any number of machines could be replaced at the same time. If you only want to add a triggerIf limit, please also specify maxInFlight.

Checking Node Health

You can see the state of your nodes with kubectl:

# List all nodes and their status
kubectl get nodes

# Show the conditions of a specific node
kubectl describe node <node-name>

A node that is being replaced shows the status NotReady until it is removed from the cluster. Shortly after, a new node with a different name appears in the list.

If you have any questions or need further assistance, please contact us.