Auto-healing Capabilities - Resurrector¶
The Resurrector is a plugin to the Health Monitor. It's responsible for automatically recreating VMs that become inaccessible.
The Resurrector continuously cross-references VMs expected to be running against the VMs that are sending heartbeats. When resurrector does not receive heartbeats for a VM for a certain period of time, it will kick off a task on the Director (scan and fix task) to try to "resurrect" that VM. The Director may do one of two things:
- create a new VM if the old VM is missing from the IaaS
- replace a VM if the Agent on that VM is not responding to commands
Under certain conditions the Resurrector will consider the system in the "meltdown" and will stop sending requests to the Director - see Resurrection eligibility.
Resurrection can be turned off per instance group via:
bosh update-config --type resurrection --name <name> <path>
Or globally for all VMs managed by the Director via:
bosh update-resurrection off
These are two distinct commands: the first manages fine-grained resurrection rules stored as Director config; the second is a global toggle that takes precedence over any resurrection config.
Note
The Health Monitor deploys with the Resurrector plugin disabled by default. To use it, you must enable the Resurrector plugin in your BOSH deployment manifest.
Resurrection eligibility¶
A VM is a candidate for resurrection when all of the following are true:
- Meltdown is not active - the Resurrector tracks instances with critical alerts within the
time_thresholdwindow. When the percentage of such instances in a deployment meets or exceedspercent_thresholdand their count also meets or exceedsminimum_down_jobs, the Resurrector enters meltdown and pauses all resurrection requests for that deployment until conditions improve. - Global pause is not in effect -
bosh update-resurrection offprevents the Director from accepting any scan and fix request, regardless of resurrection config. - No matching resurrection config rule disables it - all active named resurrection configs are evaluated together; if any matching rule has
enabled: false, the instance is skipped. - Active persistent disks do not block it - by default (
director.auto_fix_stateful_nodes: true), instances with active persistent disks are included. Set this Director property tofalseto exclude them.
Enabling and Configuring the Resurrector¶
To enable the Resurrector:
-
Change deployment manifest for the Health Monitor (typically colocated with the Director):
properties: hm: resurrector_enabled: true -
Optionally change configuration values:
- minimum_down_jobs [Integer, optional]: If the total number of instances that are down in a deployment (within time interval T) is below this number, the Resurrector will always request to fix instances regardless of
percent_threshold. Default is 5. - percent_threshold [Float, optional]: If the percentage of instances that are down in a deployment (within time interval T) is greater than or equal to this threshold and the absolute count also meets or exceeds
minimum_down_jobs, the Resurrector will not request to fix any instance. This state is called "meltdown". Both thresholds must be met or exceeded simultaneously to trigger meltdown. Default is 0.2 (20%). - time_threshold [Integer, optional]: Time interval (in seconds) used in the above calculations. Default is 600.
properties: hm: resurrector_enabled: true resurrector: minimum_down_jobs: 5 percent_threshold: 0.2 time_threshold: 600 - minimum_down_jobs [Integer, optional]: If the total number of instances that are down in a deployment (within time interval T) is below this number, the Resurrector will always request to fix instances regardless of
-
Depending on how you configured Director user management, credentials are specified in the
userandpasswordproperties or using a customclientto authenticate with the UAA.
Option a) Using UAA User Management¶
Define an additional client in the uaa.clients section of your manifest:
properties:
uaa:
clients:
hm:
override: true
authorized-grant-types: client_credentials
scope: ""
authorities: bosh.admin
secret: "hm-password"
Configure the Health Monitor to use the client with the defined secret to authenticate with the Director:
properties:
hm:
director_account:
client_id: hm
client_secret: "hm-password"
Option b) Using Preconfigured Users¶
Create new Director user so that the Resurrector plugin can communicate with the Director and query/submit information about deployments.
properties:
director:
user_management:
provider: local
local:
users:
- {name: admin, password: admin-password}
- {name: hm, password: hm-password}
Configure the Health Monitor to use the HM user and password to authenticate with the Director:
properties:
hm:
director_account:
user: hm
password: hm-password
- Deploy.
Customizing for Your Deployment¶
For most deployments, you can use the default configuration values. In very small or very large deployments, the default values may need customization, as discussed in the following examples.
Small Deployment¶
If your deployment consists of only five VMs, you may not want the Resurrector to attempt to recreate your entire deployment in the event of a catastrophic failure. In this scenario, we recommend that you set minimum_down_jobs to 1 or 2.
Large Deployment¶
If your deployment consists of 1000 VMs, and you use the defaults, the Resurrector notifies the Director to recreate any number of VMs from 1 up to 199 at a time (meltdown triggers at 200, which is 20% of 1000). This is a sliding count: only instances with a critical alert within the last time_threshold seconds are counted, so once resurrected instances' alerts age out, new failures can be fixed. Depending on your deployment, you may consider even 100 down instances a catastrophic failure. In this scenario, set percent_threshold to 0.05 so that the Director resurrects fewer than 50 instances (50 would hit the meltdown threshold).
Enabling the Resurrector with Resurrection Config¶
Note
When used together with the update-resurrection CLI command, pausing resurrection with the CLI command takes precedence (the Director rejects the scan and fix request). It is recommended to use either resurrection config or the CLI command, but not both.
It is possible to configure resurrection based on deployments and instance group names using a Resurrection Configuration file. These files override default resurrection behavior and instruct the BOSH director to resurrect (or not) based on the deployment and instance group names.
The resurrection state will be updated within 60 seconds of uploading a new resurrection config (the Health Monitor polls the Director every 60 seconds) and does not require further actions. For example:
- Upload a resurrection config, which disables resurrection.
- Wait up to 60 seconds for the Health Monitor to poll and apply the new config.
- Delete a VM of a deployment (VM gets not resurrected).
- Delete or replace the disabling resurrection config with one that enables resurrection.
- The missing VM is now eligible for resurrection.
Structure of resurrection config¶
rules:
- enabled: {true,false}
include: # (optional)
deployments: # (optional, one of [deployments, instance_groups] must be present)
- _deployment name1_
- _deployment name2_
instance_groups: # (optional)
- _instance group1_
- _instance group2_
exclude: # (optional)
deployments: # (optional, one of [deployments, instance_groups] must be present)
- _deployment name1_
- _deployment name2_
instance_groups: # (optional)
- _instance group1_
- _instance group2_
Parameters:
rules- a list of resurrection rules. When resurrection rules are interpreted for a given instance, all resurrection rules from all resurrection configs are considered for matching. If no rule matches, resurrection config considers resurrection asenabled.enabled- a boolean which enables (true) or disables (false) resurrection.exclude(Optional) - a resurrection rule which will result in the resurrection configuration being overridden wherever the specified constraints do not match.include(Optional) - a resurrection rule which will result in the resurrection configuration being overridden wherever the specified constraints match.deployments(Optional) - a list of deployments which can be used as a filter for the include/exclude directives.instance_groups(Optional) - a list of instance groups which can be used as a filter for the include/exclude directives.
General¶
- When specifying both properties,
instance_groupsanddeployments, the rule is only applied for the instance groups of the specified deployments. - When only
instance_groupsis specified, the rule will be applied to all matching instance groups across all deployments. - Multiple rules are evaluated by the
&operator. This means, if one rule withenabled: falsematches for a given instance resurrection for this instance is disabled.
Examples¶
By default, resurrection is turned on and the following examples demonstrate how this default can be overwritten.
-
Disable resurrection for a deployment
dep1by creating aresurrection.yml:rules: - enabled: false include: deployments: - dep1Running
bosh update-config --type resurrection --name disable-dep1 resurrection.ymldisables resurrection for all deployments in the include block, i.e.dep1. For all other deployments, resurrection is still enabled.Note
This config remains active until explicitly deleted. If you intend to run example 2, delete it first (
bosh delete-config --type resurrection --name disable-dep1) - whiledisable-dep1is active, it disables all ofdep1regardless of any more specific rules you add. -
Disable resurrection for an instance group
instance-group-1of a deploymentdep1by creating aresurrection.yml:rules: - enabled: false include: deployments: - dep1 instance_groups: - instance-group-1Running
bosh update-config --type resurrection --name disable-dep1-instance-group-1 resurrection.ymldisables resurrection for all instance groups of all the specified deployments, i.e.instance-group-1of deploymentdep1. For all other instance groups and deployments, resurrection is still enabled. -
Disable resurrection for all deployments except for deployment
dep1by creating aresurrection.yml:rules: - enabled: false exclude: deployments: - dep1Running
bosh update-config --type resurrection --name disable-all-but-dep1 resurrection.ymldisables resurrection for all deployments except deploymentdep1.Note
Named resurrection configs remain active until explicitly deleted, and rules from all active configs are combined. If you previously applied example 1 or example 2, delete those configs before running this one - otherwise their rules remain active:
bosh delete-config --type resurrection --name disable-dep1andbosh delete-config --type resurrection --name disable-dep1-instance-group-1.
Disabling resurrection globally¶
Resurrection can be disabled on all deployments with the update-resurrection CLI command:
bosh update-resurrection off
Resurrection can then be re-enabled on all deployments with:
bosh update-resurrection on
Alternatively, resurrection can be disabled on all deployments with this config:
rules:
- enabled: false
Both the CLI command and resurrection config options are meant to temporarily disable resurrection while you diagnose other issues. You should update the deployment manifest to permanently disable the Resurrector.
Disabling the Resurrector¶
To disable the Resurrector:
-
Change deployment manifest for the Health Monitor (typically colocated with the Director):
properties: hm: resurrector_enabled: false -
Redeploy.
-
Optionally remove Director user created for the Health Monitor.
Viewing the Resurrector's Activity¶
The Resurrector creates scan and fix tasks on the Director using the Health Monitor user. Since these are normal tasks, you can use the tasks CLI command to view them.
To view currently running / queued Resurrector activity, run the following and look for results with Description "scan and fix" and User corresponding to the Health Monitor user in your deployment:
bosh tasks --all -d ''
Similarly, to view finished Resurrector activity, run the following and look for results with Description "scan and fix" and User corresponding to the Health Monitor user in your deployment:
bosh tasks --recent --all -d ''