

Rotating the Blobstore CA¶

Ignored Instances¶

Please take note of any ignored instances with bosh instances --details. These instances will not be considered when redeploying instances. Ignored instances will only receive the new certificates when redeployed.

Introducing a New Certificate Authority (CA)¶

This procedure works by deploying both the old and the new CA on all the VMs in a transitional fashion. The old CA is purged eventually. To achieve this, the procedure requires multiple deploys of all the deployments. Note that this method is analogous to the one used for NATS CA rotation, which could be performed at the same time as this one.

Preconditions¶

The Director is in a healthy state.
All the VMs are in the running state in all deployments.
Director versions prior to 271.12 and stemcells prior to Bionic 1.36 and Windows 2019.41 need to recreate VMs as part of the redeploy steps.

Step 1: Redeploy the director with the new blobstore CA.¶

bosh create-env ~/workspace/bosh-deployment/bosh.yml \
 --state ./state.json \
 -o ~/workspace/bosh-deployment/[IAAS]/cpi.yml \
 -o add-new-blobstore-ca.yml \
 -o ... additional opsfiles \
 --vars-store ./creds.yml \
 -v ... additional vars

This adds new variables for the new CA/certificates/private_key.
The director is given a modified CA with the original CA and the new CA concatenated as ((blobstore_server_tls.ca))((blobstore_server_tls_2.ca)).
The blobstore continues to use the old certificates and private key.
Each VM/agent continues to use the old certificates to communicate with the blobstore.

add-new-blobstore-ca.yml

---
- type: replace
  path: /instance_groups/name=bosh/properties/agent/env/bosh/blobstores?/provider=dav/options/tls/cert/ca
  value: ((blobstore_server_tls_2.ca))((blobstore_server_tls.ca))

- type: replace
  path: /instance_groups/name=bosh/properties/blobstore/tls?/cert/ca
  value: ((blobstore_server_tls_2.ca))((blobstore_server_tls.ca))

- type: replace
  path: /variables/-
  value:
    name: blobstore_ca_2
    type: certificate
    options:
      is_ca: true
      common_name: default.blobstore-ca.bosh-internal

- type: replace
  path: /variables/-
  value:
    name: blobstore_server_tls_2
    type: certificate
    options:
      ca: blobstore_ca_2
      common_name: ((internal_ip))
      alternative_names: [((internal_ip))]

Step 2: Redeploy all the VMs, for each deployment.¶

The VMs need to be redeployed in order to receive the new certificates generated from the new Blobstore CA being rotated in. If the VMs are not redeployed, the agents they contain will not be able to communicate with the blobstore since they will not trust the new CA used to sign the blobstore's certificate.

Step 3: Redeploy the director to remove the old Blobstore CA.¶

bosh create-env ~/workspace/bosh-deployment/bosh.yml \
 --state ./state.json \
 -o ~/workspace/bosh-deployment/[IAAS]/cpi.yml \
 -o remove-old-blobstore-ca.yml \
 -o ... additional opsfiles \
 --vars-store ./creds.yml \
 -v ... additional vars

remove-old-blobstore-ca.yml

remove-old-blobstore-ca below is used to remove the old CA from the concatenated CAs.
The blobstore server is updated to use a new certificate and private key, generated by the new CA.
All the components can now communicate using the new CA.

---
- type: replace
  path: /instance_groups/name=bosh/properties/agent/env/bosh/blobstores?/provider=dav/options/tls/cert/ca
  value: ((blobstore_server_tls_2.ca))

- type: replace
  path: /instance_groups/name=bosh/properties/blobstore/tls?/cert
  value:
    ca: ((blobstore_server_tls_2.ca))
    certificate: ((blobstore_server_tls_2.certificate))
    private_key: ((blobstore_server_tls_2.private_key))

- type: replace
  path: /variables/-
  value:
    name: blobstore_ca_2
    type: certificate
    options:
      is_ca: true
      common_name: default.blobstore-ca.bosh-internal

- type: replace
  path: /variables/-
  value:
    name: blobstore_server_tls_2
    type: certificate
    options:
      ca: blobstore_ca_2
      common_name: ((internal_ip))
      alternative_names: [((internal_ip))]

Step 4: Redeploy all VMs, for each deployment.¶

Redeploying all the VMs will remove the old CA from them.

Opsfile Cleanup¶

In order to continue using the new CA and certificates, operators would be required to include the above opsfiles for subsequent bosh create-env commands. This is not ideal and can lead to disruptions and downtime if the old CA is used again by mistake.

The following opsfile in conjunction with the bosh interpolate command will replace the old certificate values with the new, and remove the second variable created for the above process.

# update_blobstore_var_values.yml

---
- type: replace
  path: /blobstore_server_tls
  value: ((blobstore_server_tls_2))

- type: replace
  path: /blobstore_ca
  value: ((blobstore_ca_2))

- type: remove
  path: /blobstore_ca_2

- type: remove
  path: /blobstore_server_tls_2

Create a backup of the credentials and apply the opsfile:

cp creds.yml creds.yml.backup

bosh interpolate creds.yml \
  -o update_blobstore_var_values.yml \
  --vars-file creds.yml > updated_creds.yml

mv updated_creds.yml creds.yml

Do not include the add-new-blobstore-ca and remove-old-blobstore-ca in subsequent bosh create-env commands.

Warning

Warning: If you do not perform the clean-up procedure, you must ensure that the ops files (add-new-blobstore-ca.yml and remove-old-blobstore-ca.yml) are used every time a create-env is executed going forward (which can be unsustainable). Removing the ops files would revert to the old CA, which will prevent blobstore fetching during deployments or other operations.