This topic describes cloud properties for different resources created by the Azure CPI.

AZs

Currently the CPI does not support any cloud properties for AZs.

Example:

azs:
- name: z1

Networks

Dynamic Network or Manual Network

Schema for cloud_properties section:

  • resource_group_name [String, optional]: Name of a resource group. If it is set, Azure CPI will search the virtual network and security group in this resource group. Otherwise, Azure CPI will search the virtual network and security group in resource_group_name in the global CPI settings.
  • virtual_network_name [String, required]: Name of a virtual network. Example: boshnet.
  • subnet_name [String, required]: Name of a subnet within virtual network.
  • security_group [String, optional]: The security group to apply to network interfaces of all VMs placed in this network. The security group of a network interface can be specified either in a resource pool(higher priority) or a network configuration(lower priority); if it is not specified, the default security group (specified by default_security_group in the global CPI settings) will be used.
  • application_security_groups [Array, optional]: The application security group to apply to network interfaces of all VMs placed in this network. The application security groups of a network interface can be specified either in a resource pool(higher priority) or a network configuration(lower priority).
    • This property is supported in v31+.
    • Until 2017-11-02 only West Central US supports this new feature.
    • You must reference the document to register your subscription with this new feature.

See how to create a virtual network and subnets.

Example of manual network:

networks:
- name: default
  type: manual
  subnets:
  - range: 10.10.0.0/24
    gateway: 10.10.0.1
    cloud_properties:
      resource_group_name: my-resource-group-name
      virtual_network_name: my-vnet-name
      subnet_name: my-subnet-name
      security_group: my-security-group-name
      application_security_groups: ["my-application-security-group-name-1", "my-application-security-group-name-2"]

Vip Network

Schema for cloud_properties section:

  • resource_group_name [String, optional]: Name of a resource group. If it is set, Azure CPI will search the public IP in this resource group. Otherwise, Azure CPI will search the public IP in resource_group_name in the global CPI settings.

See how to create public IP to use with vip networks.

Example of vip network:

networks:
- name: public
  type: vip
  cloud_properties:
    resource_group_name: my-resource-group

Resource Pools / VM Types

Schema for cloud_properties section:

  • instance_type [String, required]: Type of the instance. Example: Standard_A2. Basic Tier Virtual Machines should not be used if you need to bind the instance to Azure Load Balancer (ALB), because Basic Tier VM doesn’t support ALB.
  • storage_account_name [String, optional]: Storage account for VMs. Valid only when use_managed_disks is false. If this is not set, the VMs will be created in the default storage account. See this document for more details on why this option exists.
    • If you use DS-series or GS-series as instance_type, you should set this to a premium storage account. See more information about Azure premium storage. See avaliable regions where you can create premium storage accounts.
    • If you use a different storage account which must be in the same resource group, please make sure:
      1. The permissions for the container stemcell in the default storage account is set to Public read access for blobs only.
      2. A table stemcells is created in the default storage account.
      3. Two containers bosh and stemcell are created in the new storage account.
    • If this storage account does not exist, it can be created automatically by Azure CPI. But you must specify storage_account_type and make sure:
      1. The name must be unique within Azure.
      2. The name must be between 3 and 24 characters in length and use numbers and lower-case letters only.
    • If you use a pattern *keyword*. CPI will filter all storage accounts under the default resource group by the pattern and pick one available storage account to create the VM.
      1. The pattern must start with * and end with *.
      2. The keyword must only contain numbers and lower-case letters because of the naming rule of storage account name.
      3. The rule to select an available storage account is to check the number of disks under the container bosh does not exceed the limitation.
      4. The default number of disks limitation is 30 but you can specify it in storage_account_max_disk_number.
  • storage_account_max_disk_number [Integer, optional]: Number of disks limitation in a storage account. Valid only when use_managed_disks is false. Default value is 30. This will be used only when storage_account_name is a pattern.
    • Every storage account has a limitation to host disks. You may hit the performance issue if you create too many disks in one storage account.
    • The maximum number of disks of a standard storage account is 40 because the maximum IOPS of a standard storage account is 20,000 and the maximum IOPS of a standard disk is 500.
    • If you are using premium storage account, Azure maps the disk size (rounded up) to the nearest Premium Storage Disk option (P10, P20 and P30). For example, a disk of size 100 GiB is classified as a P10 option.
      1. The maximum number of disks of a premium storage account is 280 if you are using P10 (128 GiB) as your disk type.
      2. The maximum number of disks of a premium storage account is 70 if you are using P20 (512 GiB) as your disk type.
      3. The maximum number of disks of a premium storage account is 35 if you are using P30 (1024 GiB) as your disk type.
    • storage_account_max_disk_number should be less than the maximum number. Suggest you to use (MAX - 10) as the value because CPI always creates VMs in parallel.
    • Please see more information about azure-subscription-service-limits.
  • storage_account_type [String, optional]: Storage account type. Valid only when use_managed_disks is false. It is required if the storage account does not exist. It can be either Standard_LRS, Standard_ZRS, Standard_GRS, Standard_RAGRS or Premium_LRS. You can click HERE to learn more about the type of Azure storage account.
  • storage_account_location [String, optional]: Location of the storage account. This configuration is deprecated in CPI v25+: if you specify a storage account which does not exist, CPI (v25+) will create it automatically in the same location as VMs’ VNET.
  • availability_set [String, optional]: Name of an availability set to use for VMs. More details.
    • If available set does not exist, it will be automatically created.
    • If availability_set is not specified, Azure CPI will search env.bosh.group as the name of availability set.
      1. bosh release v258+ will generate a value for env.bosh.group automatically.
      2. On Azure the length of the availability set name must be between 1 and 80 characters. The name got from env.bosh.group may be too long. CPI will truncate the name to the following format az-MD5-[LAST-40-CHARACTERS-OF-GROUP] if the length is greater than 80.
    • CPI v27+ will delete the empty availability set.
  • application_gateway [String, optional]: Name of the application gateway which the VMs should be associated to.
    • This property is supported in CPI v28+.
    • You need to create the application gateway manually before configuring it. Please refer to the guidance.
  • platform_update_domain_count [Integer, optional]: The count of update domain in the availability set. Default value is 5.
  • platform_fault_domain_count [Integer, optional]: The count of fault domain in the availability set. For unmanaged availability set, default value is 3. For managed availability set, default value is 2, because some regions don’t support 3 fault domains for now
  • load_balancer [String, optional]: Name of a load balancer the VMs should belong to. You need to create the load balancer manually before configuring it. Note: Basic Tier Virtual Machines (Example: Basic_A1) doesn’t support Azure Load Balancer.
  • security_group [String, optional]: The security group to apply to network interfaces of all VMs placed in this resource pool. The security group of a network interface can be specified either in a resource pool(higher priority) or a network configuration(lower priority); if it is not specified, the default security group (specified by default_security_group in the global CPI settings) will be used.
  • application_security_groups [Array, optional]: The application security group to apply to network interfaces of all VMs placed in this resource group. The application security groups of a network interface can be specified either in a resource pool(higher priority) or a network configuration(lower priority).
    • This property is supported in v31+.
    • Until 2017-11-02 only West Central US supports this new feature.
    • You must reference the document to register your subscription with this new feature.
  • caching [String, optional]: Type of the disk caching of the VMs’ OS disks. It can be either None, ReadOnly or ReadWrite. Default is ReadWrite.
  • root_disk [Hash, optional]: OS disk of custom size.
    • size [Integer, optional]: Specifies the disk size in MiB.
      • The size must be greater than 3 * 1024 and less than the max disk size for unmanaged or managed disk. Please always use N * 1024 as the size because Azure always uses GiB but not MiB.
      • It has a default value 30 * 1024 only when ephemeral_disk.use_root_disk is set to true.
  • ephemeral_disk [Hash, optional]: Ephemeral disk to apply for all VMs that are in this resource pool. By default a data disk with the default size as below will be created as the ephemeral disk.
    • use_root_disk [Boolean, optional]: Enable to use OS disk to store the ephemeral data. The default value is false. When it is true, ephemeral_disk.size will not be used.
    • size [Integer, optional]: Specifies the disk size in MiB. If this is not set, the default size as below will be used. The size of the ephemeral disk for the BOSH VM should be larger than or equal to 30*1024 MiB. Please always use N * 1024 as the size because Azure always uses GiB not MiB.
      • If the Azure temporary disk size for the instance type is less than 30*1024 MiB, the default size is 30*1024 MiB because the space may not be enough.
      • If the Azure temporary disk size for the instance type is larger than 1000*1024 MiB, the default size is 1000*1024 MiB because it is not expected to use such a large ephemeral disk in CF currently.
      • Otherwise, the Azure temporary disk size will be used as the default size. See more information about Azure temporary disk size.
  • assign_dynamic_public_ip [Boolean, optional]: Enable to create and assign dynamic public IP to the VM automatically (to solve the azure SNAT issue). Default value is false. Only the VM without vip will be assigned a dynamic public IP when this value is set to true, and the dynamic public IP will be deleted when the VM is deleted.
  • resource_group_name [String, optional]: Name of a resource group (Available in v26+). If it is set, related resources will be created in this resource group; otherwise, they will be created in resource_group_name specified in the global CPI settings. The resources affected by this property are:
    1. Virtual Machine
    2. Network Interface Card
    3. Managed Disks (including OS disk, ephemeral disk, persistent disk and snapshot)
    4. Dynamic Public IP for the VM
    5. Availability Set

Example of a Standard_A2 instance:

resource_pools:
- name: default
  network: default
  stemcell:
    name: bosh-azure-hyperv-ubuntu-trusty-go_agent
    version: latest
  cloud_properties:
    instance_type: Standard_A2
    availability_set: <availability-set-name>
    root_disk:
      size: 30_720
    ephemeral_disk:
      use_root_disk: false
      size: 30_720

Disk Pools / Disk Types

Schema for cloud_properties section:

  • caching [String, optional]: Type of the disk caching. It can be either None, ReadOnly or ReadWrite. Default is None.
  • storage_account_type [String, optional]: Storage account type. Valid only when use_managed_disks is true. It can be either Standard_LRS or Premium_LRS. You can click HERE to learn more about the type of Azure storage account.

Example of 10GB disk:

  • disk_size [Integer, required]: Size of the disk in MiB. On Azure the disk size must be greater than 1 * 1024 and less than the max disk size for unmanaged or managed disk. Please always use N * 1024 as the size because Azure always uses GiB not MiB.
disk_pools:
- name: default
  disk_size: 10_240

Global Configuration

Schema:

  • environment [String, required]: Azure environment name. Possible values are: AzureCloud, AzureChinaCloud, AzureUSGovernment (available in v19+), AzureGermanCloud (available in v22+) or AzureStack.
  • subscription_id [String, required]: Subscription ID.
  • tenant_id [String, required]: Tenant ID of the service principal.
  • client_id [String, required]: Client ID of the service principal.
  • client_secret [String, required]: Client secret of the service principal.
  • resource_group_name [String, required]: Resource group name.
  • storage_account_name [String, optional]: Storage account name. It will be used as a default storage account for VM disks and stemcells. If use_managed_disks is false, storage_account_name is required. Otherwise, storage_account_name is optional.
  • ssh_user [String, required]: SSH username. Default: vcap.
  • ssh_public_key [String, required]: SSH public key.
  • default_security_group [String, required]: Name of the default security group that will be applied to all created VMs.
  • azure_stack [Hash, optional]: Configration for AzureStack. Available in v23+.
    • domain [String, optional]: The domain for your AzureStack deployment. Default is local.azurestack.external.
    • authentication [String, optional]: The authentication type for your AzureStack deployment. Possible values are: AzureAD, AzureStackAD or AzureStack. Default is AzureAD.
    • resource [String, optional]: The token resource for your AzureStack deployment.
    • endpoint_prefix [String, optional]: The endpoint prefix for your AzureStack deployment. Default is management.
    • skip_ssl_validation [Boolean, optional]: Toggles verification of the Azure Resource Manager REST API SSL certificate. Default is false.
    • use_http_to_access_storage_account [Boolean, optional]: Flag for using HTTP to access storage account rather than the default HTTPS. Default is false.
    • ca_cert [String, optional]: All required custom CA certificates for AzureStack. You can export the Azure Stack CA root certificate. If ca_cert is not provided, the skip_ssl_validation and use_http_to_access_storage_account must be set to true. Available in v27+.
  • parallel_upload_thread_num [Integer, optional]: The number of threads to upload stemcells in parallel. The default value is 16.
  • debug_mode [Boolean, optional]: Enable debug mode. The default value is false. When debug_mode is true:
    • CPI will log all raw HTTP requests/responses.
    • The new created VMs (only for VMs in same region with storage_account_name specified in Global Configuration) will have boot diagnostics enabled. Available in v26+.
  • use_managed_disks [Boolean, optional]: Enable managed disks. The default value is false. For AzureCloud, the option is supported in v21+. For AzureChinaCloud, AzureUSGovernment, and AzureGermanCloud, the option is supported in v26+. For AzureStack, the option is not yet supported.
  • pip_idle_timeout_in_minutes [Integer, optional]: Set idle timeouts in minutes for dynamic public IPs. It must be in the range [4, 30]. The default value is 4. It is only used when assign_dynamic_public_ip is set to true in resouce_pool. Available in V24+.

See all configuration options.

See Creating Azure resources page for more details on how to create and configure above resources.

Example with hard-coded credentials:

properties:
  azure: &azure
    environment: AzureCloud
    subscription_id: 3c39a033-c306-4615-a4cb-260418d63879
    tenant_id: 0412d4fa-43d2-414b-b392-25d5ca46561da
    client_id: 33e56099-0bde-8z93-a005-89c0f6df7465
    client_secret: client-secret
    resource_group_name: bosh-res-group
    storage_account_name: boshstore
    ssh_user: vcap
    ssh_public_key: "ssh-rsa AAAAB3N...6HySEF6IkbJ"
    default_security_group: nsg-azure

Example Cloud Config

azs:
- name: z1
- name: z2

vm_types:
- name: default
  cloud_properties:
    instance_type: Standard_A2

disk_types:
- name: default
  disk_size: 10_240

networks:
- name: default
  type: manual
  subnets:
  - range: 10.10.0.0/24
    gateway: 10.10.0.1
    dns: [168.63.129.16]
    azs: [z1, z2]
    cloud_properties:
      virtual_network_name: boshnet
      subnet_name: boshsub
- name: vip
  type: vip

compilation:
  workers: 5
  reuse_compilation_vms: true
  az: z1
  vm_type: default
  network: default

Errors

  • Invalid service principal
  http_get_response - get_token - http error: 400

Service principal is most likely invalid. Verify that client ID, client secret and tenant ID successfully work:

  $ azure login --username client-id --password client-secret --service-principal --tenant tenant-id
  

If your service principal worked and you get the above error suddenly, it may be caused by that your service principal expired. You need to go to Azure Portal to update client secret. By default, the service principal will expire in one year.

  1. Go to Azure Portal, select active directory – > ORGANIZATION-NAME – > Applications – > search your service principal name.

  2. Then choose your service principal, select Configure – > keys – > add a new key.

  • Exceeding quota limits of Core
  http_put - error: 409 message: {
    "error": {
      "code": "OperationNotAllowed",
      "message": "Operation results in exceeding quota limits of Core. Maximum allowed: 4, Current in use: 4, Additional requested: 1."
    }
  }

Either upgrade your trial account, or file a support ticket in the Azure portal to raise account quotas.

  • NicInUse
  http_delete - error: 400 message: {
      "error": {
          "code": "NicInUse",
          "message": "Network Interface /.../networkInterfaces/dc0d3a9a-0b00-40d8-830d-41e6f4ac9809 is used by existing VM /.../virtualMachines/dc0d3a9a-0b00-40d8-830d-41e6f4ac9809.",
          "details": []
      }
  }

This error indicates that unknown VM (to the Director) took up the IP that the Director is trying to assign to a new VM. Either let the Director know to not use this IP by including it in the reserved section of a subnet in your manual network, or make that IP available by terminating the unknown VM.

  • Limits of Premium Storage blob snapshots
  Error 100: Unknown CPI error 'Unknown' with message 'SnaphotOperationRateExceeded (409): The rate of snapshot blob calls is exceeded.

The BOSH snapshot operation may be throttled if you do all of the following:

The workaround is:

  • Disable snapshot temporarily.

    director:
      enable_snapshots: false
    
  • Adjust the snapshot interval to more than 10 minutes.

    • Version mismatch between CPI and Stemcell

For CPI v11 or later, the compatible stemcell version is v3181 or later. If the stemcell version is older than v3181, you may hit the following failure when deploying BOSH.

  Performing POST request:
    Post https://mbus-user:<redacted>@10.0.0.4:6868/agent: dial tcp 10.0.0.4:6868: getsockopt: connection refused

It is recommended to use the latest version. For example, Stemcell v3232.5 or later, and CPI v12 or later. You may hit the issue #135 if you still use an older stemcell than v3232.5.

  • Out of memory

If you hit Out of memory or Virtual memory exhausted, please check whether you use StandardA0 as instancetype. You should change instance_type to a VM size with more memory. Please reference the issue #230.


Back to Table of Contents


Contribute changes to this page