Note: This feature is available with bosh-release v255.5+.

Previously, if network communication was required between jobs, release authors had to add job properties to accept other job’s network addresses (e.g. a db_ips property). Operators then had to explicitly assign static IPs or DNS names for each instance group and fill out network address properties. Such configuration typically relied on some helper tool like spiff or careful manual configuration. It also led to inconsistent network configuration as different jobs named their properties differently. All of that did not make it easy to automate and operate multiple environments.

Links provide a solution to the above problem by making the Director responsible for the IP management. Release authors get a consistent way of retrieving networking (and topology) configuration, and operators have a way to consistently connect components.


Overview

First, we provide an overview of the capabilities and logistics of links through a simple example. Here, we have two jobs: an application job and a database job. The database provides its connection information through a link, which the application consumes.

In the database job’s spec file, it declares that it provides the connection information:

# Database job spec file.
name: database_job
# ...
provides:
- name: database_conn
  type: conn
  # Links always carry certain information, like its address and AZ.
  # Optionally, the provider can specify other properties in the link.
  properties:
  - port
  - adapter
  - username
  - password
  - name

properties:
  port:
    default: 8080
# ...

Likewise, the application job’s spec file declares that it consumes connection information:

# Application job spec file.
name: application_job
# ...
consumes:
- name: database_conn
  type: conn
# ...

Then, in the application job’s templates, it can use the connection information from the link:

#!/bin/bash
# Application's templated control script.
# ...
export DATABASE_HOST="<%= link('database_conn').instances[0].address %>"
export DATABASE_PORT="<%= link('database_conn').p('port') %>"
# ...

If the application uses two database connections, each provided by a separate database instance group, the link becomes ambiguous and we must resolve the ambiguity in the deployment manifest. In this case, the application job’s spec file could look like:

# Application job spec file.
name: application_job_two_db
# ...
consumes:
- name: frontend_database
  type: conn
- name: backend_database
  type: conn

And then in the deployment manifest, we disambiguate the links with provides and consumes declarations on the jobs. The database instance groupd name their database_conn links. The application job uses these names to specify which database it will use for each of its links.

# ...
instance_groups:
- name: database_1
  jobs:
  - name: database_job_1
    templates:
    - name: database_job
      release: ...
    provides:
      database_conn: {as: db1}
- name: database_2
    templates:
    - name: database_job
      release: ...
  jobs:
  - name: database_job_2
    provides:
      database_conn: {as: db2}
- name: application_ig
  jobs:
  - name: application_job_two_db
    consumes:
      frontend_database: {from: db1}
      backend_database: {from: db2}
# ...

Release Definitions

Instead of defining properties for every instance group, a job can declare links. (The job either ‘consumes’ a link provided by another job, or it can 'provide’ itself so that any jobs, including itself can 'consume’ it).

In the below yaml snippet the name field is used to differentiate between two links of the same type (db). Both the name and type that are provided can be arbitrarily defined by release authors. Other releases which consume these links must match the type specified.

For example, here is how a web job which receives HTTP traffic and talks to at least one database server may be defined. To connect to a database, it consumes primary_db and secondary_db links of type db. It also exposes an “incoming” link of type http so that other services can connect to it.

Note that when the web job is 'consuming’ db links, the name of the link does not have to match the name of the provided db link (i.e. postgres has a link called conn while the web job consumes primary_db and/or secondary_db). The mapping between the provided link named conn and the consumed link named primary_db is done in the deployment manifest file.

name: web

templates: {...}

consumes:
- name: primary_db
  type: db
- name: secondary_db
  type: db
  optional: true

provides:
- name: incoming
  type: http

properties: {...}

Note that the secondary_db link has been marked as optional, to indicate that the web job will work correctly, even if the operator does not provide a secondary_db link. Providing the secondary_db link may enable some additional functionality.

Here is an example Postgres job that provides a conn link of type db.

name: postgres

templates: {...}

provides:
- name: conn
  type: db

properties: {...}

Template Accessors

Once a release is configured to consume links, the link template accessor allows access to link information such as instance names, AZs, IDs, network addresses, etc.

  • link("...") allows access to linked instances and their properties
  • if_link("...") allows conditional access to a link (useful for optional links)

Besides just collecting all network addresses, links include information that may be useful for determining which instances should be selectively communicating (e.g. based on AZ affinity).

<%=

result = {}

result["primary"] = link("primary_db").instances.map do |instance|
  {
    "name" => instance.name,
    "id" => instance.id,
    "index" => instance.index,
    "az" => instance.az,
    "address" => instance.address,
  }
end

if_link("secondary_db") do |secondary|
  result["secondary"] = secondary.instances.map do |instance|
    {
      "name" => instance.name,
      "id" => instance.id,
      "index" => instance.index,
      "az" => instance.az,
      "address" => instance.address,
    }
  end
end

JSON.dump(result)

%>

Available instance object methods:

  • name [String, non-empty]: Instance name as configured in the deployment manifest.
  • id [String, non-empty]: Unique ID.
  • index [Integer, non-empty]: Unique numeric index. May have gaps.
  • az [String or null, non-empty]: AZ associated with the instance.
  • address [String, non-empty]: IPv4, IPv6 or DNS address. See Native DNS Support for more details.
  • bootstrap [Boolean]: Whether or not this instance is a bootstrap instance.

Properties

See link properties for including additional link information.


Deployment Configuration

Given the web and postgres job examples above, one can configure a deployment that connects a web app to the database. The following example demonstrates linking defined explicitly in the manifest by saying which jobs provide and consume a link data_db.

instance_groups:
- name: app
  jobs:
  - name: web
    release: my-app
    consumes:
      primary_db: {from: data_db}
      secondary_db: nil

- name: data_db
  jobs:
  - name: postgres
    release: postgres
    provides:
      conn: {as: data_db}

Implicit linking

If a link type is provided by only one job within a deployment, all release jobs in that deployment that consume links of that type will be implicitly connected to that provider.

Optional links are also implicitly connected; however, if no provider can be found, they continue to be nil.

Implicit linking does not happen across deployments.

In the following example, it’s unnecessary to explicitly specify that web job consumes the primary_db link of type db from the postgres release job, since the postgres job is the only one that provides a link of type db.

instance_groups:
- name: app
  jobs:
  - name: web
    release: my-app
    consumes: {secondary_db: nil}

- name: data_db
  jobs:
  - name: postgres
    release: postgres

Self linking

A job can consume a link that it provides. This could be used to determine a job’s own peers.

Implicit linking also applies.

instance_groups:
- name: diego-etcd
  jobs:
  - name: etcd
    release: etcd
    consumes:
      etcd: {from: diego-etcd}
    provides:
      etcd: {as: diego-etcd}

Example of self linking in Zookeeper release.

Custom network linking

By default, links include network addresses on the producer’s default link network. The default link network is a network marked with default: [gateway]. A release job can also consume a link over a different network.

For example, this web job will receive data_db’s network addresses on its vip network, instead of receiving network addresses from the private network.

instance_groups:
- name: app
  jobs:
  - name: web
    release: my-app
    consumes:
      primary_db: {from: data_db, network: vip}
      secondary_db: nil
  networks:
  - name: private

- name: data_db
  jobs:
  - name: postgres
    release: postgres
    provides:
      conn: {as: data_db}
  networks:
  - name: private
    default: [gateway, dns]
  - name: vip

Cross-deployment linking

Links can be formed between jobs from different deployments as long as the link is marked as shared.

Unlike links within a deployment, updating a link producing job in one deployment does not affect a link consuming job in another deployment until that deployment is redeployed. To do so, run the bosh deploy command.

Implicit linking does not happen across deployments.

Here is a deployment that provides a database:

name: data-dep
jobs:
- name: db
  jobs:
  - name: postgres
    release: postgres
    provides:
      conn: {as: db, shared: true}

Here is an app deployment that expects to use the database from the deployment above:

instance_groups:
- name: app
  jobs:
  - name: web
    release: my-app
    consumes:
      primary_db: {from: db, deployment: data-dep}
      secondary_db: nil

Next: Link properties or Manual linking.

Back to Table of Contents


Contribute changes to this page