release: github.com/concourse/concourse / 0.44.0

Github source: 3d5d2eff or master branch

This release is hella backwards incompatible. Read carefully, and ask in IRC (#concourse) if you need help!

We won’t be making such drastic changes after 1.0, but as long as we’re still figuring things out, we don’t want to collect tech debt or land on the wrong set of primitives. - Backwards-incompatible: the progression of artifacts through a build plan has been made more explicit.

Previously there was basically a working directory that would be streamed from step to step, and aggregate steps were relied on to place things under subdirectories, which is how inputs to tasks were satisfied.

Now, as a plan executes, each step’s produced artifact (for example a get step’s fetched bits or the result of a task’s execution) are stored in a pool, with the source named after the step.

This change affects many things, but the primary things you’ll notice are as follows: - When executing a task step, its inputs are collected from the pool, rather than blindly streamed from the previous step. This means aggregate is no longer required to satisfy task inputs, and can now be removed if it’s only wrapping one step.

Tasks are now _required_ to list their set of inputs, otherwise no
inputs will be streamed in. This is backwards-incompatible, but has
many advantages: it’s more explicit, more efficient, and makes it
clearer where the dependent inputs will be placed in a task’s
working directory when it runs.

When a task completes, its resulting working directory is added to
the pool, named after the task itself. This is how you would `put`
using artifacts generated by tasks.
  • The file attribute of a task step must now qualify the path with the name of the source providing the file.
  • When executing a put step, all sources are fetched from the pool. Later on we may introduce a change so that put steps declare their dependencies, but for now streaming everything in is the simplest path forward.

    The net effect of this is that any params referring to files in put steps must now qualify the path with the source name, as they’re all fetched into subdirectories.

  • Now that there’s a flat pool of sources, later steps in a build plan can now refer back to previously fetched (or generated) sources, rather than having to fetch them again.

So, if before you had a plan that looked like this:

  plan:               
  - aggregate:        
    - get: something  
  - task: generate-foo
    file: build.yml   
  - put: foo-bucket   
    params:           
      from: foo       

…it would now look like this:

  plan:                      
  - get: something           
  - task: generate-foo       
    file: something/build.yml
  - put: foo-bucket          
    params:                  
      from: generate-foo/foo 

Notably, the redundant aggregate is gone, the file attribute of the task step qualifies the filename with the name of the source containing it, and the put step qualifies the path to foo with the name of the task that it came from.

Also, the something/build.yml task would now explicitly list its inputs, if it wasn’t before. So that could mean changing:

  platform: linux              
  
  image: docker:///busybox     
  
  run:                         
    path: something/some-script

…to…

  platform: linux              
  
  image: docker:///busybox     
  
  inputs:                      
  - name: something            
  
  run:                         
    path: something/some-script

This has the advantage of making the task config more self-documenting, and removes any doubt as to what inputs will be placed where when the task starts.

Note that listing inputs in the task config is not new, and if you were already listing them before the semantics hasn’t changed. The only difference is that they’re now required. - Backwards-incompatible: worker registration is now done over SSH, using a new component called the TSA.

To upgrade, you’ll have to change your manifest a bit: - On your workers, replace the gate job with groundcrew and remove the gate properties. - The new tsa job template will have to be added somewhere, and configured with the atc credentials (the same way gate used to be configured).

Colocating `tsa` with the `atc` works out nicely, so that you can
register its listening port `2222` with your routing layer (e.g.
ELB), which will already be pointing at the ATC.

To compare, see the example AWS VPC manifest.

The main upshot of this change is it’s much easier to securely register an external worker with Concourse. This new model only needs the worker to be able to reach the ATC rather than the other way around. - Backwards-incompatible: Consul services are now automatically registered based on the jobs being colocated with the agent. For this to work, you must edit your deployment manifest and move the consul-agent job to the top of each job template list, and remove your existing Consul services configuration from your manifest. - The get and put steps from a build’s execution can now be hijacked after they’ve finished or errored. Previously they would be reaped immediately; now they stick around for 5 minutes afterwards (same semantics as tasks). - The S3 resource now defaults to the us-east-1 region. - The S3 resource no longer fails to check when the configured bucket is empty. - A new BOSH Deployment resource has been introduced. It can be used to deploy a given set of release/stemcell tarballs with a manifest to a statically configured BOSH target. The precise versions of the releases and stemcells are overridden in the manifest before deploying to ensure it’s not just always rolling forward to latest.

Upload this release version to the Director:

$ bosh upload release https://bosh.io/d/github.com/concourse/concourse?v=0.44.0

Modify deployment manifest to use this release in addition to any other used releases:

releases:
- {name: concourse, version: "0.44.0"}

Finally add needed deployment jobs and specify values for required properties.

Optionally download sha1: c6acdd181c53e925485a2eb02fa90a028ada354f release tarball locally:

# ...or download it directly using curl
$ curl -L -J -O https://bosh.io/d/github.com/concourse/concourse?v=0.44.0

# or with wget...
$ wget --content-disposition https://bosh.io/d/github.com/concourse/concourse?v=0.44.0