This release is hella backwards incompatible. Read carefully, and ask in IRC (#concourse) if you need help!
We won’t be making such drastic changes after 1.0, but as long as we’re still figuring things out, we don’t want to collect tech debt or land on the wrong set of primitives. - Backwards-incompatible: the progression of artifacts through a build plan has been made more explicit.
Previously there was basically a working directory that would be
streamed from step to step, and
aggregate steps were relied on to
place things under subdirectories, which is how inputs to tasks were
Now, as a plan executes, each step’s produced artifact (for example a
get step’s fetched bits or the result of a
task’s execution) are
stored in a pool, with the source named after the step.
This change affects many things, but the primary things you’ll notice
are as follows:
- When executing a
task step, its inputs are collected from the
pool, rather than blindly streamed from the previous step. This
aggregate is no longer required to satisfy task inputs, and
can now be removed if it’s only wrapping one step.
Tasks are now _required_ to list their set of inputs, otherwise no inputs will be streamed in. This is backwards-incompatible, but has many advantages: it’s more explicit, more efficient, and makes it clearer where the dependent inputs will be placed in a task’s working directory when it runs. When a task completes, its resulting working directory is added to the pool, named after the task itself. This is how you would `put` using artifacts generated by tasks.
fileattribute of a
taskstep must now qualify the path with the name of the source providing the file.
When executing a
putstep, all sources are fetched from the pool. Later on we may introduce a change so that
putsteps declare their dependencies, but for now streaming everything in is the simplest path forward.
The net effect of this is that any params referring to files in
putsteps must now qualify the path with the source name, as they’re all fetched into subdirectories.
Now that there’s a flat pool of sources, later steps in a build plan can now refer back to previously fetched (or generated) sources, rather than having to fetch them again.
So, if before you had a plan that looked like this:
plan: - aggregate: - get: something - task: generate-foo file: build.yml - put: foo-bucket params: from: foo
…it would now look like this:
plan: - get: something - task: generate-foo file: something/build.yml - put: foo-bucket params: from: generate-foo/foo
Notably, the redundant
aggregate is gone, the
file attribute of
task step qualifies the filename with the name of the source
containing it, and the
put step qualifies the path to
foo with the
name of the task that it came from.
something/build.yml task would now explicitly list its
inputs, if it wasn’t before. So that could mean changing:
platform: linux image: docker:///busybox run: path: something/some-script
platform: linux image: docker:///busybox inputs: - name: something run: path: something/some-script
This has the advantage of making the task config more self-documenting, and removes any doubt as to what inputs will be placed where when the task starts.
Note that listing inputs in the task config is not new, and if you were already listing them before the semantics hasn’t changed. The only difference is that they’re now required. - Backwards-incompatible: worker registration is now done over SSH, using a new component called the TSA.
To upgrade, you’ll have to change your manifest a bit:
- On your workers, replace the
gate job with
groundcrew and remove
- The new
tsa job template will have to be added somewhere, and
configured with the
atc credentials (the same way
gate used to
Colocating `tsa` with the `atc` works out nicely, so that you can register its listening port `2222` with your routing layer (e.g. ELB), which will already be pointing at the ATC.
To compare, see the example AWS VPC manifest.
The main upshot of this change is it’s much easier to securely
register an external worker with Concourse. This new model only needs
the worker to be able to reach the ATC rather than the other way
- Backwards-incompatible: Consul services are now automatically
registered based on the jobs being colocated with the agent. For this
to work, you must edit your deployment manifest and move the
consul-agent job to the top of each job template list, and remove
your existing Consul services configuration from your manifest.
put steps from a build’s execution can now be hijacked
after they’ve finished or errored. Previously they would be reaped
immediately; now they stick around for 5 minutes afterwards (same
- The S3 resource now
defaults to the
- The S3 resource no longer
fails to check when the configured bucket is empty.
- A new BOSH Deployment resource has been introduced. It can be used to
deploy a given set of release/stemcell tarballs with a manifest to a
statically configured BOSH target. The precise versions of the
releases and stemcells are overridden in the manifest before deploying
to ensure it’s not just always rolling forward to
Upload this release version to the Director:
$ bosh upload-release https://bosh.io/d/github.com/concourse/concourse?v=0.44.0 --sha1 c6acdd181c53e925485a2eb02fa90a028ada354f
Modify deployment manifest to use this release in addition to any other used releases:
releases: - name: concourse version: "0.44.0"
Finally add needed deployment jobs and specify values for required properties.
Optionally download sha1: c6acdd181c53e925485a2eb02fa90a028ada354f release tarball locally:
# ...or download it directly using curl $ curl -L -J -O https://bosh.io/d/github.com/concourse/concourse?v=0.44.0 # or with wget... $ wget --content-disposition https://bosh.io/d/github.com/concourse/concourse?v=0.44.0