Problem

BOSH core team runs a pipeline to produce a bosh release and bosh cli gem combination. It also tests them against a specific version of bats git repo.

BOSH cpi team runs a pipeline that expects to consume bosh release, cli gem and bats repo combination so that it can test it all against a specific version of cpi release. Ultimately this pipeline publishes a new cpi release referencing other asset versions.

We would like to capture group of resource versions instead of just individual resource versions, so that cpi team's pipeline can run against a specific set of resources that passed core team's pipeline.

Other examples:

• concourse release + garden release combination
• cf-release + diego release + cats src combination
• pieces of cf-release

Proposal

I think concourse should introduce a new primitive for specifying resource groups. Resource groups would be versioned similarly to any other resource. Builds would be able to produce a resource group version and consume specific resource group version.

In UI this could look like a box around multiple resources to indicate that they come as a whole.

  +--------------+
  | +----------+ |   +-------+
  | | bats-src |-|---|       |
  | +----------+ |   |       |
  | +----------+ |   |       |
  | | cli-ver  |-|---| tests |
  | +----------+ |   |       |
  | +----------+ |   |       |
  | | bosh-rel |-|---|       |
  | +----------+ |   +-------+
  +--------------+

Resources mentioned in a resource group are not any different from any other resources. They will be configured with all the other resources and have exactly same semantics.

Backing store for resource groups could work similarly to any resource's backing store. For example concourse release can include s3_resource_group that stores and retrieves files from a bucket just like s3_resource.

Contents of each resource group will only include versions for other resources so that storage/retrieval is delegated to each resource. For example contents of v1 of core-assets resource group will be:

candidate-core-assets-bucket/core-asset-1.json

(json in yaml)

bats-src:
  ref: 329ryfvbqeg83490tk549h39q40
cli-ver:
  version: 1.34.0
bosh-rel:
  version: 223

Possible configuration in the pipeline:

resources:
- name: bosh-init
  type: s3
  source:
    regexp: bosh-init-([0-9.]+)-linux-amd64
    bucket: {{init_bucket}}
- name: bats
  type: git
  source:
    uri: https://github.com/cloudfoundry/bosh-acceptance-tests.git
    branch: master
- name: bosh-release
  type: bosh-io-release
  source:
    repository: cloudfoundry/bosh

resource_groups:
- name: core-assets
  type: s3
  resources: [bosh-init, bats, bosh-release]
  source:
    regexp: core-assets-([0-9.]+)
    bucket: {{combos_buckets}}

I'll skip details of how concourse will be able to fetch specific versions of group's resources and its api format.