This proposal is to add load: foo.yml as a primitive to build plans, allowing parts of the build plan to be dynamically generated while the build is running.

Original discussion:

• New resource interface (+ add versioning) #534 (comment) (and followup comments).

This is a single change that would allow for the following:

• multi-object build advice #292 (dynamic build matrices)

  - task: generate-matrix
  - aggregate: {load: generated-matrix/steps.yml}

• Ability to select worker tags dynamically as part of a build (e.g. acquire from pool) #300 (dynamic worker placement)

  - put: workers
    params: {acquire: true}
  - task: some-task
    tags: [{load: workers/name}]

• Resource Groups #210 (dynamic version pinning)

  - get: valid-versions
    trigger: true
  - aggregate:
    - get: concourse
      version: {load: valid-versions/concourse.json}
    - get: garden-runc
      version: {load: valid-versions/garden-runc.json}

As well as allowing resources to be dynamically configured:

• Move deployment name from resource configuration -> step configuration vmware-archive/bosh-deployment-resource#13 - dynamic deployment name
• Support reading target address from a file vmware-archive/bosh-deployment-resource#6 - dynamic target address
• add option to overwrite/set organization and space with params cloudfoundry-community/cf-resource#20 - dynamic space name

There are some interesting implications to this, however.

• We can no longer know the whole build plan up-front, so we'd need to dynamically render that. Might not be too bad; maybe we just keep saving the build plan as it changes, and the UI keeps up.
• Dynamically created get steps obviously won't be able to trigger the job. I don't think this matters much as there'd probably always be a higher-level get (i.e. of the set of things to fetch) that would really trigger it anyway.
• Config validation and parsing gets tricky, now that every value can technically be replaced with {load: ...}.