I'm relatively new to Concourse, so am looking for some advice about how to structure certain types of pipelines. Of the pipelines and resource types I've seen, they all seem to be centered around a set of tasks for a single "thing", where that "thing" is usually a repository. I'm interested in using Concourse to do somewhat broader tasks, where there are multiple "things" that are being run through a single pipeline. Some concrete examples:

• Get the list of repositories from our GitHub org, run some scan on each of them (e.g. checking for a LICENSE.md), then open and issue/pull to the corresponding repository if missing.
• Get a list of domains we control from an API, then check to make sure each responds with a 200 and passes a vulnerability scan, upload the results to S3, then send a notification to the corresponding project team on any failures.

I see a couple of ways of going about these:

• Cram all of the functionality into a loop in a script in a task
  • Presumably the overall build should pass regardless of whether there were failures on the individual repositories/domains?
  • The existing built-in resource types only let you fetch/create one thing (repository, S3 file, etc.). Seems like using/making a custom resource that accepts/returns multiple objects is the "right" thing to do?
• Use parameterized pipelines for checking a single domain/repository, and automatically create a pipeline corresponding to each
  • This would make the list of pipelines veeery long.

Neither seems quite right...any advice? Feel free to close this if you want to keep the open issues limited to bugs/features.

/cc @LinuxBozo @DavidEBest @adrianwebb @dlapiduz