What is the problem you're trying to solve

We can't use profiles to enable specific services in development but not production when using depends_on. That's because the current spec doesn't allow depends_on to be optional. This means in practice profiles can't be used to enable specific services in different environments which is the goal of profiles based on its first sentence of the official documentation.

In practice this is a very big deal because in development you may want to use postgres and redis locally through Docker Compose but in production you may use managed cloud provider services for this. You could say our apps do have dependencies on these services but they are optional from Docker Compose's point of view.

Describe the solution you'd like

Imagine this set up. This actually works with v2.10.2 because there's supposedly an unfixed bug but it turns out this bug is actually what folks have been wanting for years since profiles were available.

services:
  postgres:
    profiles: ["postgres"]
  redis:
    profiles: ["redis"]
  web:
    profiles: ["web"]
    depends_on: ["postgres", "redis"]
  worker:
    profiles: ["worker"]
    depends_on: ["postgres", "redis"]
  assets:
    profiles: ["assets"]

In development you can set this environment variable in an .env file to start everything:

export COMPOSE_PROFILES=postgres,redis,assets,web,worker

In production you can set this instead:

export COMPOSE_PROFILES=web,worker

This would let you avoid running assets containers in production for various watchers and also take advantage of using your cloud provider's managed Postgres and Redis services because the local containers for those services don't need to run since the cloud provider's managed service will fulfill that dependency.

The above is a fantastic solution to let developers pick and choose which services to run. The problem is the current spec doesn't allow the above. Docker Compose will error out saying your dependent services aren't available.

One person opened an issue bringing that up and they want it fixed to be more inline with the spec stating it's confusing. Many alternative solutions were proposed in docker/compose#9795, some of which were to use multiple docker-compose.yml files or introducing more complexity into Docker Compose with competing options like COMPOSE_SERVICE_FILTER to allow for services to be defined and always running commands with --no-deps. None of which are ideal or allow the desired behavior.

A proposed solution

We already have profiles which according to the docs states:

Profiles allow adjusting the Compose application model for various usages and environments by selectively enabling services

What about adding a new necessity: "optional" property which defaults to necessity: "required" to depends_on so it can all work the same with profiles being explicit, or maybe required: false which defaults to required: true? That's in my opinion the easiest solution and it's backwards compatible for the "fixed" behavior while giving people who have already depended on the "unfixed" behavior a way to use Docker Compose in the way v2 has allowed us to use since the beginning of the year.

I'd really like this to be implemented before docker/compose#9795 is addressed. Otherwise Docker Compose v2 will introduce breaking changes with this fix since this unfixed behavior has been available since v2 was around which has been almost a year? It was even GA'd almost 6 months ago.

There is currently no workaround other than using sed and other tools to programmatically alter a docker-compose.yml file for each project before the project is upped. This has been a workaround I've used for years because it was the only way to get the desired behavior of disabling depends_on on a few services in production while still using a single docker-compose.yml file for all environments. It's hacky, cryptic, error prone and required for every project.

The proposed solution here fixes everything to where you can enable specific services in any environment while still being able to use depends_on optionally. It sticks to the goal of what profiles says it does based on its documentation and improves the quality of life for developers and overall improves the developer experience.