Problem

Currently there is no way to selectively run parts of a DAG. This would be useful for 2 distinct use cases:

1. A developer may create a DAG with lots of features, with the expectation that different users will selectively enable different parts of these features. For example one might want to run tests for the frontend but not the backend; generate API docs and check them for errors but not deploy them; etc.

2. Regardless of features, a user may want to inspect the internals of a DAG to debug an issue, optimize performance, or simply out of curiosity.

Solution

There are two possible approaches:

1. Implement this in CUE with the current APIs. For example, a developer could expose a set of parameters with inputs: params as feature flags. This has the benefit of not requiring any new engine features, and leaving the developer in control of their public interface. But it does not give the end user full control.

2. Implement selective execution via a —target flag or equivalent. This has the benefit of giving the end user full control, but weakens developer’s control over their own interface.

I recommend combining both approaches, each for a specific use case

For feature selection: parameters

When a developer creates a DAG with many features, and wants to give users control over which features to enable, they should manage this themselves with parameters.

Example:

package main

inputs: params: {
  // Deploy documentation?
  deployDocs: *false | true
  // Run frontend tests?
  runFrontendTests: true | *false
  // Run backend tests?
  runBackendTests: true | *false
}

actions: {
  if inputs.params.deployDocs: {
    deployDocs: {
      …
    }
  }
}

Developers should NOT rely on end users selecting certain targets with —target. Terraform makes a very similar recommendation in the documentation for its own —target flag:

Targeting individual resources can be useful for troubleshooting errors, but should not be part of your normal workflow.

For debugging: —target

It makes sense to implement dagger up —target for selective execution. But we should be very vigileant to avoid scope creep.

To quote the Terraform docs again:

Targeting individual resources can be useful for troubleshooting errors, but should not be part of your normal workflow.

There is a risk that the ecosystem relies on —target as an informal API, and developers rely on it for feature selection, instead of implementing their own feature flags. This would be bad because:

1. any key anywhere in your configuration is now a public API you are obligated to support: who knows whether an end user is relying on that key’s existence in an external script right now?
2. large configurations have many fields. Which ones are important to know about and potentially target? Dagger does not provide an answer, so a README or shell script probably willl… Which is exactly what Dagger seeks to solve.

If we allow this bad pattern to develop, it will be hard to root out, and the quality of the Dagger ecosystem will suffer.

Additional resources

• Use case for feature flags: Running Dagger CI in Dagger. PROPOSAL: New CI structure for building, testing and releasing #1230
• Proof-of-concept implementation of —target: Support for partially running a DAG #1302