Description

Environment variables defined in the env: block of a DAG (or in base.yaml) resolve to empty strings when the DAG is executed via dagu enqueue, but resolve correctly when executed via dagu start. This applies to both ${VAR} expansion and backtick command substitution (`echo $VAR`).

Version

• Dagu 2.3.6
• Platform: Linux (EKS pod with IAM role credentials)

Reproduction

DAG definition

name: debug-env-resolution
type: graph

env:
  - VIA_DOLLAR_BRACE: "${AWS_WEB_IDENTITY_TOKEN_FILE}"
  - VIA_BACKTICK: "`echo $AWS_WEB_IDENTITY_TOKEN_FILE`"
  - VIA_PRINTENV: "`printenv AWS_WEB_IDENTITY_TOKEN_FILE 2>/dev/null || echo MISSING`"
  - CONTROL_VAR: "static_value"
  - VIA_HOME: "${HOME}"

steps:
  - name: check
    shell: bash
    command: |
      echo "VIA_DOLLAR_BRACE=${VIA_DOLLAR_BRACE:-EMPTY}"
      echo "VIA_BACKTICK=${VIA_BACKTICK:-EMPTY}"
      echo "VIA_PRINTENV=${VIA_PRINTENV:-EMPTY}"
      echo "CONTROL_VAR=${CONTROL_VAR:-EMPTY}"
      echo "VIA_HOME=${VIA_HOME:-EMPTY}"
      echo "Direct access: AWS_WEB_IDENTITY_TOKEN_FILE=${AWS_WEB_IDENTITY_TOKEN_FILE:-EMPTY}"

Confirm the variable exists in the host environment

$ echo $AWS_WEB_IDENTITY_TOKEN_FILE
/var/run/secrets/eks.amazonaws.com/serviceaccount/token

dagu start (works correctly)

$ dagu start debug-env-resolution

Output:

VIA_DOLLAR_BRACE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
VIA_BACKTICK=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
VIA_PRINTENV=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
CONTROL_VAR=static_value
VIA_HOME=/home/coder

dagu enqueue (variables lost)

$ dagu enqueue debug-env-resolution

Output:

VIA_DOLLAR_BRACE=EMPTY
VIA_BACKTICK=EMPTY
VIA_PRINTENV=MISSING
CONTROL_VAR=static_value
VIA_HOME=/home/coder

Analysis

• dagu start executes the DAG in the CLI process, which inherits the full shell environment. All ${VAR} expansions and backtick commands resolve correctly.
• dagu enqueue sends the DAG to the start-all server daemon. The server applies system environment variable filtering, which only whitelists specific prefixes (DAGU_*, DAG_*, LC_*, HOME, PATH, SHELL, etc.). Variables like AWS_* are filtered out.
• The filtering affects all resolution methods: ${VAR}, `echo $VAR`, and `printenv VAR`. The variable is simply not present in the worker's process environment.
• Whitelisted variables like HOME resolve correctly in both modes.
• Defining the variables in base.yaml's env: block does not help — the same filtering applies during base config resolution.

Expected behavior

Variables explicitly referenced in a DAG's env: block (e.g., AWS_WEB_IDENTITY_TOKEN_FILE: "${AWS_WEB_IDENTITY_TOKEN_FILE}") should resolve identically regardless of whether the DAG is started via dagu start or dagu enqueue. If the variable exists in the server process's environment (confirmed via /proc/<pid>/environ), it should be available for expansion.

Workaround

We currently source filtered variables from /proc/1/environ in step commands:

if [ -r /proc/1/environ ]; then
  while IFS='=' read -r -d '' key value; do
    case "$key" in AWS_*) export "$key=$value" ;; esac
  done < /proc/1/environ
fi

Suggestion

Consider one of:

1. Allow env: block references to bypass the filter — if a DAG author explicitly writes AWS_FOO: "${AWS_FOO}", they're opting in to that variable. The filter should only apply to implicit/automatic propagation.
2. Add a configurable allowlist — e.g., a allowed_env_prefixes server config option so admins can add AWS_*, DOCKER_*, etc. (related to Is the DOCKER_HOST environment variable no longer effective in dagu? #1843 where DOCKER_HOST was individually added).
3. Make dagu start and dagu enqueue behave consistently — the current divergence is surprising and hard to debug.

Related issues

• bug: inconsistent environment variables behaviour #1320 — inconsistent environment variables behaviour (same root cause)
• Is the DOCKER_HOST environment variable no longer effective in dagu? #1843 — DOCKER_HOST not effective (same filtering)
• Since 2.3.3 secrets doesn't work with "env" provider #1852 — secrets env provider broken (same filtering)
• Optional environment inheritance for sub-DAGs #1812 — optional environment inheritance for sub-DAGs