env must be passed to Runner.__init__, not Runner.run() — the latter
treats unknown kwargs as flow parameters (CLI flags), not as subprocess
environment variables.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Wires up all four remaining free service emulators so the devstack can
test every Metaflow extension without a real cloud account.

Metaflow core changes:
- metaflow_config.py: add BATCH_CLIENT_PARAMS, SFN_CLIENT_PARAMS, and
  SFN_DYNAMO_DB_CLIENT_PARAMS so each boto3 client can be redirected to
  a local emulator via endpoint_url.
- batch_client.py: pass BATCH_CLIENT_PARAMS to get_aws_client.
- step_functions_client.py: pass SFN_CLIENT_PARAMS to get_aws_client.
- dynamo_db_client.py: pass SFN_DYNAMO_DB_CLIENT_PARAMS to get_aws_client.

Devstack changes (Tiltfile):
- ddb-local: amazon/dynamodb-local, port-forwarded to localhost:8765.
  Sets METAFLOW_SFN_DYNAMO_DB_CLIENT_PARAMS and METAFLOW_SFN_DYNAMO_DB_TABLE.
- sfn-local: amazon/aws-stepfunctions-local, port-forwarded to
  localhost:8082 (avoids conflict with metaflow-ui on 8083). Depends on
  ddb-local. Sets METAFLOW_SFN_CLIENT_PARAMS and METAFLOW_SFN_IAM_ROLE.
  The sfn-local container is given AWS_ENDPOINT_URL_DYNAMODB pointing at
  the ddb-local K8s service.
- azurite: mcr.microsoft.com/azure-storage/azurite on ports 10000-10002.
  Creates an azurite-secret K8s secret with well-known dev credentials.
  Sets METAFLOW_AZURE_STORAGE_BLOB_SERVICE_ENDPOINT.
- fake-gcs-server: fsouza/fake-gcs-server on port 4443. Writes
  STORAGE_EMULATOR_HOST=http://localhost:4443 to .devtools/env_local
  (source it alongside the Metaflow config; the GCS SDK picks it up
  automatically).
- write_config_files() extended to emit .devtools/env_local for env vars
  that live outside Metaflow's JSON config.
- pick_services.sh: all four new services added to the picker.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

- azurite: use in-cluster hostname in K8s secret, add azurite-init job to
  create metaflow-test container, add azurite-secret to KUBERNETES_SECRETS,
  set METAFLOW_DATASTORE_SYSROOT_AZURE, write AZURE_STORAGE_CONNECTION_STRING
  to env_local for local dev
- fake-gcs-server: add fake-gcs-secret K8s Secret with in-cluster
  STORAGE_EMULATOR_HOST, add gcs-bucket-init job to create metaflow-test
  bucket, add fake-gcs-secret to KUBERNETES_SECRETS, set
  METAFLOW_DATASTORE_SYSROOT_GS
- Quote env var values in env_local to handle semicolons (Azure conn str)

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Covers prerequisites, quickstart, service table with ports and
dependencies, dev shell usage, credentials reference, and all
Makefile targets.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Ports the pytest parametrization mechanism from the internal mf-5 repo
so the existing core tests can run against all devstack backends via a
single `pytest test/core/test_core_pytest.py` invocation.

Changes:
- test/core/conftest.py: new — pytest_generate_tests hook that reads
  enabled contexts from contexts.json and generates parametrized
  (context, graph, test_batch) combinations, mirroring the mf-5 pattern.
- test/core/test_core_pytest.py: new — test_core_combination() drives
  run_tests.py as a pytest test function, one pytest case per combo.
- test/core/run_tests.py: add `import time` and a new 'scheduler'
  executor type. The scheduler executor runs create → trigger
  --run-id-file → polls Flow(name)[run_id].finished until the async
  run completes. Resume tests are skipped for scheduler contexts since
  Argo/SFN don't support resume.
- test/core/contexts.json: enable all devstack backends (was 1 enabled,
  now 7). New/updated contexts:
    python3-all-local-azure-storage  (Azurite, local execution)
    python3-all-local-gcs            (fake-gcs-server, local execution)
    python3-batch                    (corral, minio, metadata-service)
    python3-k8s                      (minikube k8s, minio, metadata-service)
    python3-argo-workflows           (Argo on minikube, scheduler executor)
    python3-sfn                      (sfn-local + corral, scheduler executor)
  All S3/Azure/GCS sysroots use {nonce} for per-run isolation.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

sfn-local runs as a k8s pod inside minikube, but corral (the local
AWS Batch emulator) runs as a local_resource on the Docker host.
sfn-local calls the Batch API to submit and poll jobs when executing
a state machine, so it needs a reachable Batch endpoint.

- Set AWS_ENDPOINT_URL_BATCH=http://host.docker.internal:8000 in the
  sfn-local Deployment env so it reaches corral on the host (same
  pattern as AWS_ENDPOINT_URL_DYNAMODB pointing to ddb-local).
- Add corral as a dependency of sfn-local so Tilt starts corral
  automatically when sfn-local is selected.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Ports the UX test infrastructure from the internal mf-5 repo to the
OSS metaflow repo, keeping only tests that run with standard OSS
Metaflow (no Netflix-specific compute, metadata, or scheduler).

Structure:
- test/ux/conftest.py: fixtures and parametrization (runner mode by
  default; deployer mode enabled via --scheduler-type flag)
- test/ux/pytest.ini: marker definitions (basic, config, conda, etc.)
- test/ux/core/test_basic.py: hello world, project decorator, conda
- test/ux/core/test_config.py: Config, config_value, FlowMutator,
  StepMutator, config_expr corner cases (6 tests)
- test/ux/core/test_utils.py: run_flow_with_env, verify_single_run,
  deploy_flow_to_scheduler, _evict_flow_module_cache
- test/ux/core/flows/: OSS-cleaned flow files (removed Netflix-only
  Parameter kwargs: external_trigger, external_artifact)

Key details:
- _evict_flow_module_cache() clears click_api.loaded_modules between
  tests so FlowMutator sees fresh config on each run (fixes test
  isolation for mutable_flow tests run sequentially in one session)
- Config file paths use os.path.dirname(__file__) for portability
- Requires a clean OSS-only venv; see test/ux/README.md

The previous conftest only ran in local Runner mode.  This rewrite
aligns with the mf-5 pattern where basic/config tests run against
both runner *and* a remote scheduler/compute backend.

Changes:
- ux_test_config.yaml: new config file (analogous to mf-5's
  ux_test_config.yaml) with scheduler.type and compute.backend fields
- conftest.py: add compute_backend, compute_image, compute_env,
  scheduler_type, cluster fixtures that read from ux_test_config.yaml
  or CLI overrides (--compute-backend, --compute-image, --scheduler-type,
  --cluster)
- decospecs fixture auto-injects the compute backend decospec when a
  backend is configured but not already in --decospecs (mirrors mf-5
  auto-injecting titus:image=...)
- pytest_generate_tests: basic/config tests get runner+deployer modes;
  scheduler_only/data_ux tests get deployer only; deployer variants are
  skipped gracefully when no scheduler is configured

Usage without a backend (local smoke tests, current default):
  pytest test/ux/

Usage with devstack Batch + SFN:
  pytest test/ux/ \
    --compute-backend batch --compute-image <image> \
    --scheduler-type step-functions

Usage matching mf-5 (internal, via ux_test_config.yaml):
  Set scheduler.type and compute.backend in ux_test_config.yaml

Step Functions:
- StepFunctionsDeployedFlow.from_deployment(): look up state machine,
  parse flow_name/owner from start state Parameters, create fake flow
  file, return deployer
- StepFunctionsDeployedFlow.get_triggered_run(): thin wrapper over
  from_deployment
- StepFunctionsTriggeredRun.status property: reads execution status
  via describe_execution()
- StepFunctionsClient.describe_execution(): new method
- StepFunctions.schedule(): early return if no cron (avoids errors
  in sfn-local which has no EventBridge)

Batch:
- batch_client: skip jobRoleArn field when not configured

Airflow deployer (new):
- AirflowDeployer, AirflowDeployedFlow, AirflowTriggeredRun
- AirflowClient REST API wrapper
- Config vars: AIRFLOW_REST_API_URL, credentials, DAG path, namespace
- Registered in DEPLOYER_IMPL_PROVIDERS_DESC
- dev extras in setup.py: pytest, omegaconf, kubernetes

Tiltfile:
- Inject METAFLOW_S3_ENDPOINT_URL into corral containers so Metaflow
  can reach MinIO (AWS_ENDPOINT_URL_S3 alone is not sufficient as
  Metaflow explicitly calls os.getenv('METAFLOW_S3_ENDPOINT_URL'))

Tests:
- ux_test_config.yaml: add sfn-batch and airflow backends
- conftest.py: rewrite to support multiple deployer backends,
  parametrize by backend, handle sfn-batch/argo-kubernetes contexts
- test_basic.py: add deployer tests (deploy/trigger/from_deployment)
- test_config.py: add config mutation tests across backends
- test_utils.py: shared helpers for deployer lifecycle
- hello_from_deployment.py: new flow for from_deployment tests

README:
- Document sfn-batch and argo-kubernetes test invocations
- Add service dependency notes for each backend

Update all references following the PyPI package rename:
- Tiltfile: service name, serve cmd, queue name, inject-env vars
- README: service table, credential table, invocation examples
- pick_services.sh: service option

- devtools/Tiltfile: full devstack with minio, postgresql, metaflow-service,
  argo-workflows, sfn-local, ddb-local, localbatch, airflow; pre-baked test
  image resource (build-test-image), eventbridge stub; PostgreSQL persistence
  + tolerations + liveness thresholds to survive minikube node instability;
  use `minikube image load` for cross-platform image loading
- devtools/docker/metaflow-test/Dockerfile: pre-baked image with boto3/requests
  to eliminate ~2 min pip install per kubernetes/batch pod
- devtools/eventbridge_stub.py: minimal EventBridge stub for sfn-local testing
- test/ux/conftest.py: add --only-backend CLI option for CI matrix filtering
- test/ux/core/conftest.py: devstack env vars (METAFLOW_HOME, AWS endpoints)
- test/ux/core/flows/dag/: branch, foreach, nested-foreach flow fixtures
- test/ux/pytest.ini: timeout=600 for kubernetes tests
- test/ux/ux_test_config.yaml: add airflow-kubernetes backend; pre-baked image
  for argo-kubernetes and sfn-batch; enable all 4 backends
- .github/workflows/ux-tests.yml: parallel matrix over local / argo-kubernetes /
  airflow-kubernetes backends; pytest-xdist -n 6 within each job; sfn-batch
  omitted pending localbatch PyPI publication

- pytest-cov --cov=metaflow --cov-branch in every backend job
- each job uploads .coverage + XML + per-backend HTML as artifacts
- new coverage-report job: downloads all .coverage files, combines them,
  generates HTML report + XML, posts summary to $GITHUB_STEP_SUMMARY
- junit-xml per backend for GHA test tab visibility
- .coveragerc: source/omit/paths config for clean cross-job combine

- Tiltfile: use explicit + for string concat in build-test-image cmd;
  Starlark (unlike Python) requires explicit + for multi-line adjacent strings
- GHA workflow: poll `tilt get session` before `tilt wait` to avoid the
  'No tilt apiserver found' race condition on startup

…race

- Tiltfile: remove chart='airflow' kwarg from helm_remote('airflow', ...);
  Tilt v0.37.0 changed helm_remote so the first positional arg IS chart,
  making chart=kwarg a duplicate. Matches how all other helm_remote calls work.
- Workflow: pin Tilt to v0.33.11 (matches devtools/Makefile) so extensions
  behave consistently between local dev and CI.

…s there)

Test fixtures:
- Replace symlinks to internal mf-5 paths with real files so GHA can
  resolve them: helloworld.py, helloproject.py, helloconda.py,
  config_simple.json, config_simple_cmd.json

Airflow devstack:
- Disable Airflow's internal PostgreSQL subchart; use the shared
  standalone PostgreSQL instance with a separate 'airflow' database.
  This eliminates the second competing PostgreSQL pod that caused
  init-container timeout failures on GHA runners.
- Add airflow-db-init local_resource to create the airflow database
  before Airflow starts.
- Add known postgres superuser password (auth.postgresPassword) so the
  db-init command can authenticate.

GHA workflow:
- Increase minikube memory from 4096 to 6144 MB.
- Increase generate-configs wait timeout from 600s to 900s.

…esource_deps

- Add create-airflow-db.sql to PostgreSQL initdb.scripts so the airflow
  database is created atomically at first boot (before any Airflow pod
  starts). Also add auth.postgresPassword for the fallback local_resource.
- Fix airflow-db-init command quoting: use double quotes for sh -c wrapper
  so inner single-quoted SQL works correctly; add || true equivalent via
  exit 0 to tolerate "database already exists" on local reruns.
- Add resource_deps=['airflow-db-init'] to airflow-run-migrations,
  airflow-create-user, and airflow-triggerer workloads so Tilt waits for
  the DB to exist before deploying any Airflow component.
- Rename "airflow": [] to "airflow": ["postgresql"] so the component
  resolver includes postgresql as a transitive dependency.

…un-migrations)

… slow GHA runners

Airflow:
- Replace startup probe timeout tuning with startupProbe.enabled=false
  for both scheduler and webserver.  The exec-based scheduler probe was
  generating OCI exec errors on minikube/GHA runners, and both probes
  were preventing Tilt from declaring the stack healthy.  Liveness probes
  provide sufficient ongoing health checking.

Argo deployer tests:
- Add _evict_flow_module_cache() to deploy_flow_to_scheduler() so that
  config_value / FlowMutator are applied to a freshly loaded class on
  each deploy.  Without eviction, _configs_processed=True from a prior
  test causes _process_config_decorators to skip mutation, leaving
  added parameters (e.g. param3) absent from the WorkflowTemplate.
- Restructure all 6 deployer test paths to capture and use the Run
  returned by wait_for_deployed_run() directly, instead of calling
  verify_single_run() which scans by tag.  Parallel tests deploy the
  same flow (ConfigSimple / ConfigMutableFlow) to Argo, overwriting the
  shared WorkflowTemplate, causing the last-deployed tags to appear on
  all triggered runs; scanning by tag then finds 2+ runs for a single
  test.  Using the deployer's returned run sidesteps this race entirely.
- Bump per-test timeout 600s→900s to give Argo workflows enough time
  to schedule, pull images, and complete on GHA runners.

… for webserver

The apache-airflow 1.15.0 helm chart schema does not allow 'enabled' under
startupProbe, so we can't disable probes via helm values.

Instead:
- scheduler.startupProbe.command={/bin/true}: replaces the default
  'airflow jobs check' exec probe (which fails with OCI exec errors and
  deprecation warnings on GHA minikube) with a trivially-passing command.
- webserver.startupProbe.failureThreshold=60 / periodSeconds=15 /
  timeoutSeconds=30: allows up to 15 minutes for the webserver HTTP
  health endpoint to become reachable on slow GHA runners.

Deployer tests for the same backend share external scheduler state
(Argo WorkflowTemplates, SFN state machines, etc.).  Running them in
parallel causes races where two tests try to CREATE the same resource
simultaneously and one fails with 'already exists'.

Add pytest.mark.xdist_group('deployer-{backend}') to all deployer-mode
test parameters so that pytest-xdist routes them to the same worker and
they run sequentially.

- Add scheduler.livenessProbe.command={/bin/true} to match the startup
  probe fix: the default 'airflow jobs check' command prints deprecation
  warnings to stderr, causing non-zero exit codes that kill the scheduler
  container via the liveness probe (CrashLoopBackOff)
- Increase webserver memory limit from 512Mi to 1Gi and CPU from 250m to
  500m to prevent gunicorn master timeout ('No response from gunicorn
  master within 120 seconds') on resource-constrained GHA runners
- Extend webserver liveness probe thresholds (failureThreshold=10,
  periodSeconds=30, timeoutSeconds=30) to absorb transient slowness

The Airflow deployer explicitly rejects image_pull_policy in the
@kubernetes decorator (raises AirflowException). Drop it from the
airflow-kubernetes backend decospec.

The image tag (python3.9) is non-latest, so Kubernetes defaults to
IfNotPresent, which is sufficient since metaflow-test:python3.9 is
pre-loaded into minikube by the build-test-image Tilt resource.

- Apply black formatting to test/core/conftest.py and test/core/test_core_pytest.py
- Remove ux-local/ux-sfn/ux-argo/ux-airflow from tox default envlist so plain
  'tox' (run by test.yml CI) only runs unit tests, not devstack-dependent ux tests

…ts Python 3.7

On Python 3.14, typing.Union[X, Y] and typing.Optional[X] normalize
to native types.UnionType (X | Y syntax). The stub generator had no
handler for types.UnionType, causing it to fall through to str(element)
which produces pipe notation instead of typing.Union[...].

Add a types.UnionType branch in _get_element_name_with_module that
converts native union types to typing.Union[...] notation, consistent
with how typing._GenericAlias unions are handled.

- Add python_version >= '3.8' marker to localbatch in [dev] extras so
  Python 3.7 CI matrix can install metaflow[dev] without error
- Fix stub generator to handle typing.Union/Optional on Python 3.14+
  where Union types may no longer be instances of typing._GenericAlias;
  add __origin__ is typing.Union check as a fallback before the
  typing._GenericAlias branch

…x run

ubuntu-latest runners have ~14GB free which isn't enough for the sfn-batch
job (minikube + docker images for 5 services). Reclaim ~30GB upfront by
removing Android SDK, .NET, Haskell, and large packages.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>