Skip to content

change logic for experiment env variables#6674

Merged
BenWilson2 merged 3 commits intobranch-2.0from
environment-var-logic-change
Sep 2, 2022
Merged

change logic for experiment env variables#6674
BenWilson2 merged 3 commits intobranch-2.0from
environment-var-logic-change

Conversation

@BenWilson2
Copy link
Copy Markdown
Member

@BenWilson2 BenWilson2 commented Sep 1, 2022

Related Issues/PRs

#xxx

What changes are proposed in this pull request?

Update the behavior of MLFLOW_EXPERIMENT_NAME and MLFLOW_EXPERIMENT_ID logic such that values provided in either of these variables are validated against the tracking server to ensure that the user's intent is preserved and no silent failures send runs to the default experiment id.

Prior behavior:
If experiment_name or experiment_id in environment variables does not exist in the tracking server, use the default experiment_id.

New behavior:
If an experiment is active, use that experiment (ignore environment variables).
Otherwise:
If experiment_name is provided in environment variables, validate that experiment name exists in tracking server. If not, raise an Exception.
If experiment name and experiment_id are both provided, check that the experiment_name is registered in the tracking server AND that the experiment_id from the tracking server matches the environment variable experiment_id. If not, raise and Exception.
If experiment_id is provided, validate tracking server registration. If non-existent, provide a custom exception that explains that the environment variable is what is causing the issue (instead of the general tracking server exception about experiment_id not being found).

How is this patch tested?

Modification to existing test suite for the fluent api.

  • I have written tests (not required for typo or doc fix) and confirmed the proposed feature/bug-fix/change works.

Does this PR change the documentation?

  • No. You can skip the rest of this section.
  • Yes. Make sure the changed pages / sections render correctly by following the steps below.
  1. Click the Details link on the Preview docs check.
  2. Find the changed pages / sections and make sure they render correctly.

Release Notes

Is this a user-facing change?

  • No. You can skip the rest of this section.
  • Yes. Give a description of this change to be included in the release notes for MLflow users.

(Details in 1-2 sentences. You can just refer to another PR with a description if this PR is part of a larger change.)

Update the behavior of MLFLOW_EXPERIMENT_NAME and MLFLOW_EXPERIMENT_ID logic such that values provided in either of these variables are validated against the tracking server to ensure that the user's intent is preserved and no silent failures send runs to the default experiment id.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

  • area/artifacts: Artifact stores and artifact logging
  • area/build: Build and test infrastructure for MLflow
  • area/docs: MLflow documentation pages
  • area/examples: Example code
  • area/model-registry: Model Registry service, APIs, and the fluent client calls for Model Registry
  • area/models: MLmodel format, model serialization/deserialization, flavors
  • area/pipelines: Pipelines, Pipeline APIs, Pipeline configs, Pipeline Templates
  • area/projects: MLproject format, project running backends
  • area/scoring: MLflow Model server, model deployment tools, Spark UDFs
  • area/server-infra: MLflow Tracking server backend
  • area/tracking: Tracking Service, tracking client APIs, autologging

Interface

  • area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
  • area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
  • area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
  • area/windows: Windows support

Language

  • language/r: R APIs and clients
  • language/java: Java APIs and clients
  • language/new: Proposals for new client languages

Integrations

  • integrations/azure: Azure and Azure ML integrations
  • integrations/sagemaker: SageMaker integrations
  • integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

  • rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
  • rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
  • rn/feature - A new user-facing feature worth mentioning in the release notes
  • rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
  • rn/documentation - A user-facing documentation change worth mentioning in the release notes

@github-actions github-actions bot added area/tracking Tracking service, tracking client APIs, autologging rn/breaking-change Mention under Breaking Changes in Changelogs. labels Sep 1, 2022
@BenWilson2
Copy link
Copy Markdown
Member Author

Examples failing due to gpytorch issue

ImportError: cannot import name '_mul_broadcast_shape' from 'gpytorch.utils.broadcasting' (/home/runner/.mlflow/envs/mlflow-2d16f39fcdc09a59fbbb370de5a06dfb3831cc7e/lib/python3.8/site-packages/gpytorch/utils/broadcasting.py)```

Comment on lines +1644 to +1645
message=f"The provided environment variable {_EXPERIMENT_ID_ENV_VAR} "
f"`{experiment_id}` does not match the experiment id "
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
message=f"The provided environment variable {_EXPERIMENT_ID_ENV_VAR} "
f"`{experiment_id}` does not match the experiment id "
message=f"The provided {_EXPERIMENT_ID_ENV_VAR} environment variable "
f"value `{experiment_id}` does not match the experiment id "

Comment on lines +1660 to +1662
message=f"The provided environment variable {_EXPERIMENT_ID_ENV_VAR} "
f"`{experiment_id}` does not exist in the tracking server. Provide a valid "
f"experiment_id.",
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
message=f"The provided environment variable {_EXPERIMENT_ID_ENV_VAR} "
f"`{experiment_id}` does not exist in the tracking server. Provide a valid "
f"experiment_id.",
message=f"The provided {_EXPERIMENT_ID_ENV_VAR} environment variable "
f"value `{experiment_id}` does not exist in the tracking server. Provide a valid "
f"experiment_id.",

Copy link
Copy Markdown
Collaborator

@dbczumar dbczumar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks @BenWilson2 !

Copy link
Copy Markdown
Collaborator

@WeichenXu123 WeichenXu123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@BenWilson2 BenWilson2 merged commit 90ca085 into branch-2.0 Sep 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/tracking Tracking service, tracking client APIs, autologging rn/breaking-change Mention under Breaking Changes in Changelogs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants