[2.0] Make mlflow.sagemaker.deploy() / delete() APIs internal, remove CLIs by dbczumar · Pull Request #6650 · mlflow/mlflow

dbczumar · 2022-08-30T19:42:22Z

Signed-off-by: dbczumar [email protected]

What changes are proposed in this pull request?

Make mlflow.sagemaker.deploy() / delete() APIs internal, since SageMakerDeploymentClient should be used instead.

This is a follow-up for #6651

How is this patch tested?

Existing tests

Does this PR change the documentation?

No. You can skip the rest of this section.
Yes. Make sure the changed pages / sections render correctly by following the steps below.

Click the Details link on the Preview docs check.
Find the changed pages / sections and make sure they render correctly.

Release Notes

Remove the deprecated mlflow.sagemaker.deploy() / delete() APIs, which have been replaced by SageMakerDeploymentClient with MLflow Deployments.

Is this a user-facing change?

No. You can skip the rest of this section.
Yes. Give a description of this change to be included in the release notes for MLflow users.

What component(s), interfaces, languages, and integrations does this PR affect?

Components

Interface

area/uiux: Front-end, user experience, plotting, JavaScript, JavaScript dev server
area/docker: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
area/sqlalchemy: Use of SQLAlchemy in the Tracking Service or Model Registry
area/windows: Windows support

Language

language/r: R APIs and clients
language/java: Java APIs and clients
language/new: Proposals for new client languages

Integrations

integrations/azure: Azure and Azure ML integrations
integrations/sagemaker: SageMaker integrations
integrations/databricks: Databricks integrations

How should the PR be classified in the release notes? Choose one:

rn/breaking-change - The PR will be mentioned in the "Breaking Changes" section
rn/none - No description will be included. The PR will be mentioned only by the PR number in the "Small Bugfixes and Documentation Updates" section
rn/feature - A new user-facing feature worth mentioning in the release notes
rn/bug-fix - A user-facing bug fix worth mentioning in the release notes
rn/documentation - A user-facing documentation change worth mentioning in the release notes

Signed-off-by: dbczumar <[email protected]>

dbczumar · 2022-08-31T05:07:06Z

mlflow/models/__init__.py

+from mlflow.utils import env_manager as _EnvManager
+
+
+def build_docker(


Introduce an API equivalent of mlflow models build-docker so that users don't have to call mlflow.sagemaker.build_and_push_container() with special "no_push" args in order to build an MLflow Model Server docker image. This build_and_push_container should arguably be moved to push_container since it duplicates image building work done by mlflow models build-docker, but that can happen in a follow-up PR.

This is tested sufficiently by mlflow docker CLI tests, since the CLI calls this method directly. See

mlflow/tests/models/test_cli.py

Lines 416 to 492 in 8098785

@pytest.mark.parametrize("enable_mlserver", [True, False])

def test_build_docker(iris_data, sk_model, enable_mlserver):

with mlflow.start_run() as active_run:

if enable_mlserver:

mlflow.sklearn.log_model(

sk_model, "model", extra_pip_requirements=[PROTOBUF_REQUIREMENT]

)

else:

mlflow.sklearn.log_model(sk_model, "model")

model_uri = "runs:/{run_id}/model".format(run_id=active_run.info.run_id)

x, _ = iris_data

df = pd.DataFrame(x)

extra_args = ["--install-mlflow"]

if enable_mlserver:

extra_args.append("--enable-mlserver")

image_name = pyfunc_build_image(model_uri, extra_args=extra_args)

host_port = get_safe_port()

scoring_proc = pyfunc_serve_from_docker_image(image_name, host_port)

_validate_with_rest_endpoint(scoring_proc, host_port, df, x, sk_model, enable_mlserver)

def test_build_docker_virtualenv(iris_data, sk_model):

with mlflow.start_run():

model_info = mlflow.sklearn.log_model(sk_model, "model")

x, _ = iris_data

df = pd.DataFrame(iris_data[0])

extra_args = ["--install-mlflow", "--env-manager", "virtualenv"]

image_name = pyfunc_build_image(model_info.model_uri, extra_args=extra_args)

host_port = get_safe_port()

scoring_proc = pyfunc_serve_from_docker_image(image_name, host_port)

_validate_with_rest_endpoint(scoring_proc, host_port, df, x, sk_model)

@pytest.mark.parametrize("enable_mlserver", [True, False])

def test_build_docker_with_env_override(iris_data, sk_model, enable_mlserver):

with mlflow.start_run() as active_run:

if enable_mlserver:

mlflow.sklearn.log_model(

sk_model, "model", extra_pip_requirements=[PROTOBUF_REQUIREMENT]

)

else:

mlflow.sklearn.log_model(sk_model, "model")

model_uri = "runs:/{run_id}/model".format(run_id=active_run.info.run_id)

x, _ = iris_data

df = pd.DataFrame(x)

extra_args = ["--install-mlflow"]

if enable_mlserver:

extra_args.append("--enable-mlserver")

image_name = pyfunc_build_image(model_uri, extra_args=extra_args)

host_port = get_safe_port()

scoring_proc = pyfunc_serve_from_docker_image_with_env_override(

image_name, host_port, gunicorn_options

)

_validate_with_rest_endpoint(scoring_proc, host_port, df, x, sk_model, enable_mlserver)

def test_build_docker_without_model_uri(iris_data, sk_model, tmp_path):

model_path = tmp_path.joinpath("model")

mlflow.sklearn.save_model(sk_model, model_path)

image_name = pyfunc_build_image(model_uri=None)

host_port = get_safe_port()

scoring_proc = pyfunc_serve_from_docker_image_with_env_override(

image_name,

host_port,

gunicorn_options,

extra_docker_run_options=["-v", f"{model_path}:/opt/ml/model"],

)

x = iris_data[0]

df = pd.DataFrame(x)

_validate_with_rest_endpoint(scoring_proc, host_port, df, x, sk_model)

dbczumar · 2022-08-31T05:08:08Z

mlflow/models/cli.py

    )
-
-
-def _get_flavor_backend(model_uri, **kwargs):


Also needs to be called by mlflow.models.build_docker, so move into mlflow.models.flavor_backend_registry and import it from mlflow.models.cli and mlflow.models.

dbczumar · 2022-08-31T05:08:16Z

mlflow/sagemaker/__init__.py



-def deploy(
+def _deploy(


Make deploy internal

dbczumar · 2022-08-31T05:08:23Z

mlflow/sagemaker/__init__.py

+def _delete(
    app_name,


Make delete internal

dbczumar · 2022-08-31T05:08:44Z

mlflow/sagemaker/__init__.py



-def run_local(model_uri, port=5000, image=DEFAULT_IMAGE_NAME, flavor=None):
+def run_local(name, model_uri, flavor=None, config=None):  # pylint: disable=unused-argument


Make run_local conform to the interface required by mlflow deployments run-local -t sagemaker

These changes are tested by test cases that call score_model_in_sagemaker_docker_container, such as

mlflow/tests/spark/test_spark_model_export.py

Lines 292 to 297 in 8098785

scoring_response = score_model_in_sagemaker_docker_container(

model_uri=model_path,

data=spark_model_iris.pandas_df,

content_type=pyfunc_scoring_server.CONTENT_TYPE_JSON_SPLIT_ORIENTED,

flavor=mlflow.pyfunc.FLAVOR_NAME,

)

dbczumar · 2022-08-31T05:09:15Z

mlflow/sagemaker/cli.py

-        )
-    ),
-)
-def run_local(model_uri, port, image, flavor):


Obviated by mlflow deployments run-local -t sagemaker

dbczumar · 2022-08-31T05:09:26Z

mlflow/sagemaker/cli.py

-        " asynchronously using the `--async` flag, this value is ignored."
-    ),
-)
-def delete(app_name, region_name, archive, asynchronous, timeout):


obviated by mlflow deployments delete -t sagemaker

dbczumar · 2022-08-31T05:09:50Z

mlflow/sagemaker/cli.py

-        " asynchronously using the `--async` flag, this value is ignored."
-    ),
-)
-def deploy(


Obviated by mlflow deployments create/update -t sagemaker

dbczumar · 2022-08-31T05:10:45Z

tests/sagemaker/test_deployment.py

@@ -1,875 +0,0 @@
-import os


All of the coverage provided in this test suite for mlflow.sagemaker.deploy() / delete() is covered for SageMakerDeploymentClient in https://github.com/mlflow/mlflow/blob/master/tests/sagemaker/test_sagemaker_deployment_client.py

dbczumar · 2022-08-31T05:11:32Z

tests/helper_functions.py

+        f" -C image=mlflow-pyfunc -C port={port} --flavor {flavor}"
+    )
    proc = _start_scoring_proc(
-        cmd=["mlflow", "sagemaker", "run-local", "-m", model_uri, "-p", "5000", "-f", flavor],


mlflow sagemaker run-local is obviated by mlflow deployments run-local -t sagemaker and is removed above, so we migrate this test helper function that relies on sagemaker local deployment to use the mlflow deployments run-local API

Signed-off-by: dbczumar <[email protected]>

harupy · 2022-08-31T07:02:58Z

docs/source/models.rst

-To export a custom model to SageMaker, you need a MLflow-compatible Docker image to be available on Amazon ECR.
-MLflow provides a default Docker image definition; however, it is up to you to build the image and upload it to ECR.
-MLflow includes the utility function ``build_and_push_container`` to perform this step. Once built and uploaded, you can use the MLflow container for all MLflow Models. Model webservers deployed using the :py:mod:`mlflow.sagemaker`
+The :py:mod:`mlflow.deployments` and py:mod:`mlflow.sagemaker` modules can deploy


Suggested change

The :py:mod:`mlflow.deployments` and py:mod:`mlflow.sagemaker` modules can deploy

The :py:mod:`mlflow.deployments` and :py:mod:`mlflow.sagemaker` modules can deploy

Thanks for catching this!

harupy · 2022-08-31T07:05:49Z

docs/source/models.rst

+    mlflow deployments run-local -t sagemaker -m <path-to-model>  - test the model locally
+    mlflow deployments -t sagemaker create - deploy the model remotely


Suggested change

mlflow deployments run-local -t sagemaker -m <path-to-model> - test the model locally

mlflow deployments -t sagemaker create - deploy the model remotely

mlflow deployments run-local -t sagemaker -m <path-to-model> # test the model locally

mlflow deployments -t sagemaker create # deploy the model remotely

Can we replace - with #? - test the model locally looks like a command argument, but it's not.

Absolutely! Fixed!

harupy · 2022-08-31T07:07:56Z

mlflow/sagemaker/__init__.py

+        mlflow deployments run-local --target sagemaker \\
+                --name my-local-deployment \\
+                --model-uri "/mlruns/0/abc/model" \\
+                --flavor python_function\\


Suggested change

--flavor python_function\\

--flavor python_function \\

nit

harupy · 2022-08-31T07:13:38Z

mlflow/sagemaker/__init__.py

+            flavor="python_function",
+            config={
+                "port": 5000,
+                "image": mlflow-pyfunc,


Suggested change

"image": mlflow-pyfunc,

"image": "mlflow-pyfunc",

Fixed! Thanks for catching this!

harupy · 2022-08-31T07:22:03Z

Can we revert the changes in examples/pipelines/sklearn_regression?

harupy · 2022-08-31T09:29:36Z

docs/source/models.rst

-    mlflow sagemaker run-local -m <path-to-model>  - test the model locally
-    mlflow sagemaker deploy <parameters> - deploy the model remotely
+    mlflow deployments run-local -t sagemaker -m <path-to-model>  - test the model locally
+    mlflow deployments -t sagemaker create - deploy the model remotely


Suggested change

mlflow deployments -t sagemaker create - deploy the model remotely

mlflow deployments create -t sagemaker - deploy the model remotely

mlflow deployments -t sagemaker create gave the following error.

mlflow deployments -t sagemaker create Usage: mlflow deployments [OPTIONS] COMMAND [ARGS]... Try 'mlflow deployments --help' for help. Error: No such option: -t

Great catch! Fixed!

harupy · 2022-08-31T09:36:52Z

docs/source/models.rst

-* :py:func:`deploy <mlflow.sagemaker.deploy>` deploys the model on Amazon SageMaker. MLflow
-  uploads the Python Function model into S3 and starts an Amazon SageMaker endpoint serving
-  the model.
+* :py:func:`deploy <mlflow.sagemaker.SageMakerDeploymentClient>` deploys the model on Amazon


Is this line correct? In the code block below, create deploys the model.

Suggested change

* :py:func:`deploy <mlflow.sagemaker.SageMakerDeploymentClient>` deploys the model on Amazon

* :py:func:`create <mlflow.sagemaker.SageMakerDeploymentClient.creaet_deployment>` deploys the model on Amazon

Fixed and made the link read mlflow deployments create -t sagemaker, also updated run-local to mlflow deployments run-local -t sagemaker

Signed-off-by: dbczumar <[email protected]>

dbczumar

Thanks for the awesome feedback, @harupy ! I've addressed your comments :)

dbczumar · 2022-08-31T18:15:35Z

docs/source/models.rst

-To export a custom model to SageMaker, you need a MLflow-compatible Docker image to be available on Amazon ECR.
-MLflow provides a default Docker image definition; however, it is up to you to build the image and upload it to ECR.
-MLflow includes the utility function ``build_and_push_container`` to perform this step. Once built and uploaded, you can use the MLflow container for all MLflow Models. Model webservers deployed using the :py:mod:`mlflow.sagemaker`
+The :py:mod:`mlflow.deployments` and py:mod:`mlflow.sagemaker` modules can deploy


Thanks for catching this!

dbczumar · 2022-08-31T18:19:51Z

docs/source/models.rst

-* :py:func:`deploy <mlflow.sagemaker.deploy>` deploys the model on Amazon SageMaker. MLflow
-  uploads the Python Function model into S3 and starts an Amazon SageMaker endpoint serving
-  the model.
+* :py:func:`deploy <mlflow.sagemaker.SageMakerDeploymentClient>` deploys the model on Amazon


Fixed and made the link read mlflow deployments create -t sagemaker, also updated run-local to mlflow deployments run-local -t sagemaker

dbczumar · 2022-08-31T18:20:34Z

docs/source/models.rst

+    mlflow deployments run-local -t sagemaker -m <path-to-model>  - test the model locally
+    mlflow deployments -t sagemaker create - deploy the model remotely


Absolutely! Fixed!

dbczumar · 2022-08-31T18:21:01Z

docs/source/models.rst

-    mlflow sagemaker run-local -m <path-to-model>  - test the model locally
-    mlflow sagemaker deploy <parameters> - deploy the model remotely
+    mlflow deployments run-local -t sagemaker -m <path-to-model>  - test the model locally
+    mlflow deployments -t sagemaker create - deploy the model remotely


Great catch! Fixed!

dbczumar · 2022-08-31T18:21:39Z

mlflow/sagemaker/__init__.py

+        mlflow deployments run-local --target sagemaker \\
+                --name my-local-deployment \\
+                --model-uri "/mlruns/0/abc/model" \\
+                --flavor python_function\\


dbczumar · 2022-08-31T18:22:10Z

mlflow/sagemaker/__init__.py

+            flavor="python_function",
+            config={
+                "port": 5000,
+                "image": mlflow-pyfunc,


Fixed! Thanks for catching this!

harupy

LGTM!

dbczumar added 3 commits August 30, 2022 12:34

Make internal

6815c4f

Signed-off-by: dbczumar <[email protected]>

Fix

ed3f6f2

Signed-off-by: dbczumar <[email protected]>

CLI & tests

3f044d6

Signed-off-by: dbczumar <[email protected]>

dbczumar changed the title ~~[2.0] Make mlflow.sagemaker.deploy() / delete() APIs internal~~ [2.0] Make mlflow.sagemaker.deploy() / delete() APIs internal, remove CLIs Aug 30, 2022

Call internal

8f8b922

Signed-off-by: dbczumar <[email protected]>

dbczumar requested review from AveshCSingh and harupy August 30, 2022 19:54

github-actions bot added area/scoring MLflow Model server, model deployment tools, Spark UDFs rn/breaking-change Mention under Breaking Changes in Changelogs. labels Aug 30, 2022

dbczumar changed the title ~~[2.0] Make mlflow.sagemaker.deploy() / delete() APIs internal, remove CLIs~~ [WIP][2.0] Make mlflow.sagemaker.deploy() / delete() APIs internal, remove CLIs Aug 30, 2022

dbczumar added 6 commits August 30, 2022 18:36

Build_docker add

b17b79e

Signed-off-by: dbczumar <[email protected]>

Format

6f2dc2a

Signed-off-by: dbczumar <[email protected]>

Fix

39847e9

Signed-off-by: dbczumar <[email protected]>

Image and port

0ac99f3

Signed-off-by: dbczumar <[email protected]>

Change image name

44f1677

Signed-off-by: dbczumar <[email protected]>

Test fix

e494ccb

Signed-off-by: dbczumar <[email protected]>

dbczumar commented Aug 31, 2022

View reviewed changes

mlflow/sagemaker/__init__.py

def deploy(

def _deploy(

Copy link
Copy Markdown

Collaborator Author

dbczumar Aug 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make deploy internal

dbczumar commented Aug 31, 2022

View reviewed changes

mlflow/sagemaker/__init__.py

Comment on lines +442 to 443

def _delete(

app_name,

Copy link
Copy Markdown

Collaborator Author

dbczumar Aug 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make delete internal

dbczumar commented Aug 31, 2022

View reviewed changes

Fixes

d73b266

Signed-off-by: dbczumar <[email protected]>

dbczumar changed the title ~~[WIP][2.0] Make mlflow.sagemaker.deploy() / delete() APIs internal, remove CLIs~~ [2.0] Make mlflow.sagemaker.deploy() / delete() APIs internal, remove CLIs Aug 31, 2022

Fix

b91f07f

Signed-off-by: dbczumar <[email protected]>

harupy reviewed Aug 31, 2022

View reviewed changes

dbczumar added 2 commits August 31, 2022 11:23

Address awesome review comments

128e580

Signed-off-by: dbczumar <[email protected]>

Small fix

e5c9c21

Signed-off-by: dbczumar <[email protected]>

dbczumar commented Aug 31, 2022

View reviewed changes

harupy approved these changes Aug 31, 2022

View reviewed changes

dbczumar merged commit 32ee711 into mlflow:branch-2.0 Sep 1, 2022

		from mlflow.utils import env_manager as _EnvManager


		def build_docker(

	@pytest.mark.parametrize("enable_mlserver", [True, False])
	def test_build_docker(iris_data, sk_model, enable_mlserver):
	with mlflow.start_run() as active_run:
	if enable_mlserver:
	mlflow.sklearn.log_model(
	sk_model, "model", extra_pip_requirements=[PROTOBUF_REQUIREMENT]
	)
	else:
	mlflow.sklearn.log_model(sk_model, "model")
	model_uri = "runs:/{run_id}/model".format(run_id=active_run.info.run_id)

	x, _ = iris_data
	df = pd.DataFrame(x)

	extra_args = ["--install-mlflow"]
	if enable_mlserver:
	extra_args.append("--enable-mlserver")

	image_name = pyfunc_build_image(model_uri, extra_args=extra_args)
	host_port = get_safe_port()
	scoring_proc = pyfunc_serve_from_docker_image(image_name, host_port)
	_validate_with_rest_endpoint(scoring_proc, host_port, df, x, sk_model, enable_mlserver)


	def test_build_docker_virtualenv(iris_data, sk_model):
	with mlflow.start_run():
	model_info = mlflow.sklearn.log_model(sk_model, "model")

	x, _ = iris_data
	df = pd.DataFrame(iris_data[0])

	extra_args = ["--install-mlflow", "--env-manager", "virtualenv"]
	image_name = pyfunc_build_image(model_info.model_uri, extra_args=extra_args)
	host_port = get_safe_port()
	scoring_proc = pyfunc_serve_from_docker_image(image_name, host_port)
	_validate_with_rest_endpoint(scoring_proc, host_port, df, x, sk_model)


	@pytest.mark.parametrize("enable_mlserver", [True, False])
	def test_build_docker_with_env_override(iris_data, sk_model, enable_mlserver):
	with mlflow.start_run() as active_run:
	if enable_mlserver:
	mlflow.sklearn.log_model(
	sk_model, "model", extra_pip_requirements=[PROTOBUF_REQUIREMENT]
	)
	else:
	mlflow.sklearn.log_model(sk_model, "model")
	model_uri = "runs:/{run_id}/model".format(run_id=active_run.info.run_id)
	x, _ = iris_data
	df = pd.DataFrame(x)

	extra_args = ["--install-mlflow"]
	if enable_mlserver:
	extra_args.append("--enable-mlserver")

	image_name = pyfunc_build_image(model_uri, extra_args=extra_args)
	host_port = get_safe_port()
	scoring_proc = pyfunc_serve_from_docker_image_with_env_override(
	image_name, host_port, gunicorn_options
	)
	_validate_with_rest_endpoint(scoring_proc, host_port, df, x, sk_model, enable_mlserver)


	def test_build_docker_without_model_uri(iris_data, sk_model, tmp_path):
	model_path = tmp_path.joinpath("model")
	mlflow.sklearn.save_model(sk_model, model_path)
	image_name = pyfunc_build_image(model_uri=None)
	host_port = get_safe_port()
	scoring_proc = pyfunc_serve_from_docker_image_with_env_override(
	image_name,
	host_port,
	gunicorn_options,
	extra_docker_run_options=["-v", f"{model_path}:/opt/ml/model"],
	)
	x = iris_data[0]
	df = pd.DataFrame(x)
	_validate_with_rest_endpoint(scoring_proc, host_port, df, x, sk_model)



		def run_local(model_uri, port=5000, image=DEFAULT_IMAGE_NAME, flavor=None):
		def run_local(name, model_uri, flavor=None, config=None): # pylint: disable=unused-argument

	scoring_response = score_model_in_sagemaker_docker_container(
	model_uri=model_path,
	data=spark_model_iris.pandas_df,
	content_type=pyfunc_scoring_server.CONTENT_TYPE_JSON_SPLIT_ORIENTED,
	flavor=mlflow.pyfunc.FLAVOR_NAME,
	)

	The :py:mod:`mlflow.deployments` and py:mod:`mlflow.sagemaker` modules can deploy
	The :py:mod:`mlflow.deployments` and :py:mod:`mlflow.sagemaker` modules can deploy

		mlflow deployments run-local -t sagemaker -m <path-to-model> - test the model locally
		mlflow deployments -t sagemaker create - deploy the model remotely

	mlflow deployments -t sagemaker create - deploy the model remotely
	mlflow deployments create -t sagemaker - deploy the model remotely

	* :py:func:`deploy <mlflow.sagemaker.SageMakerDeploymentClient>` deploys the model on Amazon
	* :py:func:`create <mlflow.sagemaker.SageMakerDeploymentClient.creaet_deployment>` deploys the model on Amazon

Conversation

dbczumar commented Aug 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes are proposed in this pull request?

How is this patch tested?

Does this PR change the documentation?

Release Notes

Is this a user-facing change?

What component(s), interfaces, languages, and integrations does this PR affect?

How should the PR be classified in the release notes? Choose one:

Uh oh!

dbczumar Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbczumar Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harupy Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harupy commented Aug 31, 2022

Uh oh!

harupy Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

harupy Aug 31, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbczumar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dbczumar commented Aug 30, 2022 •

edited

Loading

dbczumar Aug 31, 2022 •

edited

Loading

dbczumar Aug 31, 2022 •

edited

Loading

harupy Aug 31, 2022 •

edited

Loading

harupy Aug 31, 2022 •

edited

Loading

harupy Aug 31, 2022 •

edited

Loading