NOTE: I am not sure what the client is for and what is the intended API. I tried to preserve the existing behavior as closely as possible, but maybe we should change it?

when mlflow.pyfunc.spark_udf() is called with env_manager="virtualenv" or env_manager="conda", MLflow starts a model server and uses this client to send requests to it in order to perform inference; this is the long-awaited "restore model environments in spark UDF" feature that we talked about ~ 3 years ago :). It isn't a public API, so changes are okay as long as Spark UDF continues to work.

cc @WeichenXu123

Yes. The scoring_server.client module is used in spark_udf with env_manager="virtualenv" or env_manager="conda".

NOTE: pandas is able to interpret almost anything as records format so it does not have to be a list for pandas's sake. However, from our perspective, it is better to be more restrictive so there is less variance in input encoding and it is easier to parse the input without pandas. E.g. when parsing logs etc.

Previously, we would never install MLflow. Now, it seems like we'll almost always install MLflow, since users don't typically set mlflow_home. Is this necessary? cc @WeichenXu123

Oh, nice catch, it's inverted. I want to install mlflow only when MLFLOW_HOME is set. MLFLOW_HOME (not very descriptive name in retrospect) is meant for development - so that you can test your local changes when running stuff in a conda environment. For example, in this case, the tests were failing because the conda environment was installing mllfow from pip and hence was expecting the old input.

yes I think only in running test scenario we need set install_mlflow to be true

The impact of this appears to extend beyond tests: https://github.com/mlflow/mlflow/pull/6575/files#r959175274.

commented above. hopefully it explains it. we can also rename MLFLOW_HOME to something like MLFLOW_DEV_PROJECT_HOME in a followup

Thanks!

This breaks the API contract of BaseDeploymenntClient.predict(): https://mlflow.org/docs/latest/python_api/mlflow.deployments.html#mlflow.deployments.BaseDeploymentClient.predict, which states that the API returns a pandas DataFrame, series, or numpy array (i.e pyfunc output, though list should also be added to the docs)

The goal of the original implementation was to convert the response JSON into the corresponding pyfunc output object. Can we preserve that, e.g. by adding / using a utility for parsing the response as a pyfunc output object?

Ah, this is a hard one. We could certainly preserve the behavior but I am not sure if we should because:

1. we wrap the response in a dictionary so we can return additional stuff. I think we should enable access to any additional output via the client interface.
2. It's actually impossible to know whether the response should be pandas dataframe or series or numpy array. I saw most clients will always return pandas dataframe. infer_and_parse_json returns a dataframe unless the input is explicitly a tensor input - i.e. a dict with instances or inputs field. Unfortunately, our current server does not make that distinction and just jsonizes whatever the model returned. It is a bit late to change that now unfortunately.

Therefore, I would propose we update the interface to return pyfunc reponse object that can access the prediction as dataframe or numpy array per user's request and will have access to all response fields. Wdyt? Is that too big a change too late? We could also preserve the original behavior by default and add a flag to use the new behavior.

Following from https://github.com/mlflow/mlflow/pull/6575/files#r959181856, is this class necessary? It doesn't support the full set of pyfunc outputs. Is there a utility method that we can add or extend for parsing the response as a pyfunc output object?

Responded above.