Skip to content

Pydantic 2.0 support #3778

New issue
New issue
Closed
#3948
Closed
Pydantic 2.0 support#3778
#3948
Labels
kind/featureNew feature or request

Description

@mjurkus
mjurkus
opened on Oct 2, 2023

Is your feature request related to a problem? Please describe.
I need to install feast with Pydantic 2.0 support, but it's pinned to pydantic>=1,<2

Describe the solution you'd like
Update dependency to pydantic>=2

Pydantic v2 offers improved performance (2x-50x faster), which can contribute to the overall efficiency of Feast.

Updating Pydantic to version 2 will be a valuable enhancement to the feast, and I am willing to contribute to this update if needed. I look forward to hearing your thoughts on this proposal and am open to discussing it further.

Activity

  1. mjurkus
    added
    kind/featureNew feature or request
    on Oct 2, 2023

  2. shuchu commented on Nov 7, 2023

    @shuchu
    shuchu
    on Nov 7, 2023
    Collaborator

    Here is a list of Python files that need to do transformation :

    
(.venv) ubuntu➜~/feast/sdk(master)» grep --include=\*.py -rnw ./python -e "pydantic"                                                        [21:32:27]
./python/feast/feature_server.py:14:from pydantic import BaseModel
./python/feast/infra/offline_stores/redshift.py:22:from pydantic import StrictStr, root_validator
./python/feast/infra/offline_stores/redshift.py:23:from pydantic.typing import Literal
./python/feast/infra/offline_stores/file.py:13:from pydantic.typing import Literal
./python/feast/infra/offline_stores/contrib/spark_offline_store/spark.py:14:from pydantic import StrictStr
./python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres.py:22:from pydantic.typing import Literal
./python/feast/infra/offline_stores/contrib/trino_offline_store/trino.py:8:from pydantic import Field, FilePath, SecretStr, StrictBool, StrictStr, root_validator
./python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py:13:from pydantic.types import StrictStr
./python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py:14:from pydantic.typing import Literal
./python/feast/infra/offline_stores/contrib/athena_offline_store/athena.py:20:from pydantic import StrictStr
./python/feast/infra/offline_stores/contrib/athena_offline_store/athena.py:21:from pydantic.typing import Literal
./python/feast/infra/offline_stores/bigquery.py:22:from pydantic import ConstrainedStr, StrictStr, validator
./python/feast/infra/offline_stores/bigquery.py:23:from pydantic.typing import Literal
./python/feast/infra/offline_stores/snowflake.py:25:from pydantic import Field, StrictStr
./python/feast/infra/offline_stores/snowflake.py:26:from pydantic.typing import Literal
./python/feast/infra/utils/postgres/postgres_config.py:4:from pydantic import StrictStr
./python/feast/infra/materialization/contrib/bytewax/bytewax_materialization_engine.py:13:from pydantic import StrictStr
./python/feast/infra/materialization/aws_lambda/lambda_engine.py:10:from pydantic import StrictStr
./python/feast/infra/materialization/snowflake_engine.py:10:from pydantic import Field, StrictStr
./python/feast/infra/registry/sql.py:9:from pydantic import StrictStr
./python/feast/infra/registry/snowflake.py:10:from pydantic import Field, StrictStr
./python/feast/infra/registry/snowflake.py:11:from pydantic.schema import Literal
./python/feast/infra/online_stores/contrib/cassandra_online_store/cassandra_online_store.py:36:from pydantic import StrictFloat, StrictInt, StrictStr
./python/feast/infra/online_stores/contrib/cassandra_online_store/cassandra_online_store.py:37:from pydantic.typing import Literal
./python/feast/infra/online_stores/contrib/hazelcast_online_store/hazelcast_online_store.py:29:from pydantic import StrictStr
./python/feast/infra/online_stores/contrib/mysql_online_store/mysql.py:8:from pydantic import StrictStr
./python/feast/infra/online_stores/contrib/hbase_online_store/hbase.py:8:from pydantic import StrictStr
./python/feast/infra/online_stores/contrib/hbase_online_store/hbase.py:9:from pydantic.typing import Literal
./python/feast/infra/online_stores/contrib/postgres.py:12:from pydantic.schema import Literal
./python/feast/infra/online_stores/datastore.py:22:from pydantic import PositiveInt, StrictStr
./python/feast/infra/online_stores/datastore.py:23:from pydantic.typing import Literal
./python/feast/infra/online_stores/sqlite.py:21:from pydantic import StrictStr
./python/feast/infra/online_stores/sqlite.py:22:from pydantic.schema import Literal
./python/feast/infra/online_stores/dynamodb.py:19:from pydantic import StrictBool, StrictStr
./python/feast/infra/online_stores/dynamodb.py:20:from pydantic.typing import Literal, Union
./python/feast/infra/online_stores/bigtable.py:10:from pydantic import StrictStr
./python/feast/infra/online_stores/bigtable.py:11:from pydantic.typing import Literal
./python/feast/infra/online_stores/redis.py:32:from pydantic import StrictStr
./python/feast/infra/online_stores/redis.py:33:from pydantic.typing import Literal
./python/feast/infra/online_stores/snowflake.py:8:from pydantic import Field, StrictStr
./python/feast/infra/online_stores/snowflake.py:9:from pydantic.schema import Literal
./python/feast/infra/feature_servers/base_config.py:3:from pydantic import StrictBool, StrictInt
./python/feast/infra/feature_servers/aws_lambda/config.py:1:from pydantic import StrictBool, StrictStr
./python/feast/infra/feature_servers/aws_lambda/config.py:2:from pydantic.typing import Literal
./python/feast/infra/feature_servers/local_process/config.py:1:from pydantic.typing import Literal
./python/feast/infra/feature_servers/gcp_cloudrun/config.py:1:from pydantic import StrictBool
./python/feast/infra/feature_servers/gcp_cloudrun/config.py:2:from pydantic.typing import Literal
./python/feast/repo_config.py:8:from pydantic import (
./python/feast/repo_config.py:17:from pydantic.error_wrappers import ErrorWrapper
./python/feast/repo_config.py:18:from pydantic.typing import Dict, Optional
./python/feast/repo_config.py:313:        # considered tech debt until we can implement

    pydantic/pydantic#619 or a more

  3. shuchu commented on Nov 7, 2023

    @shuchu
    shuchu
    on Nov 7, 2023
    Collaborator

    A single PR to modify all these files (and with proper testing) seems a little bit scary to me. Any suggestions?

    One possible suggestion from Pydantic's doc (https://docs.pydantic.dev/latest/migration/) is:

    1. upgrade the pydantic version from v1 to v2.
    2. use a different import to keep using variables from v1, for example:
      from pydantic.v1 import BaseModel
    3. incrementally upgrade the code base and projects.

  4. mjurkus commented on Nov 7, 2023

    @mjurkus
    mjurkus
    on Nov 7, 2023
    Author

    The migration mostly relates to imports and validation function annotations so it's not that scary.
    What is more worrying is that it also requires updating great-expectations to 0.18.1 and fastapi to 0.103.0 .

  5. mjurkus commented on Nov 7, 2023

    @mjurkus
    mjurkus
    on Nov 7, 2023
    Author

    To start the migration, this needs to be fixed and released pydantic/pydantic#7873

  6. shuchu commented on Nov 8, 2023

    @shuchu
    shuchu
    on Nov 8, 2023
    Collaborator

    I'll keep an eye on the Pydantic's bug you mentioned. Once they fixed it, maybe we can discuss the upgrading of Pydantic from v1 to v2. At the same time, let me check the "great-expectations" and "fastapi" upgrades.

    1. fastapi v.0.104.1 requires pydantic >= 2.0.0
    2. great-expectations v0.18.1 requires pydantic>=1.9.2

  7. shuchu commented on Nov 21, 2023

    @shuchu
    shuchu
    on Nov 21, 2023
    Collaborator

    I'm going to work on this issue.

  8. TomSteenbergen commented on Feb 5, 2024

    @TomSteenbergen
    TomSteenbergen
    on Feb 5, 2024
    Contributor

    Is there any update on Pydantic 2.0 support?

  9. shuchu commented on Feb 5, 2024

    @shuchu
    shuchu
    on Feb 5, 2024
    Collaborator

    sorry for the late. I'm still working on it. please wait for few days :)

  10. shuchu commented on Feb 6, 2024

    @shuchu
    shuchu
    on Feb 6, 2024
    Collaborator

    Just a progress update:
    1, (solved) I have a branch in my own account with Pydantic version equal to the latest one (v2.6). Right now, only one unit test failure left.

    = short test summary info =
FAILED sdk/python/tests/unit/cli/test_cli.py::test_3rd_party_registry_store_with_fs_yaml_override_by_env_var - AssertionError: Expected <1> to be equal to <0>, but was not.
= 1 failed, 240 passed, 25 warnings in 51.35s =

    2, (solved) There is a "mypy" issue here as I also need to update the version to "mypy~=1.1.1" (as required by Pydantic). it will create 120+ mypy lint check errors that need to be fixed. (It will be a tedious grammar change but I just need more time :))

    3, Hope there will be not too many failures while doing the integration test. //pray

  11. shuchu commented on Feb 8, 2024

    @shuchu
    shuchu
    on Feb 8, 2024
    Collaborator

    My friends, the PR is here: #3942

    I humbly ask you guys to review my PR if you have time, since this PR changes many files from the codebase.

  12. shuchu
    closed this as completedin #3948on Feb 15, 2024

  13. shuchu commented on Feb 15, 2024

    @shuchu
    shuchu
    on Feb 15, 2024
    Collaborator

    Folks, the Pydantic V2 is in the master branch. I configured the setup.py as "pydantic>=2.0.0". Especially want to thank Harry @sudohainguyen for spending time to review it!

    Enjoy!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions