Skip to content

Split integration tests per provider #4185

Closed

Description

@tokoko

Is your feature request related to a problem? Please describe.
Integration tests are currently run tests for all implementations (offline, online, registry, materialization engine) in a single pytest call. There are several problems with this approach:

  • It's easy to miss a required dependency in a particular extra as transitive dependencies from the whole project needs to be installed for tests to run.
  • Other implementation dependency might lead to downgrading the dependency for an implementation that's being tested. This means to that we will have library==0.1 tested in ci, while when user installs only the dependencies required for them, they will get library==0.2.
  • We run it local tests twice in ci, once only for local tests and then again for all. With each implementation having a separate test job, we could delay only it tests with ok-to-test label.
  • It's a chore to run tests only for the implementation that you're working on locally.

Cons:

  • we will have to compile python environments for each action run instead of now having only ci requirements file in the repo.
  • Lots of jobs in GH workflows (Is this a bad practce?)

Describe the solution you'd like
I'm thinking of splitting up into runs like this:

  • offline tests - will run tests from /offline_store and /materialization folders. will use default online store (sqlite), registry (file) and all applicable mat. engines whenever necessary.
    • test-python-integration-offline-file
    • test-python-integration-offline-duckdb
    • test-python-integration-offline-spark
    • ...
  • online tests - will run tests from /online_store and /materialization folders. will use default offline store (dask), registry (file) and all applicable mat. engines whenever necessary.
    • test-python-integration-online-sqlite
    • test-python-integration-online-redis
    • ...
  • registry tests - will run tests from /registration
    • test-python-integration-registry-file
    • test-python-integration-registry-sql
    • test-python-integration-registry-s3
    • ...

P.S. There's also e2e folder in integration tests, but at this point I'm not convinced it's actually necessary. One file is testing usage functionality which is set to be deleted anyway. test_universal_e2e.py seems to be a misplaced materialization test.

Activity

  1. stale commented on Apr 26, 2025

    @stale

    This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/featureNew feature or requestwontfixThis will not be worked on

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions