Skip to content

Fill dill serialization of tasks when using the failures app#163

Merged
gpauloski merged 3 commits intomainfrom
revert-pr-137
Sep 13, 2024
Merged

Fill dill serialization of tasks when using the failures app#163
gpauloski merged 3 commits intomainfrom
revert-pr-137

Conversation

@gpauloski
Copy link
Copy Markdown
Contributor

@gpauloski gpauloski commented Sep 13, 2024

Description

This is related to #155 which reported errors when dill attempted to serialize the closure task wrappers for failure task injection. These closures were serialized by value which caused issues with type annotations containing pydantic models which use ABCs, and dill 0.3.6, required by Globus Compute, can't pickle ABCs.

This PR addresses two aspects of this problem:

I also caught a bug in how _dependency_failure_parent_task was called. It was used like a task factory, but was incorrectly wrapped in the @task decorator.

Fixes

Type of Change

  • Bug (non-breaking change which fixes an issue)
  • Enhancement (non-breaking change which adds or improves functionality)
  • Internal (refactoring, performance, and testing)
  • Breaking (fix or feature that would cause existing functionality to not work as expected)
  • Documentation (no changes to the code)
  • Development (CI workflows, packages, templates, etc.)
  • Package (package dependencies and versions)

Testing

Tested the cholesky app alone and with failure injection using Dask, Process Pool, Ray, and Parsl. I also tested the failure modes for dependency and import which have different task wrappers.

Pull Request Checklist

Please confirm the PR meets the following requirements.

  • Relevant tags are added (breaking, bug, documentation, enhancement, package, etc.).
  • Code changes pass pre-commit (e.g., ruff, mypy, etc.).
  • Tests have been added to show the fix is effective or that the new feature works.
  • New and existing unit tests pass locally with the changes.
  • Docs have been updated and reviewed if relevant.

This is related to #155 which reported errors when dill attempted to
serialize the closure task wrappers for failure task injection. These
closures were serialized by value which caused issues with type
annotations containing pydantic models which use ABCs, and dill 0.3.6,
required by Globus Compute, can't pickle ABCs.

Changing the closures to top-level functions and using functools.partial
to inject the real task and the failure function resolves these
serialization problems; everything can now be serialized by reference.
@gpauloski gpauloski added the bug Something isn't working label Sep 13, 2024
@gpauloski gpauloski merged commit 2ddb4d8 into main Sep 13, 2024
@gpauloski gpauloski deleted the revert-pr-137 branch September 13, 2024 19:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Get parsl.serialize.errors.SerializationError when using Parsl htex to run docking and moldesign

1 participant