Skip to content

Automatically detect import_modules in PythonPackage#15370

Closed
adamjstewart wants to merge 5 commits intospack:developfrom
adamjstewart:tests/pythonpackage
Closed

Automatically detect import_modules in PythonPackage#15370
adamjstewart wants to merge 5 commits intospack:developfrom
adamjstewart:tests/pythonpackage

Conversation

@adamjstewart
Copy link
Copy Markdown
Member

@adamjstewart adamjstewart commented Mar 6, 2020

Previously, users were required to override import_modules for every PythonPackage that they wanted import tests for. Now, we automatically locate module files and packages. Users can still override import_modules for weird cases.

  • Automatically detect import_modules
  • Remove no longer needed import_modules from packages
  • Update PythonPackage documentation

Note that these import_module_tests now exist for all 850+ PythonPackages in Spack, and these tests can be run after installation has already finished, making these great smoke tests for existing Spack installations.

@adamjstewart adamjstewart added python WIP build-systems stand-alone-tests Stand-alone (or smoke) tests for installed packages labels Mar 6, 2020
@adamjstewart adamjstewart changed the title [WIP] Automatically detect import_modules in PythonPackage Automatically detect import_modules in PythonPackage Mar 6, 2020
@adamjstewart adamjstewart removed the WIP label Mar 6, 2020

def build_scripts_args(self, spec, prefix):
"""Arguments to pass to build_scripts."""
return []
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This little function has been missing since I first created PythonPackage. Oops...

return

for name in self.build_time_test_callbacks:
try:
Copy link
Copy Markdown
Member Author

@adamjstewart adamjstewart Mar 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This try-except was masking a bug in my code that I didn't discover until after I removed it. Personally, I think we shouldn't be catching the exception, and raise an error if a non-existent test function is requested. For most build systems, we implement an empty test method with pass anyway, so it shouldn't fail.

@adamjstewart adamjstewart added the documentation Improvements or additions to documentation label Mar 6, 2020
module = module[:-7]

# Python imports either use foo.bar or foo_bar, never foo-bar.
module = module.replace('-', '.')
Copy link
Copy Markdown
Member Author

@adamjstewart adamjstewart Mar 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both foo.bar and foo_bar are uncommon. I think foo.bar is slightly more common, so I chose this one. If we ever implement #2281/#2718, we will no longer need to guess, as each package will have a pypi attribute with the package name. This is similar to what we did for GNU packages.

@becker33
Copy link
Copy Markdown
Member

@adamjstewart how close is this to being ready?

@adamjstewart
Copy link
Copy Markdown
Member Author

I'm waiting for #15702 to be merged before finishing this PR. That will make it much easier to test without having to reinstall everything.

@becker33
Copy link
Copy Markdown
Member

Ok, then I will move this to the same release I have #15702 scheduled for

@adamjstewart adamjstewart marked this pull request as draft August 7, 2020 17:15
@adamjstewart
Copy link
Copy Markdown
Member Author

Closing this PR for now. The doc changes have been moved to #18181. The import_modules tests will be rewritten after #15702 has been merged, as this will make it easier to test my changes on already-installed packages.

@adamjstewart adamjstewart deleted the tests/pythonpackage branch August 19, 2020 20:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build-systems documentation Improvements or additions to documentation python stand-alone-tests Stand-alone (or smoke) tests for installed packages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants