[WIP] Speed up with targeted memoization#935
Closed
citibeth wants to merge 2 commits intospack:developfrom
Closed
Conversation
Member
Author
|
Here is the profile I'm talking about above. |
8 tasks
Member
|
@citibeth: I'm a little worried about the correctness of this -- the concretization tests are failing in the Travis output above. |
Member
Author
|
See what that does to it... I noticed the bug as soon as I started using Frankly, I doubt that this PR should be merge. It's more of a starting On Wed, May 11, 2016 at 12:44 PM, Todd Gamblin [email protected]
|
tgamblin
changed the title
Member
Author
|
@tgamblin, has this Travis check gotten stuck? |
Member
|
I kicked it off again |
Member
Author
|
I found another bug, and reverted this entire PR in my own rpo. This PR needs to die. |
olupton
pushed a commit
to olupton/spack
that referenced
this pull request
Feb 7, 2022
* New compiler: nvhpc (NVIDIA HPC SDK) (spack#19294) * Add nvhpc compiler definition: "spack compiler add" will now look for instances of the NVIDIA HPC SDK compiler executables (nvc, nvc++, nvfortran) in supplied paths * Add the nvhpc package which installs the nvhpc compiler * Add testing for nvhpc detection and C++-standard/pic flags Based on spack#19294 * Add CUDA@11 required for latest NVIDIA-HPC-SDK * Fix legacy apis : setup_environment to setup_run_environment
olupton
pushed a commit
to olupton/spack
that referenced
this pull request
Feb 7, 2022
* Update bbp-packages.yaml Adding my currentscape module to bbp-packages. * Update modules.yaml Adding my currentscape package to the whitelist. * Create package.py Added my currentscape package. * Update var/spack/repos/builtin/packages/py-currentscape/package.py Co-authored-by: Matthias Wolf <[email protected]> * Update var/spack/repos/builtin/packages/py-currentscape/package.py Co-authored-by: Matthias Wolf <[email protected]> * Update var/spack/repos/builtin/packages/py-currentscape/package.py Co-authored-by: Matthias Wolf <[email protected]> * Update package.py Modified the version. Is it better now? * update bglibpy to 4.4.6 (spack#937) * Update package.py Changing tag to match the one from source on gerrit. * Bglibpy 4.4.10 (spack#938) * update bglibpy to 4.4.6 * updated to bglpy 4.4.10 v 4.4.10 does not have the pandas, pyrsistent or the bluepy-configfile dependencies * Updates Libsonata package to include readers improvements (spack#940) * update glm to 0.9.9.3 (spack#943) * Bump neurodamus for most recent patches (spack#942) Bump neurodamus-py 2.0.0 to 2.0.2: - Fixing replay to work with multiple populations - Ensure data dir when skipping model build - Fix skipping synapse creation when weight is 0 (BBPBGLIB-673) - Fix deadlock when an exception is thrown from NEURON (BBPBGLIB-678) - Logging colors only for tty Bump neurodamus-core 3.0.0 to 3.0.1: - Avoid getting nilSecRef from objects (HPCTM-1381) * Update py-sonata-network-reduction dependencies: [email protected], [email protected] (spack#930) * Update bbp-packages.yaml Updated currentscape version. * Update package.py Updated version & tag. * Brion and Brayns are dependent on GLM (spack#944) * Steps updates (spack#941) * gmsh: add version 4.6.0 * omega-h: new version 9.32.5.dev3 * steps: new test requirements * libsonata-report: Improves initialization performance (spack#945) * adapt brion test to a new python module name (spack#946) * Adding nvidia-hpc-sdk based on upstream PR (spack#935) * New compiler: nvhpc (NVIDIA HPC SDK) (spack#19294) * Add nvhpc compiler definition: "spack compiler add" will now look for instances of the NVIDIA HPC SDK compiler executables (nvc, nvc++, nvfortran) in supplied paths * Add the nvhpc package which installs the nvhpc compiler * Add testing for nvhpc detection and C++-standard/pic flags Based on spack#19294 * Add CUDA@11 required for latest NVIDIA-HPC-SDK * Fix legacy apis : setup_environment to setup_run_environment * NEURON and CoreNEURON should use legacy units for BBP/HBP deployment (spack#947) * Update bbp-packages.yaml Adding my currentscape module to bbp-packages. * Update modules.yaml Adding my currentscape package to the whitelist. * Create package.py Added my currentscape package. * Update var/spack/repos/builtin/packages/py-currentscape/package.py Co-authored-by: Matthias Wolf <[email protected]> * Update var/spack/repos/builtin/packages/py-currentscape/package.py Co-authored-by: Matthias Wolf <[email protected]> * Update var/spack/repos/builtin/packages/py-currentscape/package.py Co-authored-by: Matthias Wolf <[email protected]> * Update package.py Modified the version. Is it better now? * Update package.py Changing tag to match the one from source on gerrit. * Update bbp-packages.yaml Updated currentscape version. * Update package.py Updated version & tag. Co-authored-by: Matthias Wolf <[email protected]> Co-authored-by: anilbey <[email protected]> Co-authored-by: Sergio <[email protected]> Co-authored-by: ppodhajski <[email protected]> Co-authored-by: Fernando Pereira <[email protected]> Co-authored-by: asanin-epfl <[email protected]> Co-authored-by: Nadir Román Guerrero <[email protected]> Co-authored-by: Tristan Carel <[email protected]> Co-authored-by: Pramod Kumbhar <[email protected]> Co-authored-by: Jaquier Aurélien Tristan <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I added a few memoizations to avoid re-doing work again and again. Not that this is how things should be in the end... but it DOES speed me up from 30 seconds to 15 for a simple "spack spec" command.
On my run...
It's true, virtual dependencies are taking a lot of time. _expand_virtual_packages() is only called 3 times. But then ProviderIndex() is instantiated 738 (!) times. Overall, 6838 specs are created --- FAR MORE than there are specs in the entire repo!
I suspect that speedup will happen through some more targeted re-use of objects. Among other things, it looks like the same Spec is being instantiated multiple times. And we shouldn't have to instantiate ProviderIndex 738 times.
In the meantime, I've shaved down the pain for myself somewhat.