Improvements to Pyomo component iteration by jsiirola · Pull Request #2829 · Pyomo/pyomo

jsiirola · 2023-05-14T23:57:05Z

Fixes # .

Summary/Motivation:

This PR cleans up how component iteration is implemented for Pyomo Components. In particular, it:

standardizes the meaning of .keys(sort=True) / .values(sort=True) / .items(sort=True) to be consistent with methods like component_data_objects(sort=True).
it reduces the amount of sorting done when walking "normal" Pyomo models (that use ordered sets)
it greatly simplifies the implementation of the Block DFS and BFS walkers
it converts 1-off flag classes to use standard Enums, and moves those enums to a new module to make importingg them without creating circular imports easier
it makes sorted iterators significantly more efficient for References
all Sets now support reversed()

Changes proposed in this PR:

(see above)

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

…t order

codecov · 2023-05-15T01:30:04Z

Codecov Report

Patch coverage: 98.82% and project coverage change: +0.02 🎉

Comparison is base (47deef1) 87.18% compared to head (47d8696) 87.21%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2829      +/-   ##
==========================================
+ Coverage   87.18%   87.21%   +0.02%     
==========================================
  Files         765      768       +3     
  Lines       88471    89952    +1481     
==========================================
+ Hits        77134    78449    +1315     
- Misses      11337    11503     +166

Flag	Coverage Δ
linux	`84.31% <98.82%> (+0.06%)`	⬆️
osx	`74.01% <98.82%> (+0.10%)`	⬆️
other	`84.48% <98.43%> (+0.06%)`	⬆️
win	`81.86% <98.82%> (+0.07%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pyomo/core/base/param.py	`81.69% <ø> (ø)`
pyomo/repn/beta/matrix.py	`74.17% <66.66%> (-0.14%)`	⬇️
pyomo/core/base/indexed_component.py	`91.21% <97.95%> (+0.72%)`	⬆️
...rib/cp/transform/logical_to_disjunctive_program.py	`96.61% <100.00%> (+0.05%)`	⬆️
pyomo/core/base/__init__.py	`100.00% <100.00%> (ø)`
pyomo/core/base/block.py	`92.40% <100.00%> (+0.95%)`	⬆️
pyomo/core/base/enums.py	`100.00% <100.00%> (ø)`
pyomo/core/base/global_set.py	`97.97% <100.00%> (+6.31%)`	⬆️
pyomo/core/base/indexed_component_slice.py	`97.15% <100.00%> (+0.06%)`	⬆️
pyomo/core/base/reference.py	`98.64% <100.00%> (+1.04%)`	⬆️
... and 2 more

... and 14 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

pyomo/core/base/enums.py

pyomo/core/base/global_set.py

pyomo/core/base/indexed_component.py

pyomo/core/base/param.py

mrmundt · 2023-05-22T20:11:18Z

pyomo/repn/beta/matrix.py


    def __delitem__(self):
        raise NotImplementedError
+


If it's not too much hassle, maybe add tests for these new methods?

Restore slice API broken by #2829

jsiirola added 21 commits May 7, 2023 10:07

Update _DeduplicateInfo.unique() to return a generator

c44bff3

Minor performance improvement for _component_data_itervalues()

43ac6ea

Update Block.block_data_objects() to return a generator

882adb7

Minor performance improvement for IndexedComponent.__getitem__

23e3e7c

Switch SortComponents, TraversalStrategy to enums

6331511

Define Set ordered_iter(), sorted_iter(), fix __reversed__

1b5512b

Simplify implementation of BFS traversal

3bcde32

Fix iteration of components indexed by unordered sets to match the se…

434166d

…t order

Deprecate component_data_iterindex

c15d65a

Move enums to a new module to break circular imports

0bddb74

Remove use of OrderedDict (only needed for Python<3.7)

fab3de9

Add support for ordered/sorted iteration to slices and references

dfe0f81

bugfix: MatrixConstraint must support extended dict API

6241c10

Improve keys/values/items support for sorting and references

6ae8212

Test ordered iteration of references

c9c0129

NFC: simplify SortComponents definition

815a0b5

bugfix: OrderedSetOf.ordered_iter() should not sort the set

6f232e1

rename 'ordered -> sort', deprecate ordered

605ab06

Fix typo

d1f944f

Apply black

882fd2e

Convert DFS iterators to use linked list and not list

0bd56c8

jsiirola added a commit to jsiirola/pyomo that referenced this pull request May 15, 2023

Adjuct baselines until Pyomo#2829 is merged

28c9409

blnicho self-requested a review May 16, 2023 18:42

jsiirola added 6 commits May 18, 2023 04:12

Add testing

c31da67

Fix typo in test file name

6d5d16c

Additional tests for code coverage

c20a435

Fix deduplication when iterating over items

a64d475

Add component deduplication tests

3ef4cd2

Fix ReferenceSet.ordered_iter()

6b9f598

jsiirola added 8 commits May 18, 2023 15:45

Remove unnecessary try-except

1fa7d93

NFC: apply black

b5d4a8d

Additional reference testing

136d8c7

Portability shim for Python 3.11

6b90f25

NFC: apply black

18b560e

Flushing out (and testing) more of the UnindexedComponent_set API

37d0007

Adding additional tests

aa6db0b

Merge branch 'main' into iter-improvements

5f94e76

mrmundt reviewed May 22, 2023

View reviewed changes

NFC: fix comment

12f4392

mrmundt approved these changes May 22, 2023

View reviewed changes

NFC: Fixing docstring typos in indexed_component.py

47d8696

blnicho approved these changes May 23, 2023

View reviewed changes

blnicho merged commit f471a97 into Pyomo:main May 23, 2023

jsiirola deleted the iter-improvements branch May 23, 2023 15:26

jsiirola added a commit to jsiirola/pyomo that referenced this pull request May 23, 2023

Updating tests now that Pyomo#2829 is merged

8a1db78

lbianchi-lbl mentioned this pull request May 25, 2023

IndexedComponent_slice wildcard methods argspec changes cause TypeError in 6.6.0 #2844

Closed

jsiirola added a commit to jsiirola/pyomo that referenced this pull request May 26, 2023

Fix backwards incompatible API change introduced by Pyomo#2829

c1369b5

blnicho added a commit that referenced this pull request May 26, 2023

Merge pull request #2849 from jsiirola/slice-wildcard-api

2cb3210

Restore slice API broken by #2829

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvements to Pyomo component iteration#2829

Improvements to Pyomo component iteration#2829
blnicho merged 37 commits intoPyomo:mainfrom
jsiirola:iter-improvements

jsiirola commented May 14, 2023

Uh oh!

codecov bot commented May 15, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrmundt May 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

jsiirola commented May 14, 2023

Fixes # .

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Uh oh!

codecov bot commented May 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mrmundt May 22, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented May 15, 2023 •

edited

Loading