Bugfix: IPOPT log parser and no objective case by mrmundt · Pull Request #3738 · Pyomo/pyomo

mrmundt · 2025-09-30T15:57:29Z

Fixes #3736

Summary/Motivation:

The issue above mentions two bugs that are related to IPOPT/parsing both the output and the sol files. This PR fixes both of those bugs and adds tests.

Changes proposed in this PR:

Make IPOPT parser more robust
Set duals and RCs to 0 when no objective is supplied
Add tests!

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

dallan-keylogic · 2025-09-30T16:22:34Z

pyomo/contrib/solver/solvers/ipopt.py

+        # If the NL instance has no objectives, report zeros
+        if not getattr(self._nl_info, "objectives", None):
+            vars_ = (
+                vars_to_load if vars_to_load is not None else self._nl_info.variable_map
+            )
+            return {v: 0.0 for v in vars_}


This fix is suitable if IPOPT converges to a feasible point, but if it converges to a point of local infeasibility, the values of the dual variables convey useful information about which bounds may be active at a solution.

Per your suggestion, though, if there are no objectives, these should be set to 0. So I don't think we change this. We capture the raw logs anyway, so folks can inspect those if they want more details.

Would it be better / more appropriate to return {} here? Unspecified duals / reduced costs are assumed to be 0, right?

@mrmundt IPOPT switches objectives when it enters into Restoration mode, using the L1 norm of the constraint violation as an objective. If it converges to an infeasible point with the exit message EXIT: Converged to a point of local infeasibility. then it is at a local minimum of the constraint violation, and we expect nonzero dual variables and reduced costs for this objective. I don't know if IPOPT makes these values available to AMPL, though.

I am not sure what to do with this information because I only understand what half of it means.

If IPOPT converges to an infeasible point, the dual variables and reduced costs are nonzero and contain useful information. I'm not sure whether IPOPT makes that information available via the interface you're using.

blnicho

It might be worth adding some additional tests for the log parser with different Ipopt print_level. We might just want to detect that a non-default print_level was used and add logic to not try to parse it. This doesn't have to hold up this PR but I was thinking about it in the context of making the parser more robust.

jsiirola · 2025-09-30T17:29:27Z

pyomo/contrib/solver/solvers/ipopt.py

+        # If the NL instance has no objectives, report zeros
+        if not getattr(self._nl_info, "objectives", None):
+            vars_ = (
+                vars_to_load if vars_to_load is not None else self._nl_info.variable_map
+            )
+            return {v: 0.0 for v in vars_}


Would it be better / more appropriate to return {} here? Unspecified duals / reduced costs are assumed to be 0, right?

pyomo/contrib/solver/solvers/ipopt.py

jsiirola · 2025-09-30T17:47:12Z

pyomo/contrib/solver/solvers/ipopt.py

+            vars_ = (
+                vars_to_load if vars_to_load is not None else self._nl_info.variables
+            )
+            return ComponentMap((v, 0.0) for v in vars_)


Is it more correct to just return an empty ComponentMap? Aren't missing entries assumed to be 0?

So I did this (both here and for get_duals), and it actually goes against what we say is a norm, surprisingly enough:

________________________________________ TestSolvers.test_no_objective_3_ipopt _________________________________________ a = (<pyomo.contrib.solver.tests.solvers.test_solvers.TestSolvers testMethod=test_no_objective_3_ipopt>,), kw = {} @wraps(func) def standalone_func(*a, **kw): > return func(*(a + p.args), **p.kwargs, **kw) ../venv-pyomo/lib/python3.12/site-packages/parameterized/parameterized.py:620: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <pyomo.contrib.solver.tests.solvers.test_solvers.TestSolvers testMethod=test_no_objective_3_ipopt> name = 'ipopt', opt_class = <class 'pyomo.contrib.solver.solvers.ipopt.Ipopt'>, use_presolve = False @parameterized.expand(input=_load_tests(all_solvers)) def test_no_objective( self, name: str, opt_class: Type[SolverBase], use_presolve: bool ): opt: SolverBase = opt_class() if not opt.available(): raise unittest.SkipTest(f'Solver {opt.name} not available.') check_duals = True if any(name.startswith(i) for i in nl_solvers_set): if use_presolve: check_duals = False opt.config.writer_config.linear_presolve = True else: opt.config.writer_config.linear_presolve = False m = pyo.ConcreteModel() m.x = pyo.Var() m.y = pyo.Var() m.a1 = pyo.Param(mutable=True) m.a2 = pyo.Param(mutable=True) m.b1 = pyo.Param(mutable=True) m.b2 = pyo.Param(mutable=True) m.c1 = pyo.Constraint(expr=m.y == m.a1 * m.x + m.b1) m.c2 = pyo.Constraint(expr=m.y == m.a2 * m.x + m.b2) params_to_test = [(1, -1, 2, 1), (1, -2, 2, 1), (1, -1, 3, 1)] for a1, a2, b1, b2 in params_to_test: m.a1.value = a1 m.a2.value = a2 m.b1.value = b1 m.b2.value = b2 res: Results = opt.solve(m) self.assertEqual(res.solution_status, SolutionStatus.optimal) self.assertAlmostEqual(m.x.value, (b2 - b1) / (a1 - a2)) self.assertAlmostEqual(m.y.value, a1 * (b2 - b1) / (a1 - a2) + b1) self.assertEqual(res.incumbent_objective, None) self.assertEqual(res.objective_bound, None) if check_duals: duals = res.solution_loader.get_duals() > self.assertAlmostEqual(duals[m.c1], 0) E KeyError: <pyomo.core.base.constraint.ScalarConstraint object at 0x11b9effc0> pyomo/contrib/solver/tests/solvers/test_solvers.py:994: KeyError

If I return an empty ComponentMap or an empty dict, then you can't actually access that item anymore.

Okay, empty ComponentMap is fine; it's empty dict for get_duals that causes these failures.

pyomo/contrib/solver/solvers/ipopt.py

pyomo/contrib/solver/solvers/sol_reader.py

jsiirola · 2025-09-30T18:11:24Z

pyomo/contrib/solver/solvers/sol_reader.py

+            cons = (
+                cons_to_load if cons_to_load is not None else self._nl_info.constraints
+            )
+            return {c: 0.0 for c in cons}


As with RC, is it more correct to return {}?

(I am happy to defer sorting out the duals / rc until the next PR (which will be a more substantive rework of the SOL parser)

pyomo/contrib/solver/solvers/sol_reader.py

Co-authored-by: John Siirola <[email protected]>

mrmundt · 2025-10-02T01:40:31Z

@AlexGisi - you pointed out a different missed thing in the IPOPT parser, though! We did only have lowercase s in the accepted chars for alpha_pr. I added that in just now. Thanks for pointing it out!

pyomo/contrib/solver/solvers/ipopt.py

pyomo/contrib/solver/solvers/sol_reader.py

pyomo/contrib/solver/solvers/ipopt.py

codecov · 2025-10-07T00:06:18Z

Codecov Report

❌ Patch coverage is 92.85714% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.31%. Comparing base (e8ed5dd) to head (31b6143).

Files with missing lines	Patch %	Lines
pyomo/contrib/solver/solvers/ipopt.py	90.90%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3738      +/-   ##
==========================================
- Coverage   89.31%   89.31%   -0.01%     
==========================================
  Files         896      896              
  Lines      103697   103728      +31     
==========================================
+ Hits        92619    92645      +26     
- Misses      11078    11083       +5

Flag	Coverage Δ
builders	`29.09% <3.57%> (+<0.01%)`	⬆️
default	`85.93% <92.85%> (?)`
expensive	`35.85% <3.57%> (?)`
linux	`86.96% <92.85%> (-2.09%)`	⬇️
linux_other	`86.96% <92.85%> (+<0.01%)`	⬆️
osx	`83.10% <92.85%> (+<0.01%)`	⬆️
win	`84.68% <92.85%> (-0.54%)`	⬇️
win_other	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pyomo/contrib/solver/solvers/ipopt.py

jsiirola

Two minor nits, but I'm not going to hold up the PR for them.

One question: is there a test that exercises the "token split" logic in the iteration log parser?

jsiirola · 2025-10-13T19:47:35Z

pyomo/contrib/solver/solvers/ipopt.py

+                try:
+                    idx = tokens[0].index('-')
+                    head = tokens[0][:idx]
+                    if head and head.rstrip('r').isdigit():


Why is this test needed? The only way for a - to be in tokens[0] is if the first two columns get merged...

jsiirola · 2025-10-13T19:48:44Z

pyomo/contrib/solver/solvers/ipopt.py

-                    continue
+                # IPOPT sometimes mashes the first two column values together
+                # (e.g., "2r-4.93e-03"). We need to split them.
+                try:


I'd still to prefer to not rely on the exception for the "normal" case where the columns are properly separated. That is, look for the -, and if it was found, then split the tokens.

jsiirola · 2025-10-13T19:50:29Z

pyomo/contrib/solver/solvers/sol_reader.py

+            cons = (
+                cons_to_load if cons_to_load is not None else self._nl_info.constraints
+            )
+            return {c: 0.0 for c in cons}


(I am happy to defer sorting out the duals / rc until the next PR (which will be a more substantive rework of the SOL parser)

mrmundt · 2025-10-14T18:47:54Z

Jenkins failures were related to #3750 , unrelated to anything else. So this is technically good to go.

Fix two parsing bugs

4d00144

mrmundt requested a review from jsiirola September 30, 2025 15:57

dallan-keylogic reviewed Sep 30, 2025

View reviewed changes

Remove usage of getattr

8032600

blnicho approved these changes Sep 30, 2025

View reviewed changes

jsiirola reviewed Sep 30, 2025

View reviewed changes

mrmundt and others added 5 commits September 30, 2025 12:41

Update pyomo/contrib/solver/solvers/ipopt.py

edf0528

Co-authored-by: John Siirola <[email protected]>

Update pyomo/contrib/solver/solvers/sol_reader.py

9e7c31b

Co-authored-by: John Siirola <[email protected]>

Address jsiirola's comments

0c7ffda

Return empty ComponentMap

b9b1453

Merge branch 'main' into ipopt-parser

00d9705

AlexGisi mentioned this pull request Oct 2, 2025

Fix two issues with ipopt output parsing #3741

Closed

S is in _ALPHA_PR_CHARS

cbc3293

mrmundt and others added 3 commits October 3, 2025 09:50

Merge branch 'main' into ipopt-parser

a9c3bf5

Add in tests for print levels (catching a new bug in the process)

b4fea1e

Residual debugging print stuff

89e021f

jsiirola reviewed Oct 6, 2025

View reviewed changes

Apply requested changes

393f61b

mrmundt requested a review from jsiirola October 6, 2025 19:23

mrmundt added 3 commits October 6, 2025 14:23

Add regex back in, fix incorrect assertion language

b5d8608

Not sure what's goiing on with windows

24ddea7

Try one more time

31b6143

mrmundt added 5 commits October 7, 2025 09:46

Print the output so I can figure out what's going on

2ab4ee8

Different print data because now the error has changed

385b872

Turn off NOWAIT

06674ed

Remove erroneous print statements

e615ddd

Log warnings instead of raising errors

94c2bcd

jsiirola added this to Pyomo 6.9.5 Release Oct 7, 2025

github-project-automation bot moved this to Todo in Pyomo 6.9.5 Release Oct 7, 2025

mrmundt moved this from Todo to Review In Progress in Pyomo 6.9.5 Release Oct 7, 2025

mrmundt added 2 commits October 7, 2025 13:58

Make the buffer msmall

4a52dfc

Introduce a labuffer increase context manager for the ipopt tests

56e8df6

jsiirola reviewed Oct 12, 2025

View reviewed changes

pyomo/contrib/solver/solvers/ipopt.py Outdated Show resolved Hide resolved

pyomo/contrib/solver/solvers/ipopt.py Outdated Show resolved Hide resolved

pyomo/contrib/solver/solvers/ipopt.py Outdated Show resolved Hide resolved

Address jsiirola comments

d9b15db

mrmundt requested a review from jsiirola October 13, 2025 19:37

jsiirola approved these changes Oct 13, 2025

View reviewed changes

github-project-automation bot moved this from Review In Progress to Reviewer Approved in Pyomo 6.9.5 Release Oct 13, 2025

mrmundt merged commit 740750d into Pyomo:main Oct 14, 2025
91 of 95 checks passed

github-project-automation bot moved this from Reviewer Approved to Done in Pyomo 6.9.5 Release Oct 14, 2025

mrmundt deleted the ipopt-parser branch October 14, 2025 20:22

Conversation

mrmundt commented Sep 30, 2025

Fixes #3736

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

Uh oh!

dallan-keylogic Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blnicho left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrmundt commented Oct 2, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsiirola left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrmundt commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dallan-keylogic Sep 30, 2025 •

edited

Loading

codecov bot commented Oct 7, 2025 •

edited

Loading

mrmundt commented Oct 14, 2025 •

edited

Loading