[cxxmodules] Use the global module index only when no rootmap candidate is found by vgvassilev · Pull Request #9592 · root-project/root

vgvassilev · 2022-01-16T07:57:54Z

In the hybrid mode when we use ROOT with modules and third party software without modules we have two name resolution systems: the global module index (GMI) and the rootmaps. In case an identifier is defined in both (eg TMVA::Event and X::Event), the GMI will take priority and load the TMVA module without later allowing the system to look for other candidates.

This patch uses the GMI only when no rootmap file has told ROOT that it provides names for the given namespace.

This fixes #9583

phsft-bot · 2022-01-16T07:58:02Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, ROOT-ubuntu2004/soversion, mac1015/python3, mac11/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2022-01-16T07:59:02Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, ROOT-ubuntu2004/soversion, mac1015/python3, mac11/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2022-01-16T08:29:57Z

Build failed on ROOT-ubuntu2004/soversion.
Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.core.metacling.test.gtest_core_metacling_test_TClingTest

phsft-bot · 2022-01-16T08:32:14Z

Build failed on ROOT-performance-centos8-multicore/default.
Running on olbdw-01.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See console output.

Errors:

[2022-01-16T08:19:33.952Z] 53/2226 Test Rename print_wrapper to get_wrapper_code. #642: tutorial-hist-fillrandom ..........................................................................***Failed Error regular expression found in output. Regex=[: error:] 0.42 sec
[2022-01-16T08:20:53.334Z] 886/2226 Test Add support for LZ4 in classic build #593: tutorial-graphs-gerrors ...........................................................................***Failed Error regular expression found in output. Regex=[: error:] 1.11 sec
[2022-01-16T08:20:53.334Z] 889/2226 Test Change Snapshot option from file cache to TTree auto-flush parameter #595: tutorial-graphs-gmultierrors ......................................................................***Failed Error regular expression found in output. Regex=[: error:] 1.08 sec
[2022-01-16T08:20:53.603Z] 890/2226 Test Refactor of test/stressMathCore.cxx to use Google Test #594: tutorial-graphs-gerrors2 ..........................................................................***Failed Error regular expression found in output. Regex=[: error:] 1.19 sec
[2022-01-16T08:20:53.603Z] 891/2226 Test gROOTMutex -> RWLock #596: tutorial-graphs-graph .............................................................................***Failed Error regular expression found in output. Regex=[: error:] 1.18 sec

Failing tests:

phsft-bot · 2022-01-16T09:04:42Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft11.dyndns.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

projectroot.core.metacling.test.gtest_core_metacling_test_TClingTest

phsft-bot · 2022-01-16T09:12:55Z

Build failed on mac11/cxx17.
Running on macphsft23.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Errors:

[2022-01-16T08:34:27.071Z] 66/2328 Test Revert "Work around vtable index issue due to base class definition b… #682: tutorial-hist-fillrandom ..........................................................................***Failed Error regular expression found in output. Regex=[: error:] 0.61 sec
[2022-01-16T08:39:52.619Z] 749/2328 Test Allow for globbing on Google Test sources #633: tutorial-graphs-gerrors ...........................................................................***Failed Error regular expression found in output. Regex=[: error:] 0.71 sec
[2022-01-16T08:39:52.886Z] 750/2328 Test Add support for named arguments #634: tutorial-graphs-gerrors2 ..........................................................................***Failed Error regular expression found in output. Regex=[: error:] 0.65 sec
[2022-01-16T08:39:52.886Z] 751/2328 Test Remove float.h from the libc.modulemap. #635: tutorial-graphs-gmultierrors ......................................................................***Failed Error regular expression found in output. Regex=[: error:] 0.66 sec
[2022-01-16T08:39:52.886Z] 752/2328 Test [TypeTraits] Move general-purpose metaprogramming utilities from TDataFrame's utils to ROOT TypeTraits #636: tutorial-graphs-graph .............................................................................***Failed Error regular expression found in output. Regex=[: error:] 0.61 sec

Failing tests:

And 12 more

phsft-bot · 2022-01-16T09:31:33Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, ROOT-ubuntu2004/soversion, mac1015/python3, mac11/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2022-01-16T09:55:08Z

Build failed on ROOT-ubuntu2004/soversion.
Running on root-ubuntu-2004-1.cern.ch:/home/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

phsft-bot · 2022-01-16T09:55:30Z

Build failed on ROOT-debian10-i386/cxx14.
Running on pcepsft10.dyndns.cern.ch:/build/workspace/root-pullrequests-build
See console output.

Failing tests:

phsft-bot · 2022-01-16T09:56:16Z

Build failed on ROOT-performance-centos8-multicore/default.
Running on olbdw-01.cern.ch:/data/sftnight/workspace/root-pullrequests-build
See console output.

Failing tests:

…te is found In the hybrid mode when we use ROOT with modules and third party software without modules we have two name resolution systems: the global module index (GMI) and the rootmaps. In case an identifier is defined in both (eg TMVA::Event and X::Event), the GMI will take priority and load the TMVA module without later allowing the system to look for other candidates. This patch uses the GMI only when no rootmap file has told ROOT that it provides names for the given namespace. This fixes root-project#9583

phsft-bot · 2022-01-16T09:59:44Z

Starting build on ROOT-debian10-i386/cxx14, ROOT-performance-centos8-multicore/default, ROOT-ubuntu16/nortcxxmod, ROOT-ubuntu2004/soversion, mac1015/python3, mac11/cxx17, windows10/cxx14
How to customize builds

phsft-bot · 2022-01-16T11:10:44Z

Build failed on mac11/cxx17.
Running on macphsft23.dyndns.cern.ch:/Users/sftnight/build/workspace/root-pullrequests-build
See console output.

Failing tests:

And 4 more

pcanal · 2022-01-17T13:04:28Z

@vgvassilev I see In case an identifier is defined in both (eg TMVA::Event and X::Event), the GMI will take priority and load the TMVA module without later allowing the system to look for other candidates.. Could you explicit in the commit log why an unrelated fully qualified name (TMVA::Event) takes precedence over another fully qualified name (X::Event) and why this is usually not an issue (i.e. Is there any other cases (other than module vs rootmap) where this could lead to problems?)

vgvassilev · 2022-01-17T21:02:32Z

@vgvassilev I see In case an identifier is defined in both (eg TMVA::Event and X::Event), the GMI will take priority and load the TMVA module without later allowing the system to look for other candidates.. Could you explicit in the commit log why an unrelated fully qualified name (TMVA::Event) takes precedence over another fully qualified name (X::Event)

The GMI contains Identifier->list_of_modules mapping. This means that a lookup for Event will give all modules in a fully modularized system. However, until today, we have not considered (nor had the tests for) the hybrid approach where parts of the names are resolved from a rootmap files.

vgvassilev · 2022-01-19T08:04:56Z

@smuzaffar, could you test this PR for CMSSW as it would likely end up in v6-26 and is in the core of ROOT name resolution mechanism.

cc: @davidlange6

smuzaffar · 2022-01-20T13:02:32Z

@vgvassilev , cmssw tests for latest root master + this change are now running via cms-sw/cmsdist#7558

smuzaffar · 2022-01-20T23:08:36Z

@vgvassilev , this lookgs good in cmssw root master branch based IBs.

vgvassilev · 2022-01-21T08:23:40Z

@smuzaffar, thanks a lot for the prompt feedback!

smuzaffar · 2022-01-21T08:38:38Z

@vgvassilev , I also tested this for our CXXMODULE IBs and most of relvals failed with errors like

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d3a1b7/21842/runTheMatrix-results/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/step1_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS.log

----- Begin Fatal Exception 21-Jan-2022 04:35:59 CET-----------------------
An exception of category 'DictionaryNotFound' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=BeamHaloProducer label='generator'
   [2] Calling ProductRegistryHelper::addToRegistry, checking dictionaries for produced types
Exception Message:
No data dictionary found for the following classes:

  map<string,ULong64_t>

Most likely each dictionary was never generated, but it may
be that it was generated in the wrong package. Please add
(or move) the specification '<class name="whatever"/>' to
the appropriate classes_def.xml file along with any other
information needed there. For example, if this class has any
transient members, you need to specify them in classes_def.xml.
Also include the class header in classes.h

A type listed above might or might not be the same as a
type declared by a producer module with the function 'produces'.
Instead it might be the type of a data member, base class,
wrapped type, or other object needed by a produced type. Below
is some additional information which lists the types declared
to be produced by a producer module that are associated with
the types whose dictionaries were not found:

  edm::HepMCProduct

----- End Fatal Exception -------------------------------------------------

this is a different error than what we get in out CXXMODULE IBs https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/raw/slc7_amd64_gcc10/CMSSW_12_3_CXXMODULE_X_2022-01-19-2300/pyRelValMatrixLogs/run/135.1_TTbar_13+TTbarFS_13+HARVESTUP15FS+MINIAODMCUP15FS/step3_TTbar_13+TTbarFS_13+HARVESTUP15FS+MINIAODMCUP15FS.log ?

vgvassilev · 2022-01-21T08:54:45Z

@smuzaffar, indeed, now the failure looks very basic.

@davidlange6, I remember we've seen such a failure but did we manage ever to solve it?

Axel-Naumann

LGTM!

davidlange6 · 2022-01-26T18:38:47Z

On Jan 21, 2022, at 9:54 AM, Vassil Vassilev ***@***.***> wrote: @smuzaffar, indeed, now the failure looks very basic. @davidlange6, I remember we've seen such a failure but did we manage ever to solve it?

No, we didn't... though the only dictionary not found errors we have now are of a different sort (lets see if they go away)

…

— Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android. You are receiving this because you were mentioned.

vgvassilev requested review from Axel-Naumann and pcanal as code owners January 16, 2022 07:57

vgvassilev force-pushed the lower-gmi-priority branch from b5b4eb4 to 122eb25 Compare January 16, 2022 07:58

vgvassilev force-pushed the lower-gmi-priority branch from 122eb25 to be8fce4 Compare January 16, 2022 09:31

vgvassilev force-pushed the lower-gmi-priority branch from be8fce4 to e94089f Compare January 16, 2022 09:59

vgvassilev mentioned this pull request Jan 17, 2022

Interpreter autoload lookup failure when runtime_cxxmodules=ON #9583

Closed

1 task

jalopezg-git assigned vgvassilev Jan 17, 2022

jalopezg-git added the in:C++ modules label Jan 17, 2022

Axel-Naumann approved these changes Jan 26, 2022

View reviewed changes

vgvassilev merged commit 84cc694 into root-project:master Jan 26, 2022

Conversation

vgvassilev commented Jan 16, 2022

Uh oh!

phsft-bot commented Jan 16, 2022

Uh oh!

phsft-bot commented Jan 16, 2022

Uh oh!

phsft-bot commented Jan 16, 2022

Failing tests:

Uh oh!

phsft-bot commented Jan 16, 2022

Errors:

Failing tests:

Uh oh!

phsft-bot commented Jan 16, 2022

Failing tests:

Uh oh!

phsft-bot commented Jan 16, 2022

Errors:

Failing tests:

Uh oh!

phsft-bot commented Jan 16, 2022

Uh oh!

phsft-bot commented Jan 16, 2022

Failing tests:

Uh oh!

phsft-bot commented Jan 16, 2022

Failing tests:

Uh oh!

phsft-bot commented Jan 16, 2022

Failing tests:

Uh oh!

phsft-bot commented Jan 16, 2022

Uh oh!

phsft-bot commented Jan 16, 2022

Failing tests:

Uh oh!

pcanal commented Jan 17, 2022

Uh oh!

vgvassilev commented Jan 17, 2022

Uh oh!

vgvassilev commented Jan 19, 2022

Uh oh!

smuzaffar commented Jan 20, 2022

Uh oh!

smuzaffar commented Jan 20, 2022

Uh oh!

vgvassilev commented Jan 21, 2022

Uh oh!

smuzaffar commented Jan 21, 2022

Uh oh!

vgvassilev commented Jan 21, 2022

Uh oh!

Axel-Naumann left a comment

Choose a reason for hiding this comment

Uh oh!

davidlange6 commented Jan 26, 2022 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants