MPIEvaluator: Fix logging by explicitly passing logging level

EwoutH · EwoutH · commit 2eb434e2ae2f · 2023-11-02T20:18:11.000+01:00
Pass the log level explicitly to the run_experiment_mpi() function.

The level passed to log_to_stderr isn't available when the _logger object is initiated earlier. To migrate this problem, we explicitly pass it.

The initializer first took care of that, but since that wouldn't allow multiple consecutive calls to the MPIEvaluator it had to be removed again.

Now there are two sets of options:
- ema_logging.log_to_stderr(level=n, set_root_logger_levels=True) --&gt; Everything level n
- ema_logging.log_to_stderr(level=n, set_root_logger_levels=False) --&gt; Evaluator module level n, internal MPI / EMA process level 10 (default)

When level=10, set_root_logger_levels True or False doesn't matter.

Might have performance impact by setting up a logging.basicConfig for every experiment.
diff --git a/ema_workbench/em_framework/evaluators.py b/ema_workbench/em_framework/evaluators.py
@@ -443,8 +443,11 @@ def finalize(self):
     def evaluate_experiments(self, scenarios, policies, callback, combine="factorial"):
         ex_gen = experiment_generator(scenarios, self._msis, policies, combine=combine)
         experiments = list(ex_gen)
+        log_level = _logger.getEffectiveLevel()
 
-        packed = [(experiment, experiment.model_name, self._msis) for experiment in experiments]
+        packed = [
+            (experiment, experiment.model_name, self._msis, log_level) for experiment in experiments
+        ]
 
         _logger.info(
             f"MPIEvaluator: Starting {len(packed)} experiments using MPI pool with {self._pool._max_workers} workers"
@@ -462,7 +465,9 @@ def run_experiment_mpi(packed_data):
 
     rank = COMM_WORLD.Get_rank()
 
-    experiment, model_name, msis = packed_data
+    experiment, model_name, msis, level = packed_data
+
+    logging.basicConfig(level=level, format="[%(processName)s/%(levelname)s] %(message)s")
     _logger.debug(f"MPI Rank {rank}: starting {repr(experiment)}")
 
     models = NamedObjectMap(AbstractModel)
