Conversation
|
The biggest challenge is finding a replacement for the memory_logger function in solve_network and solve_operations_network. The code from vresutils can be seen here https://github.com/FRESNA/vresutils/blob/master/vresutils/benchmark.py |
|
One could try a minimal memory_logger class that is put into import os
import time
import threading
import psutil
class MemoryLogger:
def __init__(self, filename, interval):
self.stop_event = threading.Event()
self.thread = threading.Thread(target=self.start, args=(filename, interval))
self.thread.start()
def __exit__(self, *args):
self.stop_event.set()
self.thread.join()
def calculate_memory(self, proc):
total_memory = proc.memory_info().rss
for child in proc.children(recursive=True):
total_memory += child.memory_info().rss
return total_memory
def start(self, filename, interval):
with open(filename, 'w') as f:
while not self.stop_event.is_set():
process = psutil.Process(os.getpid())
total_memory = self.calculate_memory(process)
f.write(f"{time.time()} {total_memory / 10**6}\n")
time.sleep(interval)Requires with MemoryLogger(filename='memory.log', interval=30.0):
computationally_demanding_function() |
|
Or one could use the memory logger from snakemake (via benchmark). But in general, I think it would be good to allow disabling it, as memory logging can slow down processes significantly. |
|
Agree! The advantage of the vresutils memory logger is that it monitors and logs RAM usage continuously, but honestly I am not sure how important it is... |
|
Looking into snakemake versions of benchmark https://github.com/snakemake/snakemake/blob/main/snakemake/benchmark.py, I believe snakemake has already well documented its resident set size (RSS), virtual memory size and proportional set size (PSS) through its automatic benchmark log. Let's just remove vresutils.benchmark from the code without replacing it. |
|
I'm happy with this proposal |
|
Looks great! Finally |
Changes proposed in this Pull Request
Remove the depreciated vresutils from pypsa-eur.
Affected scripts
Checklist
envs/environment.yaml.config.default.yaml.doc/configtables/*.csv.doc/release_notes.rstis added.