BUG: Pyinstrument causes pickling error #109
Comments
ReproductionBreaking case:This version of the code runs fine if called without pyinstrument, but crashes when profiled. import pickle
class Simulation:
def __init__(self, duration_in_ticks):
self.now = 0
self.end = duration_in_ticks
def run(self):
while self.now < self.end:
print(f'Executing tick {self.now}')
self.now += 1
else:
with open('out.pickle', 'wb') as file:
pickle.dump(self, file)
if __name__ == "__main__":
sim = Simulation(10)
sim.run()Terminal, no crash: Pyinstrument, crash: Modified caseThis version disables the pickle.dump and then profiles fine class Simulation:
def __init__(self, duration_in_ticks):
self.now = 0
self.end = duration_in_ticks
def run(self):
while self.now < self.end:
print(f'Executing tick {self.now}')
self.now += 1
else:
with open('out.pickle', 'wb') as file:
# pickle.dump(self, file)
pass
if __name__ == "__main__":
sim = Simulation(10)
sim.run() |
|
Thanks for the recreation! I've played with it a little, and I'm not sure how to fix it! I copied the 'wrapping' code from built-in Python modules cProfile and profile, and they also have the same crash! perhaps in the |
|
Yes, that is the difference. dunder_main.py import sys
print(sys.modules['__main__'])$ pyinstrument dunder_main.py
<module '__main__' from '/Users/joerick/Projects/cibuildwheel/env/bin/pyinstrument'>$ python dunder_main.py
<module '__main__' from 'dunder_main.py'>I found this Stackoverflow answer that describes the issue perfectly. In that, they outline some hacky steps to fix. But actually I think this is a better solution to your problem - do as little as possible in your entrypoint file. Something like this- import yourmodule
yourmodule.main()I see the above as a fairly common pattern in Python generally, because things like relative imports are more sane once you're out of the entrypoint file also. |
|
Thank your for the research! I tested and your workaround succeeded, including pickling. Perhaps it's worth including this in a "Known Issues" or "Limitations" section? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I'm using pyinstrument to profile a simulation. The simulation consists of a singleton class
Simulationthat handles ticking and similar state logic, and when it finished it pickles self for inspection later. This is required because many different (fully independent) simulations are run in parallel, and analysis only happens after completion.Specifically, after the simulation stops it runs these lines:
Right now I'm profiling execution of a single Simulation without the multiprocessing. Pyinstrument works fine if I comment out the
pickle.dumpand replace it withpass. The error only occurs when running the script with pyinstrument, not during regular execution.Traceback:
I currently don't have time to write a minimum viable product, but will update this issue later.
System details:
The text was updated successfully, but these errors were encountered: