Unfortunately, this problem is not easy to solve in a general manner.

Background

By default, HydPy stores only the last calculated values for each sequence object. This means, after a simulation run has been perfomed, only the values at the end of the simulation period are available. The user has to tell HydPy, for which sequence object the whole simulated time series shall be stored. One could e.g. write (hp beeing the HydPy instance)...

hp.prepare_statesequences(ramflag=True)

...to keep the simulated values of all state sequence objects available in working memory.

If this would result in an exceedance of the maximum amount of random access memory, one can instead write...

hp.prepare_statesequences(ramflag=False)

....to keep the simulated values of all state sequence objects available in on disk.

Then each n-dimensional time series of a single state sequence object is stored in a single binary file. During simulation, all binary files are opened simultaniously. For a model with e.g. 10 different state sequences and applied in a project with 100 subcatchments, there would be 1000 files to be handled, which is not allowed on all systems.

Quick solutions

Be selective

Generally "activate" only those series which you are really interested in. Instead of activating all state series, you could only activate those of a sequence object x by writing:

for (name, element) in hp.elements:
    element.model.sequences.states.x.activate_ram() # or "activate_disk()"

Generally prefer to keep date in random access memory, which is much faster than storing data on disk.

Change your system

The maximum number of files to be worked with simultaniously depends on the system you are working on. The same is true for the available amount of rapid access memory. Through using a more current Python version (e.g. Python 3.6 / 64bit) you can use more rapid access memory than when using older versions (e.g. Python 2.7 / 32bit) and are eventually allowed to handle more files. I did not check this so far, but Unix systems generally seem to be less restrictive regarding the number of files than windows systems.

Modify your system

You might be allowed to increase the maximum number of files manually. On Windows you could eventually increase the value by writing:

import win32file
win32file._setmaxstdio(2048)

On Linux you could eventually set a much larger maximum number:

import resource
resource.setrlimit(resource.RLIMIT_NOFILE, (-1, -1))

Check what this number really is by writing:

import resource
resource.getrlimit(resource.RLIMIT_NOFILE)

General solution

Maybe it would be a wise choice, to merge all single binary files in one huge binary file or at least in a small number of relatively large binary files. This would require the sequence objects to become more dependend on each other, as some general instance would have to assign their data to a certain position in the associated binary file. One would have to make sure this would not affect the users freedom in handling and modifying time series data in HydPy.

IOError: [Errno 24] Too many open files #18

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions