Repository navigation
AccessViolationException when execution code from multiple AppDomains #595
Description
Environment
- Pythonnet version: pythonnet 2.3.0
- Python version: python 3.6.1
- Operating System: Windows 10
Details
I am using PythonNet in a WCF application. The python code is executed from different appdomains.(IIs app pool recycling, different worker etc. ). First time the execution works but all of the other executions done on a different appdomain fail with System.AccessViolationException even if the initial domain was unloaded.
This simple code reproduces the issue:
class Program
{
static void Main(string[] args)
{
ExecuteNewAppDomain(new Action(PythonCode));
ExecuteNewAppDomain(new Action(PythonCode));
}
private static void PythonCode()
{
PythonEngine.Initialize();
using (Py.GIL())
{
var module = PythonEngine.ImportModule("numpy");
}
PythonEngine.Shutdown();
}
public static void ExecuteNewAppDomain(Action action)
{
AppDomain domain = null;
try
{
domain = AppDomain.CreateDomain("New App Domain: " + Guid.NewGuid());
var domainDelegate = (AppDomainDelegate)domain.CreateInstanceAndUnwrap(
typeof(AppDomainDelegate).Assembly.FullName,
typeof(AppDomainDelegate).FullName);
domainDelegate.Execute(action);
}
finally
{
if (domain != null)
AppDomain.Unload(domain);
}
}
private class AppDomainDelegate : MarshalByRefObject
{
public void Execute(Action action)
{
action();
}
}
}
System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt.' on the following line
var module = PythonEngine.ImportModule("numpy");
Activity
den-run-ai commentedon Jan 10, 2018
@AdamStefan perhaps this is because the native parts of pythonnet that load Python into .NET appdomain are not unloaded between AppDomains, because the native code is loaded into the process memory, not AppDomain:
https://stackoverflow.com/questions/10014821/c-dll-does-not-unload-with-appdomain
dmitriyse commentedon Jan 10, 2018
Also I found that numpy does not supports python engine load unload cycle. So, on a secondary python engine load - numpy import produces crash.
den-run-ai commentedon Jan 10, 2018
@dmitriyse have you tried importlib with numpy? Anyway, re-initialization of Python interpreter has issues even when using Python C-API from native code without .NET:
https://docs.python.org/3/c-api/init.html#c.Py_FinalizeEx
The Python interpreter cannot control all memory that gets loaded into process, and in that sense numpy has lots of "unmanaged" parts, e.g. BLAS/LAPACK/MKL.
This is confirmed here:
https://stackoverflow.com/questions/7676314/py-initialize-py-finalize-not-working-twice-with-numpy
Here is an issue where you can complain to fix this in numpy :) probably using lots of static variables in numpy codebase:
numpy/numpy#8097
den-run-ai commentedon Jan 29, 2018
closing this issue, because we can't really do anything about upstream numpy project. see linked issue.
JanKrivanek commentedon Feb 5, 2018
One way to solve this issue (if multiple app domains are needed) is by calling Initialize/Shutdown just once per process (can be ensured e.g. by cross app domain singleton).
Also you need to be carefull not to call PythonEngine.Initialize() (even indirectly - e.g. by using Py.GIL()) on other domains, as managed statics variables do not reflect state of main app domain.
So the repro code adjusted so that it doesn't crash any more looks like:
Side effect of this approach is that the engine - and also importet modules - stay in memory. That's why the 're-import' doesn't crash; but it also means that re-importing same module will not reflect code changes (which can be issue e.g. for hosting some ad-hoc scripting engine). I'm not sure how to tackle this (probably by using importlib.reload)
JanKrivanek commentedon Feb 5, 2018
So for reloading one can use PyImport_ReloadModule (exposed via PythonEngine.ReloadModule) - that however doesn't work recursively.
dmitriyse commentedon Feb 5, 2018
Probably for this case out-of-domain PythonEngine initialization support should be added to the pythonnet.
It's not so hard. We needs to introduce flag
And if this flag is set, Runtime.Py_Initialize() and Runtime.Py_Shutdown() should not be called.
dmitriyse commentedon Feb 5, 2018
Classic Asp.net and IIS is very hackish in it's nature, so this trick is not so dirty for IIS/Asp.Net environments.
JanKrivanek commentedon Feb 5, 2018
Flag would be nice; but can be misused sam as can be misused Initialization/Shutdown in multi-domain environemnt.
I'd say either leave it as is (so that it's not possible to screw-up in single app-domain scenario) or use cross-appdomain singleton for Engine.initialized flag (as than it would work correctly in both scenarios)