Open
Description
Environment
- Pythonnet version: 3.0.0.post1
- Python version: 3.8.10
- Operating System: Windows 11
- .NET Runtime: .NET Framework 472
Details
-
Describe what you were trying to get done.
I am getting random access violations during shutdown in pythonnet. The issue can be re-produced with a C# function as simple as
public static int Add(int a, int b) => a + b;
I get the access violation; however, it is not deterministic at all. Some times, it succeeds as expected, other times I get this. Note that I am upgrading from pythonnet 2.5.2 to pythonnet 3.0.0.post1 so this code used to work. All I am doing is invoking this code from python. I am running this python code that uses pythonnet in pytest. I haven't seen this when we were using pythonnet 2.5.2.
- If there was a crash, please include the traceback here.
Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at Python.Runtime.Runtime.PyObject_TYPE(BorrowedReference op)
at Python.Runtime.Runtime.NullGCHandles(IEnumerable`1 objects)
at Python.Runtime.Runtime.TryCollectingGarbage(Int32 runs, Boolean forceBreakLoops)
at Python.Runtime.Runtime.Shutdown()
at Python.Runtime.PythonEngine.Shutdown()
at Python.Runtime.Loader.Shutdown(IntPtr data, Int32 size)Metadata
Metadata
Assignees
Labels
No labels
Activity
filmor commentedon Oct 18, 2022
Can you please give the exact example that produces the error for you (Python and C# code)?
rohanjain101 commentedon Oct 18, 2022
We're using pythonnet in many different scenarios, and this issue seems to be happening randomly. This is the simplest example I can find that (occasionally) reproduces the failure
Note that we are not able to consistently repro the issue. We are running a series of unit tests using pytest, and every run, we get different tests failing with this same stack trace. Is there something specific we could be doing in our code that may cause this crash in the new release? I ask because we're getting this issue when upgrading to pythonnet3.0.0post1, we're currently on pythonnet2.5.2 and haven't had this issue.
filmor commentedon Oct 18, 2022
One major difference is that older versions of Python.NET didn't even try to properly clean up in the end. How are your tests set up that you get test failures due to shutdowns in them? Are they spinning up Python sub-processes? Can the issue be reproduced by just taking the example that you gave and running it a few thousand times or is this just an example from your test suite?
rohanjain101 commentedon Oct 18, 2022
This is an example from the test suite, I'll try running it a few thousand times to see if I can repro. We're just using pytest, AFAIK, it creates one python process, runs a set of tests sequentially, then terminates the python process. What's interesting is that the tests themselves pass, but when the python process is shutting down, that's where we start getting these random access violations.
Do we need to have certain cleanup code in between each test that runs in the process?
rohanjain101 commentedon Oct 19, 2022
Still working on a repro outside of our unit tests, is there anything from the stack trace that may help you identify if there's something we're doing that may cause this? Pytest runs all the tests in one python process I confirmed that, so the shutdown should only be happening once, shouldn't be spinning up python sub-processes.
donno commentedon Dec 1, 2022
Pythonnet version: 3.0.1
Python: 3.8 and 3.11.
I encountered a crash with the same traceback as the first post when running unittests with Python Development Mode activated (
-X dev)I was able to reproduce this crash outside of my tests with the following sample:
Save the above as issue1977.py and run with
python -X dev issue1977.pyon Windows 10 results in an access violation except.In my case I am not using System.Windows.Form but a different assembly and once I construct a class from that assembly after importing then I get this issue.
Python 3.7 doesn't have experience the same problem.
rohanjain101 commentedon Dec 1, 2022
Thanks @donno . Are you able to repro this crash deterministically or do you have the same issue as us where it seems to be happening randomly? @filmor is there anything we can do to avoid this crash?
donno commentedon Dec 2, 2022
For me, the above snippet is deterministic., it results in the System.AccessViolationException every time.
filmor commentedon Dec 2, 2022
The actual issue is not the dev mode as such, but that it uses the
debugmode for the allocator. If you just want to use all of the other warnings, you can explicitly force the non-debug one by setting thePYTHONMALLOC=defaultenvironment variable. I don't see the exact same backtrace, though, so my hunch is that this is a different bug:donno commentedon Dec 3, 2022
Ah, yes my traceback had the call to ReadInt32 rather than PyObject_TYPE from the first post.
Thanks for narrowing down my specific case and providing an alternative to still get the benefits I was aiming for with the dev mode.
maxcapodi78 commentedon Dec 6, 2022
@filmor I tried disabling the PYTHONMALLOC default but still get it (see below)
Should I do something before the fix is released to avoid the issue?
maxcapodi78 commentedon Dec 6, 2022
@filmor Ok I've realized that settings PYTHONMALLOC=malloc made the error disappear? So, maybe should be used this instead of default?
filmor commentedon Dec 6, 2022
None of these should be necessary, but running Python.NET with anything but the default allocation is undertested (as in, not at all). My guess is that somewhere in one of these debug hooks, some .NET code is evaluated and throws (across FFI boundaries). I'll have some time to debug this this weekend.
rohanjain101 commentedon Jan 6, 2023
@filmor just wondering if there's something we should do in the mean time to mitigate this issue?
jgmarcel commentedon Feb 7, 2023
I confirm I am also experiencing this error:
Unfortunately I cannot provide you with the C# code that triggers the problem because I did not write it myself. My Python code that calls that code is very much similar to what @donno posted above.
I am running python 3.9.16 and pythonnet 3.0.1. The error does not manifest itself in version 2.5.2.
15 remaining items
siegfriedpammer commentedon Jun 3, 2023
Yes, please add an option to disable the shutdown code completely. I have had cases where simply
Ctrl+Cing the process caused a .NET stack trace to be printed, which should not be the case. I have tried monkey-patching thepythonnet.unloadfunction, but it made everything worse, that is, even more annoying stack traces printed to the console.filmor commentedon Jun 4, 2023
agoodwin commentedon Sep 6, 2023
Hi @filmor, I had a look for the
AppDomainswitching ticket you were referring to in your third point, would I be right to understand that it's these ones: #1781 and #2008?And just to clarify, would I be right to understand that when those tickets are resolved, we should expect to have an option to disable
AppDomainsupport (or, it would just always be disabled in the use-case of using Python.NET to invoke .NET code from Python), and if this was to be done then we should then expect this "random access violation on shutdown" behaviour to simply disappear as a nice side-benefit?vreindelESZ commentedon Sep 25, 2023
@filmor I have the same Problem. This code does NOT create any error messages and always works perfectly fine:
but this code which is just a loop always produces the error:
(German) errormessage:
Unbehandelte Ausnahme: System.AccessViolationException: Es wurde versucht, im geschützten Speicher zu lesen oder zu schreiben. Dies ist häufig ein Hinweis darauf, dass anderer Speicher beschädigt ist.
bei Python.Runtime.Runtime.PyObject_TYPE(BorrowedReference op)
bei Python.Runtime.Runtime.NullGCHandles(IEnumerable`1 objects)
bei Python.Runtime.Runtime.TryCollectingGarbage(Int32 runs, Boolean forceBreakLoops)
bei Python.Runtime.Runtime.Shutdown()
bei Python.Runtime.PythonEngine.Shutdown()
bei Python.Runtime.Loader.Shutdown(IntPtr data, Int32 size)
doctormachine commentedon Oct 31, 2023
To avoid the above exception error, I have removed the below lines from the Runtime.cs file:
if (!HostedInPython && !ProcessIsTerminating)
{
// avoid saving dead objects
TryCollectingGarbage(runs: 3);
duyang76 commentedon Mar 6, 2024
We get similar issue in our application as well. Here are more details of our use case:
I spent a lot of time to go over the pythonnet code, and debugging info into pythonnet, step through the execution in Visual Studio. I think I have a better understanding of the pythonnet implementation (which is complicated but elegant) and may find something relevant (at least to our use case).
The code below illustrates that the allocated memories are reused later. If the code is complicated enough to cause the heap to shrink, we can expect the exception to happen.
Here is the output. I am not sure what is the correct way to measure ref count of a python object created by the .net code, so I tried all 3 ways I am aware of. They return different results, but we can see the pattern here: when the objects are still in use, the .net ref count is 1 less than python ref count; when the objects are not in use, the .net ref count is kind of random (which indicates that the memory is reused).
Possible fix? I tried to increase the ref count by 1 after the object is created.
It seems to work (no more exceptions), but it also means that these allocated objects will stay in the heap forever. Experts, any suggestion for a more robust fix?
Frawak commentedon May 7, 2024
I am also struggling with this issue. I am trying to dig into it. @duyang76 has the right idea that the problem is something in regards to the reference count but the solution
is not correct because
var py = Runtime.PyType_GenericAlloc(tp, 0);already increases the reference count for the object undertp.However, in the method
NullGCHandles(CLRObject.reflectedObjects)within theShutdown(when the exception occurs) all references inCLRObject.reflectedObjectshave a reference count of zero.The question is: Why?
The
NewReferencewhich are created with thatvar py = Runtime.PyType_GenericAlloc(tp, 0);and whoseIntPtrare added toCLRObject.reflectedObjectsmust have been disposed (along with all other references to the object) while these pointers remain.And (this is my intepretation) because of the zero reference count the memory is opened up to be reclaimed.
EDIT: Basically related to this comment: #1977 (comment)
@lostmsu But the crux is that the exception is thrown while trying to null the GC Handles. Why is this done after the reference count is decremented to zero?
Fix access violation exception on shutdown (pythonnet#1977)
Fix access violation exception on shutdown (pythonnet#1977)
Fix access violation exception on shutdown (pythonnet#1977)
Fix access violation exception on shutdown (pythonnet#1977)
Fix access violation exception on shutdown (pythonnet#1977)
Frawak commentedon Oct 11, 2024
This issue can be closed, right?
I tested 3.0.4 again.
Panzerfury commentedon Oct 21, 2024
I still get:
Exit code is -1073741819 (Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at Python.Runtime.Runtime.PyErr_Clear()
at Python.Runtime.Exceptions.Clear()
at Python.Runtime.PythonEngine.Initialize(System.Collections.Generic.IEnumerable
1<System.String>, Boolean, Boolean) at Python.Runtime.PythonEngine.Initialize(Boolean, Boolean) at Python.Runtime.PythonEngine.Initialize() at Program.<Main>$(System.String[]) at DynamicClass.InvokeStub_Program.<Main>$(System.Object, System.Span1<System.Object>)at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)
at System.Reflection.MethodBase.Invoke(System.Object, System.Object[])
at Microsoft.Extensions.Hosting.HostFactoryResolver+HostingListener.b__10_0())
In my tests of my C# web API.
I initialize my python in my program.cs
var path = builder.Configuration.GetSection("PythonDllLocation").Value;
Runtime.PythonDLL = path;
PythonEngine.Initialize();
PythonEngine.BeginAllowThreads();
And my OnShutDown(){
AppContext.SetSwitch("System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization", true);
PythonEngine.Shutdown();
AppContext.SetSwitch("System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization", false);
}
Each integrationtest seems to spin up a program.cs on it's own (i didn't write the test suite).
Before I had my Initialize and shutdown in my service class. Which seemed to work fine. But it was rather slow, having to spin up the python engine on each request.
Frawak commentedon Oct 21, 2024
That callstack is on initialization. I would suggest a separate issue.