remote debugging embedded interpreter

Apr 11, 2013 at 4:39 AM
I am trying to use ptvs to debug Python scripts and plugins for Luxology Modo (a 3D animation application). I am having only intermittent and unpredictable success using the Default Attach to Process Transport. I would like to try the new remote debugging tools in 2.0 but I have not been able to attach to Modo. When I look at what is happening on the Modo side, it seems that the server_thread daemon is not looping, so I wonder if the embedded interpreter is locking it out.
Thanks,
Ben
Coordinator
Apr 11, 2013 at 5:10 AM
From a brief glance at Modo wiki covering Python, it sounds like it spins up sub-interpreters for different scripts, and tears them down once the script is done. If that is the case, it would explain why regular attach does not work reliably, and it might also explain why you didn't have much luck with remote - have you been using ptvsd.wait_for_attach() to ensure that your script does not progress until the debugger is able to attach?
Apr 11, 2013 at 4:35 PM
Hi pminaev,
Thanks for the quick reply and for looking into it!
I have tried using ptvsd.wait_for_attach() without success.
For a little more detail, Modo has 2 ways of running Python code. There is an old "fire and forget" method of running scripts. I believe this is the method which uses the sub-interpreters. When I use this approach and wait_for_attach(), it seems like I can attach using remote debugging but then the breakpoints are not hit and Modo crashes when the script completes.
The new version of Modo (v701) has a persistent Python interpreter and a new deep Python API. This is actually the one I am most interested in using (although debugging the "fire and forget" scripts would be great too). Anyway, the new API allows us to create first-class plugins (servers since the SDK is based on COM) such as commands. When I try to use remote debugging by either calling enable_attach() from the plugin or from the interactive connection to the interpreter, I can't attach at all. I can see port 5678 opened up and listening but there is no response to attempts to attach. When I add a logging statement to the server_thread daemon, it only fires once. As an experiment I also tried connecting to 5678 via telnet and it just hanged until I closed Modo and then it connected and got the 'PTVSDBG' reply. So my guess is that the persistent interpreter is keeping the daemon from running for some reason.
I did accidentally find one method which sometimes works but I don't know why. If I run enable_attach() from the interactive interpreter and then attach using Default Transport, it will eventually attach after I run my command plugin a few times, but it is unpredictable how long it will take to attach.

Here is some documentation for Modo's new Python API: http://sdk.luxology.com/wiki/Python_API
Here is some discussion of attaching other debugging IDE's (I guess most of them have the listener on the debugger side): http://forums.luxology.com/topic.aspx?f=119&t=74845

Thanks again,
Ben
Apr 11, 2013 at 8:24 PM
Oh well, I got the following response from Luxology:
Python's threading module has never worked in modo (and still doesn't under the new API).
I'm still curious if there's a workaround to make it work, or possibly a more reliable way to connect with the Default transport.

Ben
Coordinator
Apr 11, 2013 at 8:43 PM
Is it specifically threading module (the high-level interface) that they don't support, or threads in general? i.e. is import _thread also unavailable / not working, or not? If _thread is working, it shouldn't be hard to reimplement the attach server in terms of that. The fact that other IDEs are able to debug seems to indicate that this is the case.

If they don't allow any threads, though, that would significantly complicate things. The attach server itself does not need a separate thread if it can block while waiting for the attach, but the debugger itself needs to have a background helper thread to process commands from VS. And the latter is necessary regardless of whether you're using default or remote transport. For the lack of threads, it might be possible to concoct some convoluted scheme with Py_AddPendingCall-based polling, but I doubt that would be reliable.
Apr 11, 2013 at 10:01 PM
OK I got it working on the embedded interpreter with your suggestion. I just changed the server_thread_func from a thread to a loop and added a 'break' at the end of the 'ATCH' elif block. It's not pretty and of course I can't detach reattach but the debugging seems to work just fine! The fire-and-forget interpreter still doesn't work but the problem there wasn't the same.

I guess you're right about the _thread module working then.

Thanks,
Ben
Coordinator
Apr 11, 2013 at 10:53 PM
Glad to hear it worked! Can you also try doing it the thread way, i.e. instead of:
server_thread = threading.Thread(target = server_thread_func)
server_thread.daemon = True
server_thread.start()
do:
thread.start_new_thread(server_thread_func, ())
and remove all mentions of _attached event (it's only needed if you want wait_for_attach to work, and I can figure something out to replace the use of threading.Event later, so long as _thread works...).

If this works, go ahead and open a feature request on me to change attach_server.py to replace all uses of threading with _thread (or, if you prefer, submit your code as a contribution - here's how!).
Apr 12, 2013 at 5:48 AM
Unfortunately it did not work to replace the threading module calls with the thread module. It seems to behave exactly the same as before. I certainly have no idea why threading works in the debugger but not in the attach function.

On the positive side, I now have the debugger working in the case of "fire and forget" scripts as well as plugins, using tips from people who have implemented other Python debuggers with Modo. So now I can use PTVS to debug everything I need to in Modo. It would be nice to have the attach daemon working but I'm happy using it this way. I guess I should just find out the cleanest way to run the server_thread_func as a loop before sharing the altered code with other Modo people.

Thanks,
Ben
Coordinator
Apr 12, 2013 at 6:22 AM
Ideally we want PTVS debugging to work out of the box with all commonly used apps that host Python for extensibility. So I'd really like to get to the bottom of it and figure out what needs to be done to make this happen, ideally before we ship 2.0, so that you can get the "it just works" experience out of the box with the final version - with some kind of graceful functionality degradation if necessary (e.g. the workaround you're currently using if there's no way to have the attach server running on the background thread after all).

Unfortunately, it looks like Luxology doesn't have a trial version yet that I could debug to find out what's going wrong. They promise a trial soon, though, so hopefully I'll get a chance to investigate in detail eventually.
Apr 12, 2013 at 3:18 PM
Yes, that would certainly be the ideal. For now I can offer to share a non-daemon version with any other Modo users who want to try it.

Ben
May 3, 2013 at 2:23 AM
Hi pminaev,
Luxology just released a trial version of Modo: http://www.luxology.com/trymodo/
Unfortunately, if you want the full trial with the SDK, it costs $25. Maybe you could contact them though.

Ben
Aug 5 at 10:14 AM
I've been trying to debug in Modo recently and I've discovered that attaching will work if you replace Modo's version of python27.dll with an official build from python.org.

Visual Studio will occasionally crash though, and Modo will crash if you try to use the debug interactive window. The latter problem appears to be because Modo requires all Python threads to call this method:

lx.service.Thread().InitThread()

Entering that into Debug Interactive before anything else seems to have stopped the Modo crashes. No idea about the VS crashes, which are intermittent. Here's a callstack for one:
    mscorlib.dll!System.ThrowHelper.ThrowArgumentOutOfRangeException() + 0x49 bytes 
    mscorlib.dll!System.Collections.Generic.List<Microsoft.PythonTools.Debugger.PythonThread>.this[int].get(int index) + 0x23 bytes 
    Microsoft.PythonTools.dll!Microsoft.PythonTools.Repl.PythonDebugProcessReplEvaluator.PythonDebugProcessReplEvaluator(Microsoft.PythonTools.Debugger.PythonProcess process) + 0x39 bytes 
>   Microsoft.PythonTools.dll!Microsoft.PythonTools.Repl.PythonDebugReplEvaluator.AttachProcess(Microsoft.PythonTools.Debugger.PythonProcess process) + 0x6d bytes  
    Microsoft.PythonTools.dll!Microsoft.PythonTools.Repl.PythonDebugReplEvaluator.OnEngineBreakpointHit(object sender, Microsoft.PythonTools.Debugger.DebugEngine.AD7EngineEventArgs e) + 0x10 bytes    
    Microsoft.PythonTools.Debugger.dll!Microsoft.PythonTools.Debugger.DebugEngine.AD7Engine.OnBreakpointBindSucceeded(object sender, Microsoft.PythonTools.Debugger.BreakpointEventArgs e) + 0xb7 bytes 
    Microsoft.PythonTools.Debugger.dll!Microsoft.PythonTools.Debugger.PythonProcess.HandleBreakPointSet(System.IO.Stream stream) + 0x56 bytes   
    Microsoft.PythonTools.Debugger.dll!Microsoft.PythonTools.Debugger.PythonProcess.DebugEventThread() + 0x283 bytes    
    mscorlib.dll!System.Threading.ThreadHelper.ThreadStart_Context(object state) + 0x6f bytes   
    mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0xa7 bytes  
    mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx) + 0x16 bytes  
    mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state) + 0x41 bytes    
    mscorlib.dll!System.Threading.ThreadHelper.ThreadStart() + 0x44 bytes   
    [Native to Managed Transition]  
Could it be related to the need to call InitThread()?
Coordinator
Aug 5 at 4:33 PM
No, this is actually a broader bug - we had a race in debug REPL attach code whereby it could end up trying to attach before the debugger has enumerated threads in the Python process, and crash because there was no thread on which it could run. It's rare in normal use, but some scenarios seem to trigger it much more reliably.

It's fixed now, but the fix hasn't been pushed out to CodePlex yet. That said, our usual dev build schedule is weekly, so there should be a build to test soon.