Mixed-Mode debugging with namespaced Python?

Dec 23, 2014 at 4:48 PM
Hello,

Had a quick question for which I couldn't find any answers in the documentation or discussion threads:

Can I do mixed-mode python/C++ debugging with a namespaced python 2.7 interpreter?

I have a namespaced build of Python 2.7. I just do a standard build of python 2.7, then after it's built go through the binary and change the names from "Py..." to "Px..." for most symbols. Also python27.dll is named pxthon27.dll. My extension module (written in C++) is then linked against this namespaced python.

Question: can I do mixed-mode debugging with this python interpreter? Does PTVS rely on recognizing any particular symbols within the interpreter binary? I've been trying, but can't seem to get it to transition from one language to another. I can successfully debug either the C++ or Python separately but not mixed. Is it possible?

If it helps, I have some flexibility in which symbols are renamed. For example if you said that "you could do mixed-mode debugging so long as this or that particular symbol are not namespaced" then I could work with that. Also I could potentially make modifications to any PTVS config files or binaries (but hoping not to have to do these things).

Any help is appreciated, thank you
David
Coordinator
Dec 23, 2014 at 9:00 PM
Generally speaking, this is an unsupported scenario. The implementation does look up a bunch of functions, types and global statics by name (hence the need for symbols), and renaming any of those risks breakage. The exact set is implementation-defined and is likely to change from version to version.

If you still want to have a definitive list (either to know what you shouldn't touch, or to change the code to reflect your renames), you can do so based on the source code. All the mixed-mode debugging stuff is under Python\Product\Debugger\DkmDebugger. Here are the things that you'll need to look at:

Types - look under Proxies\Structs. Every class in there inheriting from PyObject (+ PyObject itself) will try to look up the correspondingly named struct definition in the symbols. Furthermore, it will also use the global static variable to retrieve the corresponding PyTypeObject instance - e.g. for PyLongObject, it will look for a global variable named PyLong_Type. Some of those have [PyType] attributes applied to them which override the name of that variable - e.g. on PyObject, it's [PyType(VariableName = "PyBaseObject_Type")].

For other variables, search for uses of GetStaticVariable<...> in the code that are called with string literals

For functions, search for GetFunctionAddress called with string literals, and also CreateRuntimeDllFunctionBreakpoint and CreateRuntimeDllFunctionExitBreakpoints.

Finally, search for methods decorated with [StepInGate] attribute - each of those corresponds to a Python function with the same name.

I believe this should cover everything in global scope. There's also a bunch of uses of struct fields, which are much harder to track (because a lot of them are in form of string literals that are fed to the C++ expression evaluator at runtime), but from what you've described, these shouldn't be affected in your case.
Dec 23, 2014 at 9:01 PM
Correction: when I enter debug mode (no matter what settings I try) I can debug the C++ part but not the python part. So basically my C++ breakpoints are obeyed by my python breakpoints are not (the icon is not filled). Is is possible that PTVS cannot debug python itself if the interpreter is namespaced?
Dec 23, 2014 at 9:09 PM
Ok thank you for the reply, I'll take a look into changing those things-
Coordinator
Dec 23, 2014 at 9:23 PM
Thinking a bit more about it, it may actually be easiest for you to change the code to basically apply the same rename operation that you did to Python source, at runtime. This is because all those things that I've listed above ultimately go through one of the two points to obtain the symbol information: DiaExtension.GetSymbol or DiaExtension.GetTypeSymbol. You could just add a snippet to both of those that inspects the name, and if it matches the pattern, rename it - e.g. Py* to Px*.
Dec 23, 2014 at 11:02 PM
Ok that sounds even better, I'll give that a try! Thanks again
Jan 4, 2015 at 9:56 PM
Hi, do I get it right, that currently PTVS debugger cannot jump from python code to C++ swig compiled module code?
dpacbach, did you have any luck in implementing mixed-mode debugging?
Coordinator
Jan 4, 2015 at 11:56 PM
It can jump to any kind of C or C++ code, it doesn't really care if it's generated or not. The problem with Cython, SWIG, Boost.Python etc is that you'll end up in the first function where control flow enters after it leaves the Python DLL, and that's usually some fancy templated or generated stuff converting the arguments etc; you'll have to step through all of it to end up in the code that you actually wrote.

It should be possible to tell the C++ debugger to skip through it using this:
http://blogs.msdn.com/b/vcblog/archive/2013/06/26/just-my-code-for-c-in-vs-2013.aspx
But I haven't experimented with it yet.
Jan 5, 2015 at 4:56 PM
Update: Success

I made three small changes to the code and succeeded in getting mixed-mode debugging with a namespaced python (by namespace I mean that the symbols in the python dll are renamed from Py* to Px*, and the python dll itself is renamed to pxthon??.dll):

1) Added a snippet of code to DiaExtension.GetSymbol and DiaExtension.GetTypeSymbol to rename the symbols that are requested from the python dll from Py* to Px* as suggested by pminaev
2) Changed "Py_InitializeEx" to "Px_InitializeEx" in LocalComponent.InjectHelperDll
3) Changed the regex in PythonDLLs.pythonName to look for pxthon??.dll
Jan 6, 2015 at 6:42 PM
Hi pminaev,
I think I'm almost there: I can now start from either C++ or python and set breakpoints in either C++ or python and then step through. But I'm not yet able to step into C++ code from python code. When I attempt to do that it will be equivalent to just having pressed "continue," so it will just run and won't stop until my next break point which could be in either C++ or python. I'm guessing there may be some issue with the StepInGate part of the code? Something that I was not sure about was whether I have to modify the C expression strings inside the [StepInGate] functions? I tried looking into how those are evaluated but coudn't find where in the code it happens... Any ideas? Thank you!

Regards,
David
Coordinator
Jan 6, 2015 at 7:01 PM
Step-in gates are basically methods that are handlers for breakpoints that are set on the correspondingly named functions in Python source. There's a bunch of magic there that uses Reflection to do the mapping. Consequently, the names have to match - for example, there is a step-in gate called PyCFunction_Call, which you need to rename to PxCFunction_Call for your customized interpreter.

Again, instead of renaming every method, you can change the code that creates breakpoints for them and automatically adjust Py->Px. That code is at the very end of the constructor of TraceManagerLocalHelper:
                foreach (var methodInfo in _handlers.GetType().GetMethods()) {
                    var stepInAttr = (StepInGateAttribute)Attribute.GetCustomAttribute(methodInfo, typeof(StepInGateAttribute));
                    if (stepInAttr != null &&
                        (stepInAttr.MinVersion == PythonLanguageVersion.None || _pyrtInfo.LanguageVersion >= stepInAttr.MinVersion) &&
                        (stepInAttr.MaxVersion == PythonLanguageVersion.None || _pyrtInfo.LanguageVersion <= stepInAttr.MaxVersion)) {

                        var handler = (StepInGateHandler)Delegate.CreateDelegate(typeof(StepInGateHandler), _handlers, methodInfo);
                        AddStepInGate(handler, _pyrtInfo.DLLs.Python, methodInfo.Name, stepInAttr.HasMultipleExitPoints);
                    }
                }
Note where it passes methodInfo.Name to AddStepInGate - that's where you can adjust it.

I don't think you need to worry about C expression strings inside the methods. These should only be used for locals and struct fields, and you haven't renamed those. They are evaluated by passing them to the C++ debugger - you can treat CppExpressionEvaluator as a black box encapsulating that. If you wanted to adjust names of those fields, I suppose you could do it in the implementation of CppExpressionEvaluator, but you'd need to parse the expressions somehow, which would be much more complicated - right now it just passes the string to the C++ debugger as is.
Feb 3, 2015 at 8:22 PM
Hi pminaev,

I've gotten to the point where I can basically do everything: I can step through python or C++, I can set breakpoints in either, and I can step from C++ into python. Everything seems to work except for one thing: stepping from python into C++ (this is of course in mixed-mode).

So I inserted quite a few debug statements in the PTVS code to see what's happening, and everything appears ok:

1) I hit "step in" on the line of python code which is direct call to a boost::python (C++) method
2) All of the stepin gate break points are added (~ 20 of them)
3) Then it arrives in the PyObject_Call stepingate handler in TraceManagerLocalHelper.cs and evaluates the C++ expression func->ob_type->tp_call yielding a non-null value
4) Calls OnPotentialRuntimeExit and then reaches the line where it will create the breakpoint (CreateBreakpoint) -- meaning that the pointer is referencing code outside of the interpreter, which I think is correct in this case)
5) bp.Enable() runs without throwing an exception

All of this seems to be fine... except that it never stops at the breakpoint... is just keeps running. Do you know what could cause this? I don't have debug symbols for the boost::python module where it should step into... is that a problem? How does the mixed-mode debugger behave if a step-in target is in a C++ module with no debug symbols?

In any case, the boost::python function should then call my own C++ code which is where I eventually want to go... but it seems that it is simply not respecting the the step-in-target-breakpoint even though it appears to be created correctly. Can you think of any reason for this behavior? (FYI ctypes should not be involved here)

Thank you!
David
Coordinator
Feb 3, 2015 at 8:34 PM
Thank you for investigating it so deeply!
All of this seems to be fine... except that it never stops at the breakpoint... is just keeps running. Do you know what could cause this? I don't have debug symbols for the boost::python module where it should step into... is that a problem? How does the mixed-mode debugger behave if a step-in target is in a C++ module with no debug symbols?
Since that low-level breakpoint is set directly at a given native address, rather than at a source line, I would expect it to be hit regardless of having or not having symbols. However, once it is hit, the Python debug engine will basically assume that execution flow has left its boundaries, and tell the debugger to continue stepping with the next available engine. I'm not sure what the behavior of the latter will be when the current instruction is somewhere it doesn't have the source code for. It may well be that it'll cancel the step then.

But if the breakpoint is rather not hit at all (i.e. you don't see the corresponding callback ever executed), that would be surprising. Can you clarify whether this is the case?
Feb 3, 2015 at 9:40 PM
Hi pminaev,
But if the breakpoint is rather not hit at all (i.e. you don't see the corresponding callback ever executed), that would be surprising. Can you clarify whether this is the case?
The [StepInGate] handler for PyObject_Call is indeed executed; then it decides to set a breakpoint inside of OnPotentialRuntimeExit:
var bp = _process.CreateBreakpoint(Guids.PythonStepTargetSourceGuid, funcPtr);
Then shortly thereafter the StepCompleteNotification.Handle method is called, which calls the OnStepComplete method in TraceManagerLocalHelper to close the breakpoints. Are there any other handlers that are invoked? Is it possible that the target break points are being deleted too soon? Thank you
Coordinator
Feb 3, 2015 at 9:49 PM
The handler for that created breakpoint is TraceManager.OnStepTargetBreakpoint. If you can see that being hit and calling OnStepArbitration, then the Python debugger has correctly handled the step to the runtime boundary and passed it on to the next runtime in the chain (that being native). So if that's happening, then based on your observed behavior I think we can reasonably conclude that step-into becomes step-over if there are no symbols at the native boundary.

(One other way to verify it is to try pure native debugging with two modules, where the call chain is A->B->A, and A has symbols but B does not; if you try to step in from A, do you end in A on the other side, or does it just step over?)
Feb 3, 2015 at 10:56 PM
Hi pminaev,

Yes, I've verified that OnStepArbitration is being called... here are my debug statements:
PyObject_Call enter [StepInGate]
     evaluated tp_call=788784752 from func->ob_type->tp_call
     about to create breakpoint with GUID: 5653d51f-7824-41a0-9ce5-96d2e4afc18b, funcPtr=788784752
PyObject_Call exit [StepInGate]

OnNativeBreakpointHit
  OnStepTargetBreakpoint
    OnStepArbitration

StepCompleteNotification.Handle
When I do a step-in actually it doesn't even do a step-over, it just continue to the end of the program as if I had just it "continue" (I have no explicit breakpoints set). So you think that this is due to a lack of symbols for the step-in target? I should add that, although the step-in target has no symbols, it will in turn call my own C++ code for which there are symbols. Is there a way that I can set the native debugger in Visual Studio so that it won't ignore the breakpoint in the event of no symbols for the step-in target? Thank you
David
Feb 3, 2015 at 11:18 PM
Here is maybe another hint (?): in my python program I have:
import myextensionmodule
and this module is a pure extension module (DLL named myextensionmodule.dll) and I also have the .pdb in the same folder, and the console indicates that the symbols were loaded for this module. But yet, when I try to step-in to this import statement it just runs to the end of the program with no breaks.

I would expect that when I step in it should jump right into my C++ code in myextensionmodule.dll, for which debug info is available. Do you think this is related to my general step-in problem or is it expected? Thank you
David
Coordinator
Feb 4, 2015 at 12:44 AM
I'm not sure if there is a way to tell the native debugger to do that. When stepping native -> native, it's possible by switching to Disassembly view; in that mode, when the cursor is on the call instruction, stepping in will always step in to the target, regardless of whether there are symbols or not. But for cross-language stepping you can't really enable that view in advance ...

Having said that, I would still sort of expect it to do a step-over in that case, not just run to completion. I'll need to look into why it's being handled that way, perhaps there's more to it.

With symbols for everything, I definitely expect this to work, though, and I would like to investigate. We'll need to narrow it down, though. Do you observe the same problem with a simple hello world type app with stock Python?
Feb 4, 2015 at 8:20 PM
Hi pminaev,

Thanks again for the help; I'll do some more investigation and try installing debug symbols for the target native code. However I wanted to show you something that could be a hint as to what's going on (?). So I start debugging my python script and then go right up to the line with the native call that I want to step into (should ideally take me to boost::python C++ code). Then right before I step in, I clear the console output, then step in. Here is what I get in the console after hitting step-in:
exceptions.AttributeError
exceptions.KeyError
exceptions.KeyError
exceptions.StopIteration
First-chance exception at 0x765AC41F in python.exe: Microsoft C++ exception: boost_1_53_0::python::error_already_set at memory location 0x0027F068.
First-chance exception at 0x765AC41F in python.exe: Microsoft C++ exception: boost_1_53_0::python::error_already_set at memory location 0x0027F068.
exceptions.StopIteration
First-chance exception at 0x765AC41F in python.exe: Microsoft C++ exception: boost_1_53_0::python::error_already_set at memory location 0x0027F140.
First-chance exception at 0x765AC41F in python.exe: Microsoft C++ exception: boost_1_53_0::python::error_already_set at memory location 0x0027F140.
exceptions.StopIteration
First-chance exception at 0x765AC41F in python.exe: Microsoft C++ exception: boost_1_53_0::python::error_already_set at memory location 0x0027F218.
First-chance exception at 0x765AC41F in python.exe: Microsoft C++ exception: boost_1_53_0::python::error_already_set at memory location 0x0027F218.
exceptions.KeyError
The thread 0x287c has exited with code 0 (0x0).
Is there a way for me to know what these exceptions mean or how to debug them (if necessary)? I actually get a lot of these kinds of messages even before the step-in. Are they suppose to appear in the console even though I'm using a Release build of the extension? As you can see, the program just runs to completion, so the boost::python C++ code is getting executed, it's just that the debugger never stops there (I haven't yet obtained debug symbols for this code, will try that next).

Thank you
David
Feb 4, 2015 at 9:21 PM
Actually scratch that last question -- I looked into those exceptions and I don't believe they're relevant...
Feb 5, 2015 at 2:49 AM
Hi pminaev,

Update: the step in works when I have debug symbols for the target! (in this case the native code target is boost::python)

But now I need to figure out how to get visual studio to skip over the boost::python code and go straight to my C++ code so that I don't have to step through code that is not mine..... do you know if "Just my code" would work for C++ in that situation? I gave it a try (along with telling visual studio explicitely to not load symbols for boost::python), but the end result is either that (1) I have to step through the boost::python, or (2) it ignores the breakpoint completely...

Thanks again for you help
David
Coordinator
Feb 5, 2015 at 4:02 AM
Great to hear that it's working!

Regarding C++ Just My Code, I don't know the answer to this question, but I was always curious about it myself. Have you tried following the instructions here to add all the Boost Python source files to JMC list, but keep the symbols around?