mixed mode debugging for embedded python

Jun 23 at 11:12 PM
Does PTVS support mixed mode debugging for embedded python? e.g. I have a C#, C++/CLI projects in a solution, where C++/CLI calls Python function from an imported module according to https://github.com/dabeaz/python-cookbook/blob/master/src/15/calling_python_from_c/embed.c

How do I get to Python stack during debugging?
Coordinator
Jun 24 at 12:04 AM
Yes, this is supported. I don't think there's any way to enable Python debugging when initiating it from a C++ or a C# project, so you'll need to attach to the running process instead (Debug -> Attach to Process), and check the corresponding languages in the attach dialog instead of using the default choice.
Jun 24 at 3:03 PM
When I attached, should I select automatic option? If yes, then it complains "Failed to attached debugger: Python interpreter not found"
Jun 24 at 3:10 PM
Ok, I see that I need to initialize python instance with Py_Initialize() before attaching. So now I need to figure out how to restructure the code for this.
Coordinator
Jun 24 at 6:48 PM
No, automatic is the wrong thing. Automatic will attach with Python debugging only. You want to switch to the mode where you choose yourself, and manually select all the code types that you want to debug - in your case it would be Python, Native and Managed.

Attaching that way should work even if you haven't called Py_Initialize yet (and even if you haven't loaded python??.dll yet).
Jun 24 at 10:48 PM
Thanks, I set Python, Native and Managed (works with v2.0?) and now when I get to something like PyObject_Call(), then Python27.pdb is not loaded:

...\python-2.7.7-pdb\python27.pdb: PDB does not match image.

I have Python 2.7.7 from Anaconda, but looks like your symbols for official Python do not work:

Python 2.7.7 |Continuum Analytics, Inc.| (default, Jun 11 2014, 10:41:43) [MSC v.1500 32 bit (Intel)] on win32

So do I need to obtain Python27.pdb for Anaconda?
Coordinator
Jun 25 at 1:28 AM
We don't actually ship symbols ourselves, we just link to the ones on python.org that they publish.

And yes, it needs symbols that match the interpreter that you're actually running with. If it's a custom distro, and they have rebuilt their Python DLL themselves, then it will require symbols that was produced by that build (even if the source was the same - this is the unfortunate consequence of the way PDBs work in general). If you do your own custom build (e.g. because you want to develop against a debug version of Python), then you'll also need the symbols from that build.

So, yes, to get this to work with Anaconda, you will need to ask them to provide symbols for python##.dll and _ctypes.pyd (though it might be easier for them to just provide the full set, just like the official builds do). One convincing argument in favor of that would be that Enthought Canopy started doing so recently, precisely so that it'd work with PTVS mixed-mode debugging :)
Jul 1 at 8:21 PM
I installed Python 2.7.7 32-bit on Windows from official website and it is missing python27.dll. How can I get this DLL?
Coordinator
Jul 1 at 8:23 PM
If you mean that it's missing from C:\Python27\DLLs (or wherever you installed it to), it's because the installer puts it into C:\Windows\SysWOW64.
Jul 1 at 8:55 PM
Ok, then even official Python complains: "python27.pdb: PDB does not match image." for C:\Windows\SysWOW64\python27.dll version 2.07.7150.1013

pdb for 2.7.7 obtained from Python website:



Coordinator
Jul 1 at 10:07 PM
You're right, the symbols on python.org are incorrect. I'm getting them fixed, but in the meanwhile you can get the correct symbols from https://portalvhds58z5h5xq9rbcq.blob.core.windows.net/python/python-2.7.7-pdb.zip
Jul 8 at 12:01 AM
Looks like the new symbols you posted work. But now the VS2013 with PTVS are looking for source code *.c files for Python, instead of jumping into Python code directly.

What am I doing wrong to go to embedded Python module code from native/managed side?


Coordinator
Jul 8 at 3:34 AM
Are you seeing .py (or .pyc) files in the Modules debugger toolwindow (it can be opened via Debug -> Windows -> Modules when you're debugging)?
Jul 8 at 4:11 PM
No, I do not see py/pyc files in "Modules" during code execution. python27.dll and other pyd symbols are loaded. The call stack between C/C++/C# has:

[Frames below may be incorrect and/or missing, no symbols loaded for mscorwks.dll]
[External Code]



Jul 8 at 5:45 PM
In the attach window I selected Managed (v2.0, ...), Native, and Python. Should Managed Compatibility Mode be selected as well?
Coordinator
Jul 8 at 6:16 PM
I don't think this will work with "Managed (v3.5, v3.0, v2.0)" actually - this mode basically implies Managed Compatibility. What this means is that rather using the new VS debug engine that permits mixing any language with any other language (so long as its debugger support targets that new engine), it'll use the old managed and native debuggers that only support mixing between each other. Support for older .NET versions was never updated to use the new debugger. So you need to be using "Managed (4.0)" and not be using compatibility mode for it to work...

We should have better diagnostics for this, though. I'm surprised that it is even possible to check Python and Managed (2.0) together, but you're right, it works (right up until the point where you start debugging and it actually doesn't). I'm not sure if it's possible to disable this without affecting Managed (4.0), but even in the worst case it should be possible to add a message box explaining the issue.
Coordinator
Jul 8 at 6:19 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Jul 8 at 7:56 PM
Yep, I tried mentioning before that we are using .NET v2.0 for this particular project. If target framework is changed .NET 4.0, then these new DLLs are not compatible with a lot of other components in .NET 2.0. Can Managed debug engine v4.0 attach to .NET 2.0 DLL?


Coordinator
Jul 8 at 8:27 PM
My apologies - I see the earlier mention now, but it looks like I glazed over it. :(

Technically speaking, what matters is the version of CLR that is running, not which version a particular assembly is compiled for. It is possible to have an assembly that was compiled against 2.0 to load in CLR 4.x - it's just a matter of adjusting the application config accordingly, and could be done specifically for debugging.

There's no way to make this work with CLR 2.0, unfortunately - the new Managed 4.x engine does not support it, and the old one does not support arbitrary mixed-mode.
Jul 8 at 9:54 PM
Edited Jul 8 at 9:54 PM
appconfig does load .net 4.0 runtime, but debugger cannot consume the symbols.

The message with the warning comes up if I first attach using 4.0 managed debugger and then switch back to 2.0 debugger.

Image
Coordinator
Jul 8 at 10:17 PM
What kind of error message do you get regarding symbols when running on .NET 4.0?

I just tried making a simple console app like so:
using System;
class Program {
    static void Main() {
        Console.WriteLine(Environment.Version);
        long i = 0;
        while (i >= 0) {
            i += 1;
        }
        Console.WriteLine(i);
    }
}
compiling it with csc.exe from 3.5, and running it under 4.5 with the following config:
<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
    <supportedRuntime version="v4.0" />
  </startup>
</configuration>
This does print out 4.0.30319.34014, so it's definitely running under .NET 4. When attaching using the "Managed (v4.0, v4.5)" engine (VS 2013 Update 2, in case that matters), it does load the symbols for the DLL, and breaking inside it shows the source code as expected.
Jul 9 at 3:05 PM
Edited Jul 9 at 3:34 PM
Pavel,

Finally we got it working.

1) I did not have before useLegacyV2RuntimeActivationPolicy, so the appconfig of DLL wrapper (executable) for this DLL did not cause loading of .NET 4.0, useLegacyV2RuntimeActivationPolicy is very handy - this forced to load .NET 4.0

2) Also I disabled the flag in Debug -> Options and Settings -> General -> Use Managed Compatibility Mode After this the C++ related warning is generated in VS2013, although this looks not relevant: https://www.dropbox.com/s/65oavtb4nv0zegk/c%2B%2B%20incompatible.PNG

3) Now I can step into Python code!
Coordinator
Jul 9 at 5:33 PM
Awesome! It's too bad that we don't have a better story around this. Well, at least now it's sort of documented on the forum :)

The C++ warning is, I believe, about C++/CLI. On the blog post about the new managed engine, they mentioned that its debugger support was never updated to the new engine. I would expect that it'd still give you basic managed debugging - i.e. show (raw) callstack and variables - but that expression evaluation wouldn't work.
Jul 16 at 7:16 PM
Edited Jul 16 at 7:16 PM
BTW, I was able to attach just to Python part of code (only selected Python code type). But after un/re-installing PTVS, either VS2013 crashes or Python code cannot be stepped into. Is this a bug or I need some special settings or things changed in the latest version? Also for my colleague this approach Python only did not work at all, although mixed-mode works for both of us.
Coordinator
Jul 16 at 7:50 PM
Attaching to Python alone is a different code path - we use our primary Python debugger there (which doesn't mingle with C++, but knows a lot more about Python), and attach is also a fair bit more complicated in order to insert the necessary hooks into a running interpreter. But if mixed-mode works, then regular attach should, generally speaking, also work (it's possible to have one work but not the other, in theory, but that involves exotic scenarios like tweaking the core parts of the interpreter source code).

If VS crashes while attaching, that's definitely a major bug that needs fixing. It should record an error event in the Windows Event Log (under Applications) - can you take a look there and share the offending call stack? And, perhaps, a fix to that would also fix the problem that your colleague is having.
Jul 16 at 8:52 PM
Edited Jul 17 at 2:10 PM
I sent all files to [email removed], even mixed mode did not work until I switched between Managed v2.0 to Managed v4.0. This is not trivial how mixed mode is working.
Jul 17 at 2:07 PM
Jul 25 at 3:30 PM