Debug embedded Python on single system?

Oct 24, 2013 at 2:41 PM
Hi,

I currently use winpdb to debug an embedded Python project. The key step is the inclusion in the source code of
import rpdb2; rpdb2.start_embedded_debugger(password)
and then I'm able to debug the process using the winpdb window. I'd like to do the same using PTVS and not use winpdb any more.

Using Visual Studio 2010 with PTVS, I set breakpoints and then "Attach" to the process that is embedding the Python code. However, this causes the process to crash.

Little help? You would think that PTVS could do the same as winpdb, but maybe there are other steps that I'm missing here.


Thanks!
Coordinator
Oct 24, 2013 at 3:44 PM
Which Python version are you attaching to? Does it happen to be a custom-built interpreter with some modifications, by chance?

PTVS Attach to Process works very differently, in that it does not require you to add any modifications to your code, but rather goes to the process and tries to hook into various Python internal machinery to do the attach. Python itself was never written to support this model, so PTVS has to jump through a lot of hoops to make it happen - and all this set up is rather brittle and heavily dependent on dealing with a specific Python version with known layout of internal data structures etc. That's why it's only supported on CPython, and then only specifically on the official versions that were out when we shipped (for 2.0 RTM, we support 2.5 to 3.3).

However, if you want something that is more like your winpdb experience, have a look at our remote debugging support with ptvsd. This has a similar model where you have to explicitly start the debugging server in your code by adding a couple of lines, and then you can attach to it from VS. The attach process there is more stable, with no special version-specific hacks, and so does not depend on Python version or implementation - if we can debug the interpreter when launching it from VS, we can also attach to it with ptvsd.
Oct 24, 2013 at 8:16 PM
Thanks for your prompt reply. I followed the instructions exactly for the ptvsd option and I did manage to get it to connect to the process and then later stop at an appropriate breakpoint. However, another problem is that the Debug interactive window
does not respond. All of the calltips in the main editor window work fine. All of the controls in the main window (e.g., step-into, step-over) work fine. Only the debug interactive window does not seem to work. Unfortunately, I really need this window. Any
help appreciated. By the way, I am dealing with Python 2.7 CPython downloaded from the main python website. It is not a custom build. The application itself embeds this version, not another custom build up its own devise. Thanks again. On 10/24/2013 8:44 AM,
pminaev wrote:
Oct 29, 2013 at 6:47 PM
pminaev, I updated the thread. Do you have any more ideas on this? Thanks! On 10/24/2013 8:44 AM, pminaev wrote:
Coordinator
Oct 29, 2013 at 7:10 PM
Can you clarify the symptoms? Are you unable to type anything in the Interactive window at all? Or when you do and press Enter, it just sits there and never completes the evaluation?
Oct 29, 2013 at 9:32 PM
I type into the interactive window (e.g. print self), hit enter and nothing happens. The cursor icon just spins endlessly. Stopping the debugger is the only way to make this stop. On Tuesday, October 29, 2013 12:10:48 PM, pminaev wrote:
Oct 31, 2013 at 9:31 PM
Any ideas here? Thanks! On Tuesday, October 29, 2013 2:32:51 PM, reckoner wrote:
Coordinator
Oct 31, 2013 at 9:54 PM
Does something as simple as evaluating 42 work? Is the python process consuming any CPU after evaluating print self and does stepping in the debugger still work after doing it? I'm mainly curious if the process might be deadlocked or if it might be actively running code - one of which is likely what's happening giving that the prompt isn't coming back.
Nov 1, 2013 at 2:08 AM
No, evaluating anything does not work. The interactive debugger window is locked up, but the GUI continues to work – I can step thru the code using the buttons so debugging by using the mouse to step and to Inspect the various variables works great! The
only thing that doesn't seem to work is the interactive debugger window. Thus, the process is not deadlocked because I can continue to debug using the GUI tools. It is not somehow running more code in the background that is not hooked into the visual debugger,
otherwise, I would not be able to stop it and step thru it. Any other ideas? On Thursday, October 31, 2013 2:54:54 PM, dinov wrote:
Nov 4, 2013 at 2:19 PM
Any more ideas on this? On 10/31/2013 7:07 PM, reckoner wrote:
Coordinator
Nov 7, 2013 at 6:18 PM
Sorry for the late reply.

Is the REPL hang specific to this particular project that you're trying to debug, or even some particular stop location in that project? Or do you see the same problem even when attaching to a simple hello world type of app?

The only thing I can think of right away is that the REPL prints out values of expressions that are typed in, and they are printed both to REPL and to process standard output. If the latter is somehow intercepted, e.g. if it's a pipe that is not read from because the process that reads from it is also paused. Of course, a bug is always another possibility; we'll need to think about how to best diagnose if that is the case.
Nov 7, 2013 at 11:04 PM
Outside of this particular application, I don't have any trouble at all with the debugger or anything else. It works perfectly. It's just with this particular application that I'm having a problem. One issue is that because Python is embedded in this application
(i.e. it is not attached to any console), I'm not sure where standard output is going. is there a way to somehow stop the debugger from attempting to send to the standard output, and instead just send to the debugger window? thanks! On 11/7/2013 11:18 AM,
pminaev wrote:
Coordinator
Nov 7, 2013 at 11:18 PM
There's no option for this, but luckily this is something that's entirely implemented in Python, so you can easily change it. Locate the following file, adjusting the path as necessary for your system:

C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.0\visualstudio_py_repl.py

In that file, look at the definition of class _ReplOutput at line 1106. In its __init__ method, change the following line:
self.old_out = old_out
to:
self.old_out = None
This will prevent anything from reaching the original stdout, with all output going only to the debugger windows.
Nov 8, 2013 at 8:58 PM
It works! Thanks! I am now able to use the REPL in the debugger for this application. For the posterity of this thread, I actually made the change you suggested on all 3 versions of visualstudio_py_repl.py that I had on my system (as listed below) because
making the change to the file on the path you specified did not work. C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.0 \visualstudio_py_repl.py C:\Program Files (x86)\Microsoft Visual Studio
11.0\Common7\IDE\Extensions\Microsoft\Python Tools for Visual Studio\2.0\ptvsd\visualstudio_py_repl.py C:\Python27\Lib\site-packages\ptvsd\visualstudio_py_repl.py Thanks again! On 11/7/2013 4:18 PM, pminaev wrote:
Coordinator
Nov 8, 2013 at 9:03 PM
Sorry, I totally forgot that you're using ptvsd to attach - and that has its own copy of that file (we really should combine the two, there's no particular reason for two copies).

Thank you for confirming that it works! Now that we know what the issue is, this is definitely a strong argument in favor of adding stdout suppression as a debugger option. It might also be possible to add some kind of timeout to REPL output so it doesn't get permanently blocked even when that option is not enabled.
Coordinator
Nov 8, 2013 at 9:06 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.