about remote debugger performance

Apr 1 at 3:02 AM
I trying to get remote debugger from PTVS working with a large project, containing ~10MB Python 2.7 code. Python is embedded to app and linked statically, so I haven't access to DLL.
Most of system modules are not included. Python source is read from binary package and folders
are not supported. Custom heap used and interpreter state reset without memory leaks implemented.
I running ptvsd from attach_server.py, init.py, main.py are not used. I have no ctypes module and cannot add it easily, so I commented out locals_to_fast() in visualstudio_py_debugger.py.
Currently attach to debugger from visual studio 2012 works. If I press "pause" on debugger, it tells
me "couldn't pause running process". If I add breakpoint in root source file, it stops on breakpoint, but not in all cases, depends on line number.
But it takes too long time 30sec~1+min to stop on breakpoint. Is it by design?
I tried simple script without embedded python on clean install - debugging works much faster.
When I using similar script on embedded python, debugger hangs for a long time before stop on breakpoint. Also cannot stop on some breakpoints.
All breakpoints except root script are currently may be set without a problem, except it doesn't stop on them.
Also if I try breakpoint step run, it works as continue execute instead.
I tried to debug remote debugger source for breakpoints. Still cannot understand how it works,
but found that: only root file displayed in debugger modules page as root.py, but other files are module, and not module.py. So I guess I should fix it by adding .py to filename somewhere in remote debugger, if You can give me the hint where, that will help a lot.
Also some of modules are not displayed on modules page. Still don't know why.
If I tring to add some variable on visual studio debugger watch page, first time it fails, I press enter->5 sec hang->cannot evaluate expression. Then if I press refresh, it gives me right value after ~7 seconds. Mouse over variables doesn't work, it hangs for ~5 sec, no popup displayed.

I tested it on visual studio 2010, and tried latest 2.1 dev build(03/14), all issues above are still can be reproduced.
If You have some hints/workarounds how to fix these please share it here.
Apr 1 at 3:38 AM
Edited Apr 1 at 3:39 AM
also visual studio attached to remote debugger sometimes crashes, but I don't know how to collect debug information where it crashed.
In system log there are 2 events:
log name:         Application
source:           .NET Runtime
date:            2014/04/01 12:19:56
event ID:       1026
task category:      none
level:           error
keyword:         crash
user:          N/A
computer:       personal info deleted
description:
application:devenv.exe
framework version:v4.0.30319
description: prosess has been stopped , unhandled exception
exception information:System.Collections.Generic.KeyNotFoundException
stack:
   place System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].get_Item(Int32)
   place Microsoft.PythonTools.Debugger.PythonProcess.HandleExecutionResult(System.IO.Stream)
   place Microsoft.PythonTools.Debugger.PythonProcess.DebugEventThread()
   place System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   place System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   place System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   place System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   place System.Threading.ThreadHelper.ThreadStart()

event XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name=".NET Runtime" />
    <EventID Qualifiers="0">1026</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-04-01T03:19:56.000000000Z" />
    <EventRecordID>30369</EventRecordID>
    <Channel>Application</Channel>
    <Computer>personal info deleted</Computer>
    <Security />
  </System>
  <EventData>
    <Data>application:devenv.exe
framework version:v4.0.30319
description: prosess has been stopped , unhandled exception
exception information:System.Collections.Generic.KeyNotFoundException
stack:
   place System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].get_Item(Int32)
   place Microsoft.PythonTools.Debugger.PythonProcess.HandleExecutionResult(System.IO.Stream)
   place Microsoft.PythonTools.Debugger.PythonProcess.DebugEventThread()
   place System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   place System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   place System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   place System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   place System.Threading.ThreadHelper.ThreadStart()
</Data>
  </EventData>
</Event>
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
log name:         Application
source:           Application Error
date:            2014/04/01 12:19:57
event ID:       1000
task category:      (100)
level:           error
keyword:         crash
user:          N/A
computer:       personal info removed
description:
error application name: devenv.exe、version: 11.0.61030.0、timestamp: 0x5270a974
error module name: KERNELBASE.dll、version: 6.1.7601.18229、timestamp: 0x51fb1116
exception code: 0xe0434352
error offset: 0x0000c41f
error process ID: 0xe88
error process start time: 0x01cf4c9627fb2da9
error process path: C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\devenv.exe
error module path: C:\Windows\syswow64\KERNELBASE.dll
report ID: 7fce76b1-b94c-11e3-840c-df9941fb26b3
event XML:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2014-04-01T03:19:57.000000000Z" />
    <EventRecordID>30370</EventRecordID>
    <Channel>Application</Channel>
    <Computer>personal info removed</Computer>
    <Security />
  </System>
  <EventData>
    <Data>devenv.exe</Data>
    <Data>11.0.61030.0</Data>
    <Data>5270a974</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.1.7601.18229</Data>
    <Data>51fb1116</Data>
    <Data>e0434352</Data>
    <Data>0000c41f</Data>
    <Data>e88</Data>
    <Data>01cf4c9627fb2da9</Data>
    <Data>C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\devenv.exe</Data>
    <Data>C:\Windows\syswow64\KERNELBASE.dll</Data>
    <Data>7fce76b1-b94c-11e3-840c-df9941fb26b3</Data>
  </EventData>
</Event>
Coordinator
Apr 1 at 7:34 AM
This sounds like a rather heavily customized interpreter. Having module names without .py is definitely very unusual (we get them from sys.modules on attach, and later from f_code.co_filename of frames as they appear for code loaded after attach). I also wouldn't be surprised if the host tries to use some of the Python debugging APIs itself for some reason, and it & PTVS debugger end up stepping on each other's toes - we try to play nice by calling the previous hooks and generally being as non-invasive as possible, but if the other party is not cooperative it might not be enough.

Regarding perf, the only thing I can think of is that perhaps the host is itself running a lot of Python code, and the debugger ends up observing all that? Normally we don't debug e.g. standard library code, so no trace hooks set for those frames - but in your case, since all the code is embedded into the host, we can't tell that it's the standard library.
Coordinator
Apr 1 at 7:36 AM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Apr 1 at 8:44 AM
Edited Apr 1 at 9:47 AM
Thanks for having look this one.

Our Python is based on vanilla CPython 2.7, just deleted some modules, modified module loader, custom memory heap, no debugging APIs exist I think.

I'll try to fix module names to be more compatible.
update: looking in the source(=import.c), module path was ..."". fixed, but still can't stop on breakpoints except root and step debugging doesn't work. even if I put python source to path displayed in modules window..
Apr 2 at 8:31 AM
Edited Apr 2 at 8:32 AM
Still struggling it all to make it work with my embedded python.

I tried Winpdb, and it can even stop on all breakpoints, allows step debugging, and works very fast.
No code fixes were required, just added some modules, and it worked.
But I can't see globals/locals for some reason, although they output correctly if I run exec print <...> in debug console.
Also some source code cannot be displayed in Winpdb window, and some of that source has no .py extension.
Probably that means my import module Python code fix is still incomplete. So Winpdb is almost works for me :).

As for issues in the top of topic, they may be linked to import path too.
Apr 4 at 6:03 AM
Edited Apr 4 at 6:09 AM
Well, something strange occurs in my python core code, so I don't know how to fix import paths.
But in Winpdb we have such a cool feature, I just found it, like source provider.
For example, Your remote debugger server absolutely have no idea where Your script source located.
May be its in database, may be its even generated, or encrypted, or... <insert anything You can imagine here>.
And, in Winpdb, You make a function like this:
def my_cool_new_source_provider(filename):
    found=find_my_file(filename)
    if found:
        sourcecode=read_file(get_full_path(filename))
        return sourcecode
    else:
        raise IOError
And then, when start debugger:
your_debugger_name_here.startme(source_provider=my_cool_new_source_provider)
Have we any chance that similar feature will be added to ptvsd?
Apr 4 at 9:36 AM
Finally solved all problems in Winpdb by hacking its source :).
Unfortunately can't get ptvsd work in the same environment.
Actually winpdb requires more dependencies, but much more stable and faster.