Attach to embedded python application: timed out

May 22, 2012 at 3:29 PM

Hi,

I'm using Python Tools 1.1.1 and every time I try to attach the debugger to my embedded Python application, I get a "timed out" error instantaneously (which is weird for a time out!)  My Python version is 2.7.3 and when I get the time out, I have this callstack:

accept in socket line 201
tick in __init__ line 1646
start in __init__ line 1551
_start_http_thread in servers line 65
run in threading line 501
__bootstrap_inner in threading line 533
__bootstrap in threading line 510

accept is in the standard python socket module and the other methods are in CherryPy or the threading module.  Line 201 of socket is the line of the definition of the accept function.

At first, I thought it was caused by embedded Python not being supported by Python Tools, but I created a simple C++ application running code from a test Python module and I can attach to it and debug it without any problem.  I have no idea why it doesn't work with our main application.

Please help me!

Francis

May 23, 2012 at 4:01 PM

I'm now able to attach the debugger to my application and break.  I disabled all the CherryPy stuff.  But now, all I can see are threads waiting forever for an event (which is kind of normal) but I can't see my main thread and my breakpoints in it are not doing anything.  Another weird thing is that when I'm breaked in the debugger, my application is still running!  It's a server and it continues to process queries.  Visual Studio also becomes unresponsive when debugging.  It also works only one time, if I detach, it waits for a long time and shows errors and then, if I try to attach again, VS will crash.

I really don't understand what's going on.

May 23, 2012 at 8:17 PM

I tried to debug it with Aptana and it works flawlessly.  In Aptana, I've seen that the application is running a big number of threads, much more than the 3 or 4 there was in Visual Studio (It's a really big application, I'm far from knowing everything about it).

So, I think the problem is related to the great number of threads.  I tried PTVS 1.5 Alpha and it seems to be a little bit better than 1.1.1, but it's still not working.  I can now see a lot more threads than before, but it's still very unstable and my breakpoints are not working.

Any help would be really appreciated.

Coordinator
May 23, 2012 at 8:40 PM

Is the time out error being displayed inside of VS? 

This sounds a lot like http://pytools.codeplex.com/workitem/615 - but that was supposed to be fixed for 1.1. 

Is Python code actively still running when you attach?  We should be able to ultimately stop all of the threads once they run some Python code, but any native pieces of code we cannot aggressively block. 

One thing to try might be adding a thread which is looping and doing nothing (e.g. a while True: pass).  If that allows you to successfully attach then it means something is going wrong with the attach when the threads are blocked.

Does this repro with just a simple CherryPy hello world app?  If so we should get a bug on it and just repro and fix it on our side (unless you really want to debug into it, in which case I'm willing to help :) )

May 24, 2012 at 1:02 PM

Yes, the time out error is displayed inside VS.  With alpha 1.5, I can click continue 2-3 times and then the execution continues and the debugger remains attached.  It was maybe the same with 1.1.1, but I don't think I tried to click continue when it happened.

There is some Python code running when I attach, but I don't think the thread I'm interested in is running though.  I'll try to resume what I have now:

  • I can attach the debugger to my application, but if CherryPy is running, I have to click continue a couple of times if the "timed out" message appears.
  • After that, I can break (using the "pause" debug button) and I can see my threads.  If I set a breakpoint in the code I'm interested to debug and continue execution, the breakpoint won't hit.
  • If I click on "pause" again, the pause and play button are disabled and only the stop button is enabled.
  • If I click "stop", after about a minute, I get the following message:
    "Debugging is being stopped, but is not yet complete.  You can force debugging to stop immediately, but any process being detached may be terminated instead."
    The only button is "Stop now" and it detaches the debugger.  This instance of VS is now doomed and if I try to attach again, it crashes with the following exception in Microsoft.PythonTools.Debugger.dll: System.InvalidOperationException with the following call stack

    Microsoft.PythonTools.Debugger.DebugEngine.AD7Engine.Send
    Microsoft.PythonTools.Debugger.DebugEngine.AD7Engine.SendThreadStart
    Microsoft.PythonTools.Debugger.DebugEngine.AD7Engine.OnThreadCreated
    Microsoft.PythonTools.Debugger.PythonEngine.OnThreadCreated
    Microsoft.PythonTools.Debugger.PythonEngine.DebugEventThread
    ...

I didn't try to reproduce the bug with a simple CherryPy app because that is just one of the many services provided by our application and not the one I'm interested in right now.

I don't know if that can help you, but we're inside a Python virtual environment.  Also, the core of the app, and what I want to debug, is a remote method call server.  We have about 25 C++ threads waiting for RMC calls and executing the corresponding Python code.

I'll continue to investigate and try to understand the core of our system.  Meanwhile, if you have any idea, please let me know!

Thanks!

May 24, 2012 at 3:38 PM
Edited May 24, 2012 at 3:39 PM

Good news!  I created a small application reproducing my issue!  Here is the C++ code:

#include <Windows.h>
#include <process.h>
#undef _DEBUG
#include <Python.h>

PyObject *g_pFunc;

void Thread(void*)
{
    while (true)
    {
        PyGILState_STATE state = PyGILState_Ensure();
        PyObject *pValue;

        pValue = PyObject_CallObject(g_pFunc, 0);
        if (pValue != NULL) {
            printf("Result of call: %ld\n", PyInt_AsLong(pValue));
            Py_DECREF(pValue);
        }
        else {
            PyErr_Print();
            fprintf(stderr,"Call failed\n");
            return;
        }
        PyGILState_Release(state);

        Sleep(1000);
    }
}

void main()
{
    PyObject *pName, *pModule;

    Py_Initialize();
    PyEval_InitThreads();
    PyEval_ReleaseLock();
    pName = PyString_FromString("test2");

    pModule = PyImport_Import(pName);
    Py_DECREF(pName);

    if (pModule != NULL) {
        g_pFunc = PyObject_GetAttrString(pModule, "test");

        if (g_pFunc && PyCallable_Check(g_pFunc))
        {
            DWORD threadID;
            threadID = _beginthread(&Thread, 1024*1024, 0);
            threadID = _beginthread(&Thread, 1024*1024, 0);

            while (true);
        }
        else
        {
            if (PyErr_Occurred())
                PyErr_Print();
        }
        Py_XDECREF(g_pFunc);
        Py_DECREF(pModule);
    }
    else
    {
        PyErr_Print();
        return;
    }
    Py_Finalize();
    return;
}

And the corresponding test2.py module:

def test():
    for i in xrange(10):
        print i

    return 0

If I execute this program and attach the debugger, I'm getting the same result as I get with our main application (I wrote this trying to simulate what is going on in our server).  When I try to break, it will tell my "unable to break" and my breakpoints in Python are not working.  If I try to stop the debugger, I get the same thing as I described in my previous message.

Tell me if you can reproduce the issue and if you have a workaround or a bug fix.  Thanks a lot!

May 29, 2012 at 11:40 AM

Anything new?  Did you manage to reproduce my issue?  Can you tell me if it's a bug or a limitation of Python Tools and if it's fixable or if there's a workaround?  Thank you!

Coordinator
Jun 5, 2012 at 6:14 PM

Sorry we've been heads down on trying to get our next beta out, we're going to spend some time focusing on bugs right after it's released (which is real soon now).  So hopefully we'll be able to figure it out soon.

Jul 4, 2012 at 5:21 PM

I'm still waiting for an answer about this issue...  anyone had time to look at it?

Thanks!

Coordinator
Jul 5, 2012 at 9:49 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Jul 5, 2012 at 9:50 PM

I get the exact same results as you - I've opened a bug and I'm looking into it.  Again, sorry for the delay and thanks for your persistence!

Jul 6, 2012 at 11:27 AM

Great, thank you!