Using PTVS with DeadlineCommand (Thinkbox company)

Oct 21, 2013 at 11:37 AM
Hi,

I developp scripts and plugins for Deadline Render Farm written in Python, and I'd like to use PTVS.
I tryed 'Attach To Process' selecting Python source only. It attaches, but breakpoints are not working.
I did a special environment for Deadline (which calls DeadlineCommand).

It uses :
Deadline Version: 5.2.0.47700
FranticX Version: 2.0.0.47694

Direct debug session does not work either (using IronPython launcher).
Do you have an idea?

Nico
Thanks
Coordinator
Oct 21, 2013 at 5:51 PM
Did you use IronPython for your plugins? Attach to Process using Python mode only works for regular CPython; for IronPython, you need to attach using Managed mode.
Coordinator
Oct 21, 2013 at 5:52 PM
Direct debug session does not work either (using IronPython launcher).
Can you clarify? Are you starting your script directly, and it doesn't run, or it runs but breakpoints aren't hit?
Oct 22, 2013 at 9:54 AM
First thanks a lot for your fast answer.

So I work with IronPython scripts run under Deadline.

First I tried your recommandation:
  • Execute an IronPython runned with 'deadlinecommand.exe -ExecuteScript "Path_to_IronPython_script"'
    Standalone execution.
  • Attach to process using 'Managed mode'
  • Breakpoints (set on the python code file) appear disabled and won't be hitten.
    Displayed comment is '... No symbols have been loaded for this document'
Then, I tried:
  • Execute the same way...
  • Attach to process using 'Python mode' (which according to you, does not work)
  • Breakpoints appear enabled but won't be hitten.
Finally, I tried (Direct debug session):
  • Add a new Python Environment for Deadline (Path, Windows Path, Architecture...)
    in Visual Studio.
  • Create an empty Python project
  • Set path to my iron python script as Startup File
  • F5, and fails
If you have any idea?

Thanks
Coordinator
Oct 22, 2013 at 11:18 AM
In order to debug IronPython in managed mode, you need to enable debug mode when running it. When running ipy.exe from command line, this is the -X:Debug option. I'm not sure what the equivalent setting would be for Deadline - if they run IronPython in a separate process, then I would expect them to have a configuration dialog somewhere that lets you specify path to its executable as well as any arguments passed to it, and you'll need to put -X:Debug there.

With respect to setting up a direct debug session - setting it up like that would only work if Deadline is actually a Python interpreter that can be invoked as such. For example, if the launch mode for your empty project is set to "IronPython (.NET) launcher" in project properties -> Debug tab, then the executable should accept IronPython options such as -X:Debug, and execute the script that is directly passed as a parameter. You can try injecting additional arguments like -ExecuteScript for the executable via the "Interpreter Arguments" setting in project properties, but if it specifically needs the filename following that argument on the command line, I don't think it will help you. So using Attach is the best bet, provided that you can force Deadline to enable debugging mode for IronPython.
Oct 23, 2013 at 10:14 AM
I tried remote debug

Add this code in my Deadline python script.
import ptvsd
ptvsd.enable_attach(None)
ptvsd.wait_for_attach()

Then I do attach to process using python remote debugging (unsecured), with my user name @ local machine.

Breakpoints are not hit.
If I break debug, I am stop in the middle of nowhere (Source Not Available, and no callstack), BUT in watch window I can see loaded modules and
global variables and content.

Does anyone have an idea of what I should do, to get those breakpoints and callstack ?

Thanks
Marked as answer by nmeyer on 10/23/2013 at 7:42 AM
Oct 23, 2013 at 3:46 PM
In fact I am able to put breakpoints in all files EXCEPT the first loaded script.
If you need to debug python script run inside Deadline context with PTVS (at least VS 2012), remote debugging is the real solution.

Thanks
Coordinator
Oct 23, 2013 at 5:50 PM
When you break debugging where it says "Source Not Available", can you inspect the value of __file__ and post it here?
Oct 31, 2013 at 12:14 PM
If I select the top of the callstack, python entry script, __file__ is set with none.
Coordinator
Oct 31, 2013 at 6:01 PM
Edited Oct 31, 2013 at 6:02 PM
Ah, that would be the problem, then. It indicates that Deadline does not load & run the initial script properly - most likely they just open the file themselves, read it into a memory buffer, and pass it over to Python without specifying the filename, e.g. by calling PyRun_String. As a result, when debugger attaches, it has no idea where the running code actually came from, and so can't map it to the source file & breakpoints inside.

I would suggest filing a bug (that __file__ is not properly set in the initial script, and give them this thread as context) against Deadline. It should be a fairly trivial thing to fix - they should call the API that takes the filename parameter, and then make sure that they pass the proper name to it.

In the meantime, the workaround is adding one layer of indirection - have your initial script do nothing but import the actual script file with your code. Importing uses the Python import machinery, which takes care of storing the filename properly, which is why your other modules work.

By the way, you can even get away with using a single file for all of this, by doing something like this (assuming the file is foo.py):
import sys
if __file__ is None:
   import foo
   sys.exit()

# Your script follows
...
i.e. the file just imports itself if it discovers that __file__ is not properly set. This has the nice benefit that it'll keep working without any changes if/when Deadline will fix the issue.
Nov 4, 2013 at 2:26 PM
Edited Nov 4, 2013 at 2:27 PM
It does not work.
I tried import of the module, and I even tried to set __file__ variable directly.
Nothing works.
Coordinator
Nov 4, 2013 at 4:20 PM
Setting __file__ directly will not help, because that's not where the debugger looks for it (it uses f_code.co_filename on the frame object). It's just a convenient place to observe the load value to determine whether it was wrong.

You mentioned earlier that you could set breakpoints in scripts other than the initial one. Does that still work? Are you importing those other scripts, or are you using some other way to launch them from your main script?
Nov 4, 2013 at 4:35 PM
Nevertheless import did not work, even if my module was there.
The python script is listed inside sys.modules, but if expand it in the debug watch it can read:
<module 'my_script' from 'none'>

False alerte, excepted entry python script, I am able to set breakpoint (break, step, ...) anywhere.
Coordinator
Nov 4, 2013 at 4:44 PM
Hmm... I suspect that they are doing something even more funky than I originally thought, and actually importing the startup script (instead of exec'ing it), but without the appropriate file information - that's the "from 'none'" part in the above, it should be "from 'my_script.py'".

Can you check if you have an entry in sys.modules for the startup script even if you don't use the recursive self-import trick that I've used above? If you do see the entry with incorrect source file path there still, then try this modification to the script:
import sys
if __file__ is None:
   del sys.modules['foo']
   import foo
   sys.exit()

# Your script follows
...
Nov 4, 2013 at 5:32 PM
Well, when I do that it exit when import ... is called.

But I can do step by step in a kind of blind mode, debugger telling the line debugged, so that I can read the line treated.
It works even if it is not very convenient.
Coordinator
Nov 7, 2013 at 7:04 PM
This is also strange, since it would indicate that the script ends up with __file__ being None even when it's properly imported.

There is probably some way to work around that, as well, but at this point I think it's easier to just avoid doing anything in the main script. Just put all logic in a secondary script, and have the main script import that (and do nothing else).