Intellisense with CERN's ROOT framework

Nov 15, 2012 at 7:14 AM

Hi,

First let me state that I'm very new to Python programming, so please forgive me if I said anything that make no sense.

I'm working with CERN's ROOT framework (http://root.cern.ch). It is a C++ framework, but has a Python wrapper. Some info about that here: http://wlav.web.cern.ch/wlav/pyroot/

Basically, I cannot get the intellisense to work with it. If I use pytool's interactive windows, (almost) everything shows up in auto-completion after I put in "from ROOT import *" but I cannot get it working at all in the editor. I tried "generate intellisense DB" but that doesn't help.

My question is, is it possible to make this works?

Coordinator
Nov 15, 2012 at 9:36 PM
Edited Nov 15, 2012 at 9:36 PM

I've just had a quick look at their Python wrapper and it is possible that they've completely defeated our current IntelliSense scraper with this:

sys.modules[ __name__ ] = ModuleFacade( sys.modules[ __name__ ] )

Basically, rather than having Python classes they create a dynamic module. I'll have a look at how we might be failing, but while I get this built it would be helpful if you could send the contents of the diagnostic window (under Tools->Python Tools->Diagnostic Information in VS) to ptvshelp@microsoft.com (feel free to find/replace personally identifying path names). This will include any errors that our scanner encountered while looking at ROOT.

(Incidentally, the reason it works in the interactive windows is because those use dynamic information (like dir() and getattr()) while the editor's completions do not.)

Nov 15, 2012 at 10:33 PM

Thank you for your reply.

I sent the info to the email you mentioned. Noted that in the project I added libPyRoot.dll and libPyRoot.pyd to the references and C:\root\bin and C:\root\lib to the search path. Hopefully that was the right thing to do.

Coordinator
Nov 15, 2012 at 11:25 PM

Thanks. The email hasn't come through yet, but I'm pretty confident that the reason it doesn't work is because the ROOT designers don't want it to work:

http://wlav.web.cern.ch/wlav/pyroot/using.html

Since it is foreseen that most people will use the simple approach anyway, the request to copy all from module ROOT will not actually result in copying all ROOT classes into the current namespace. Instead, classes will still be bound (and possibly loaded) on an as-needed basis. Note carefully how this is different from other python (extension) modules, and what to expect if you use the normal inspection tools (such as e.g. 'dir()'). This feature prevents the inspection tools from being swamped by an enormous amount of classes, but they can no longer be used to explore unknown parts of the system (e.g. to find out which classes are available). Furthermore, because of this approach, <tab>-completion will usually not be available until after the first use (and hence creation) of a class.

If we've managed to scrape libPyRoot.pyd then you may be able to get information by writing "import libPyRoot as ROOT", but since this is clearly not the intended use it is unlikely to be a complete solution. The best solution is probably for ROOT to change their bindings to be introspection friendly - I'd guess that performance was the original reason, but that should not be as big an issue as when it was first written.

Sorry I can't be more helpful, but if we come up with a way to make this work then we'll post it here, so stay subscribed, and thanks for using PyTools.

Nov 16, 2012 at 4:29 AM

I see. So from my understanding, any IDE will run into the same problem, correct? I maybe able to make a case if that's true. Also, I don't have to keep trying other IDE if this is not a specific problem of pytools.

Also, since this is an open source framework, from what you see so far, is there an easy "hack" that I can do to change that behavior? I just want it to work on my local workstation. When the performance is actually an issue, I'll run the script on a computing grid anyway.

I'll stay subscribed. Thank you for your help!

Coordinator
Nov 16, 2012 at 6:33 PM

Any IDE that does not actually execute the code should have the same issues, but there may be an IDE that can do it (though I don't know of any myself, and there are a lot of potential issues when running arbitrary Python code for IntelliSense purposes).

I don't think the performance is actually an issue, at least not now. It looks like the original version was written around Python 2.2 (about 10 years ago). As for making it work, it doesn't look like there are any easy hacks since the Python types just don't exist until they are lazily created. Probably the best option is a script to generate our database files from some other source (documentation, or a plain-text file with class and method names). I'm not sure whether such a source exists (or could be generated from the C++ files?) but it's quite easy for anyone to extend our database:

The files are in %LocalAppData%\Python Tools\CompletionDB\{interpreter GUID}\{version} - so for your project that will be CompletionDB\2af0f10d-7135-4994-9156-5d01c9c11b7e\2.6. The .idb files are just Python dictionaries written out with pickle, so you can read them with

x=pickle.load(open('{filename}.idb', 'rb'))

and write with

pickle.dump(x, open('{filename}.idb', 'wb'), pickle.HIGHEST_PROTOCOL).

You should just be able to add files (restart VS) and get the new completions, whether the Python packages exist or not. (The '.$memlist' files are not essential - they are used to improve performance when regenerating the database, which will also revert any changes made by hand.)

Now, actually doing this is a pretty involved task and clearly well beyond what we expect of our users. Then again, since our users are developers and quite often highly motivated developers, we're happy to have this info out there to see what people can do with it. If you come up with a script to generate DBs for ROOT, we'll happily point people towards you when they ask.

Alternatively, you could look at modifying the PythonScraper.py file (in the PyTools install directory, either under "%LocalAppData%\Microsoft\VisualStudio\<version>\Extensions\Microsoft\Python Tools for Visual Studio\1.5" or "%ProgramFiles(x86)%\Microsoft Visual Studio <version>\Common7\IDE\Extensions\Microsoft\...") in a way that could be contributed back for our next release, though I'm inclined to see this as a specialised case that would be better served by an independent script. (We will look at updating PythonScraper.py as well, but other features will probably take priority unless it turns out to be really easy.)

Nov 16, 2012 at 8:42 PM

There are automatically generated references to all class online here: http://root.cern.ch/root/html534/ClassIndex.html Though since it's not local, I guess that doesn't help? I probably won't try to write a script myself because I'm more of a user in this case. It's really too bad they decide to make PyROOT this way.

Well, I'll keep an eye on this thread and hope you can do some magic. Thanks again.

Apr 13, 2013 at 10:05 PM
Hi,

I searched around for a clue to see if there is any update from ROOT team on this part. Here is one interesting thing I found.
Jean-François,

given that ROOT can auto-load classes (not to mention auto-generate e.g. STL in the cling-world), "from ROOT import *" is rather ill-defined ...

Making tab-completion (and dir() through dir) work is an independent matter. That could look e.g. in the class table of already known classes as well as the set of globals or even rootmap entries (leaving the actual loading out of the picture).

Cheers,
Wim
from http://root.cern.ch/phpBB3/viewtopic.php?f=14&t=16127&p=69692&hilit=tab#p69689. Please check especially the last three post.

Again, I'm not sure how much this is relevent to my question, but it seems related.
Coordinator
Apr 15, 2013 at 5:29 PM
It's relevant, but Wim's fix won't our issue except in the Interactive Window, which is the only place we use dir() to find members.

We try and avoid executing code wherever possible, since there's no telling what it may do. If someone puts an "shutil.rmtree('.')" in a module, we don't want to be the ones responsible for making it happen (or making it happen in the wrong place).

Ultimately, it comes down to Python's dynamic nature making it extremely hard to obtain some information at design time. When it is used to dynamically generate classes (as it is here, and in many other libraries) there is nothing we can do. Static analysis requires static definitions.
Apr 15, 2013 at 7:08 PM
Understood. Thanks for looking into it.