No IntelliSense on own pyd

Sep 18, 2013 at 12:44 PM
Edited Sep 18, 2013 at 1:00 PM
I have two pyd modules, that are C/C++ wrappers around C# code. I'm trying to reference them in a Python Tools project and get IntelliSense working. My Python project it self runs fine.

I tried placing the two pyd files in the DLL's folder of my Python installation, and did a refresh of the completion DB. But with no luck. Also opening the pyd file in the object browser, results in an empty view. What am I doing wrong and how can I fix this?

Thanks in advance. The project looks very nice, but at work I have to compete with a college who is using PyCharm.
Coordinator
Sep 18, 2013 at 3:42 PM
You should be able to add a reference in your project. Right click on the 'References' node in Solution Explorer, 'Add Reference' and the 'Browse' tab. From here you can find your .pyd file. (However, I just tried it and found a bug... not sure if it's in 2.0 RC or if it's new, so this may not be working right now. It will be fixed for RTM.)

If you really want to compete with your colleague, you could include your C++ projects in the same solution and add a reference between the two. (This takes a bit more setup, especially in getting the right compilers, but if you're modifying the modules then it's very convenient.) Or you could enable mixed-mode debugging (Project Properties, Debugging) and step between your .py files and .cpp files.
Sep 19, 2013 at 8:34 AM
Thanks for verifying it. I installed version 1.5 in my Visual Studio 2010 installation (running 2.0 in VS2012), and experienced the same problem. Can it be that I have a problem with my pyd file. Is there any way to inspect/verify it? As you could expect, in PyCharm, it all works.

Integrating the Python project in the same solution, was already in my mind.
Coordinator
Sep 19, 2013 at 3:38 PM
We will load the pyd into a running Python process and just use dir() to find the information we need. If you have multiple versions of Python installed, we may be trying to use the wrong one - make sure that your project is set to the right environment (instructions page, or look in Project Properties on 1.5).

I've checked out (and resolved) that bug, and it looks like it wasn't in 2.0 RC, so you should be fine to keep using the later version. We also fixed a number of other things in 2.0 RC that could prevent us from scraping a pyd correctly, so you've got a much better chance with it than with 1.5.

(Alternatively, if you can share your .pyd I'm happy to take a look myself. Put it somewhere we can download it and email a link to ptvshelp@microsoft.com. If we are having trouble with it then there's still a chance we can fix it for 2.0.)
Sep 24, 2013 at 1:48 PM
Sorry for letting the time pass by.
I have downloaded the latest development version of 20 September and at first sight it is an improvement. Now I'm able to add both pyd files. First I got an error message on one of them.
Sharing the pyd is a bit complicated, I need to ask this to a few people inside our company. For the moment a couple of them are on business trip.

But let me explain our setup, we deliver Python with our product. Mainly because it needs to be started with an 'app.config' file in place for the WCF connection settings and logging. We also place the two modules in the same directory.
Beside the standard Python 64-bit 3.3 installation, I have added a Python Environment to our product specific Python installation and selected this for the Python project I'm developing in VS.
I have started this environment in the Interactive Window and after doing an import of the modules, the dir() function returns nicely all the functions of the modules.

When refreshing the Completion DB of my specific environment it keeps saying the DB needs a refresh. When clicking on the question mark, it's says that one of my modules isn't analysed. Maybe this gives you a hint.
Image

A further observation is that I can't open the OScript pyd file in the object browser, it also gives a warning in the Solution Explorer (it also shows the file extension). The OmseScript file looks fine, but opening in the object browser gives no content for the file.
Image

Is there anything I can do to solve these problem; turn on logging, try a certain scenario, ...?
Coordinator
Sep 24, 2013 at 3:54 PM
There are a few logs you could look at, and you may even be able to solve it yourself. But the first question I have is exactly where is the file?

The easiest way to find the logs is to look at our Diagnostic Info page. In VS, click Tools->Python Tools->Diagnostic Info. You'll get a dialog with a lot of text in it. If you're short on time, just copy it into an email to ptvshelp@microsoft.com (there may be some full file paths that include your username - you can sanitise these if you like). Otherwise, feel free to read it :)

Because OScript is a compiled modules, we do all the scraping in Python code - if something has gone wrong there's probably a traceback in the log. We'll be interested in the traceback, but you may also be able to add your own fix to one the .py files in our install directory. (We're on a tight schedule for our 2.0 release, so the sooner you can get this to us the better. I assume OScript is the one that you maybe can't share?)

Also, there are still one or two issues with .pyd references in the build you have, and there will still be some in RTM :( . The most reliable approach is to put them into the environment directly (as you have). We should pick up all .pyds in the same directory as python.exe, in DLLs and in Lib.
Coordinator
Sep 25, 2013 at 8:35 PM
Thanks for the logs. (We like to keep the discussion public where possible so others can benefit from the answers.)

As you noticed, your pyds are failing to load. The error is not helpful, but typically this is due to some dependent DLLs being missing. There's a great tool at http://dependencywalker.com/ that will show you exactly which DLLs are needed for your pyd, and it will show you which ones aren't being found. Normally it's pretty easy to resolve from this point, either by moving/copying the files or changing PATH settings.

And yes, if you've managed to get an .idb file for the module in another environment (with the same Python version) it is substitutable for the one that won't work. Feel free to copy/paste it into another database.
Sep 26, 2013 at 3:39 PM
Edited Sep 26, 2013 at 3:48 PM
I was indeed able to generate *idb files without error after running the ExtensionScraper.py in our own Python environment. Placing these files in the CompletionDB directory of PythonTools, shows me IntelliSense for the modules (after restarting VisualStudio).

Stays the problem that refreshing the DB of the Python Environment gives an error on OScript and that the object browser isn't working. And off-course this is cumbersome, luckily our modules aren't changing that frequently.

I used DependencyWalker and it makes clear that the problem was with the Python version. Our modules are compiled against a specific version, now looking at ABI. After analyzing the logging further, I saw the scraper is already using our custom Python environment and also defines a working directory. The OmseScript was sraped without errors, but the OScript kept throwing errors; log
Now it becomes clear we have made a mistake in this module. Importing it, invokes the init method that goes looking for a config file and also opens a Corba connection. Off course this fails, if the servers aren't running. If I make sure the servers are running, while loading the module, I got IntelliSense working out of the box and our custom Python Environment isn't reporting any errors.

What about the Object Browser? Does this also work on the idb files. I'm still not able to get it work.

(Sounds perfectly right to discuss things in public, I wasn't sure how to handle this.)
Coordinator
Sep 26, 2013 at 4:01 PM
Glad you got it sorted, and it sounds like we helped expose an unexpected design (incidentally, this is the reason we avoid running code inside VS - it may have been crashing entirely!)

Unfortunately we don't show idb files in the object browser. I personally would love to, but we've always got higher priorities. The best chance of getting it is if someone contributes a patch (or if I just do it over a rainy weekend). If you want to look inside, you can grab our source code and build AnalysisBrowser, which is a debugging tool we use, or you can simply use the pickle module to read (or edit ;) ) them in Python.
Sep 26, 2013 at 4:22 PM
Alright, then I maybe misinterpreted the functionality described on the site. How is the Object Browser intended to work for PyTools? Maybe someone can extend documentation about that.

Thanks for helping us. Maybe I will grab the code once and do something with it. Thanks for pointing me to the AnalysisBrowser.
Thanks again, now I'm confident enough to deploy it with my colleagues.

BTW: Did you hear the news of PyCharm, they open sourced it? Still love PyTools for it's integration with VS.
Coordinator
Sep 26, 2013 at 4:46 PM
Yeah, that screenshot is a poor choice - somebody made a project containing the standard library for the example. Object Browser will really only show you the objects in your project.

We're on a bit of a documentation rewrite right now, so expect it to improve :)

(Yep, we heard the PyCharm news. It's good to have some competition in this area - it forces us to try and go one better :) )
Sep 27, 2013 at 5:23 PM
Eum, I'm even not able to get the structure/objects of the current project been view in the object browser. Will I open a new thread for that or leave for what it is?
Coordinator
Sep 28, 2013 at 5:53 PM
This would be https://pytools.codeplex.com/workitem/1920. It's fixed now, but we haven't pushed the sources with the fix to CodePlex yet.