1
Vote

PTVS + Anaconda distribution

description

I'm trying to get PTVS + Anaconda distribution working. Anaconda installed in C:\Python27\Anaconda. win7 64bit, python 64bit, pvts 2.0 rc with vs 2013 rc. I get the error that the completion database fails. However completion DB does NOT fail for C:\Python27, and for IronPython.

ANAPYTHONPATH=C:\Python27\Anaconda\;C:\Python27\Anaconda\Scripts\

file attachments

comments

denfromufa wrote Oct 6, 2013 at 6:17 PM

This is the error:

Database at C:\Users...\AppData\Local\Python Tools\CompletionDB\12.0\c76002b0-1b6c-4c57-b51d-a4f29bc1efb8\2.7 is corrupt or an old version

Zooba wrote Oct 6, 2013 at 6:54 PM

Under the Tools\Python Tools menu you'll find a menu Diagnostic Info. Can you copy the contents of that into a text file and either attach it here or email it to ptvshelp@microsoft.com (in a zip file if necessary)? You can also look through at the errors and resolve them yourself if you like - it's probably just a configuration mismatch.

denfromufa wrote Oct 7, 2013 at 12:54 AM

thanks, I have sent it.

Zooba wrote Oct 7, 2013 at 4:49 PM

Thanks. I've had a look and I can see why we're failing (out-of-memory because we're not correctly splitting libraries into chunks) but it's not obvious what the cause is.

I'll investigate today and let you know if there's a workaround you can apply yourself.

denfromufa wrote Oct 7, 2013 at 6:32 PM

additionally why does the PTVS parser of libraries has to do everything from scratch when new libraries are added?

Zooba wrote Oct 7, 2013 at 6:51 PM

If the only files added are inside subdirectories of site-packages, then it shouldn't. When files are added to Lib or Lib\site-packages directly then we need to redo everything because we allow packages to import these modules. (Alternatively, if you change the standard library we redo everything, and we consider modules (not packages) in site-packages to be part of the standard library.)

For example, six.py is normally installed into site-packages and then used by packages. We treat it as part of the standard library so that this works with our (admittedly not ideal) analysis.

It looks like your issue is that we're treating all of the packages in site-packages this way, which is what's causing you to run out of memory. I'm still not sure why we're doing this.

What version of Anaconda are you using? 1.7.0?

Zooba wrote Oct 7, 2013 at 11:44 PM

So I haven't been able to figure this out today. I suspect that something has gone slightly strange with your Anaconda installation.

Could you run the following command in a command prompt and attach or email the results?
dir /s/b C:\Python27\Anaconda\* > FileList.txt

Zooba wrote Oct 8, 2013 at 4:27 PM

Actually, don't worry about that. I've spotted the problem, and it's easy for you to resolve :)

At the top of your log file is this configuration for Anaconda (I'm republishing it for when people run into this issue in the future):
        Factory: Anaconda
        Version: 2.7
        Arch: Amd64
        Prefix Path: C:\Python27\Anaconda
        Path: C:\Python27\Anaconda\python.exe
        Windows Path: C:\Python27\Anaconda\pythonw.exe
        Lib Path: C:\Python27\Anaconda\Lib\site-packages
        Path Env: ANAPYTHONPATH
There are two corrections that need to be made. The main one is that Lib Path (Library Path in the Environment Options dialog) should be C:\Python27\Anaconda\Lib - without site-packages. The second is that the Path Environment variable should be PYTHONPATH - we need to make this clearer, but the value here is what the interpreter expects (we use this to pass Search Paths from the project).

Once you've fixed up the library path, you should be good to go. Sorry I didn't spot it earlier, and I'll make sure we get improvements to that dialog in for our next release (after 2.0).

denfromufa wrote Oct 13, 2013 at 3:47 PM

Thanks for figuring out my issue! The solution worked for me.
  1. Why does IronPython have it own Path Env. variable?
  2. How is PYTHONPATH used for Anaconda distribution if it has its own Python executables and libraries? What has to be added to PYTHONPATH from Anaconda in order for Anaconda to work with PTVS properly?

Zooba wrote Oct 13, 2013 at 4:57 PM

The PYTHONPATH environment variable is a way of adding entries to the sys.path list from outside of a Python script. sys.path is used when importing modules - each path in the list is searched for a module matching the name that you're trying to import (along with a paths derived from the location of the module doing the importing). So if you have modules in a folder that isn't in sys.path by default, you can set PYTHONPATH before running python.exe to add it in. For example:
C:\Python27>python.exe -c "import sys, pprint; pprint.pprint(sys.path)"
['',
 'C:\\Python27\\python27.zip',
 'C:\\Python27\\DLLs',
 'C:\\Python27\\lib',
 'C:\\Python27\\lib\\plat-win',
 'C:\\Python27\\lib\\lib-tk',
 'C:\\Python27',
 'C:\\Python27\\lib\\site-packages']
When we set a value, it gets added in near the top of the list.
C:\Python27>set PYTHONPATH=C:\Another\Folder\Somewhere
C:\Python27>python.exe -c "import sys, pprint; pprint.pprint(sys.path)"
['',
 'C:\\Another\\Folder\\Somewhere',   # here it is
 'C:\\Python27\\python27.zip',
 'C:\\Python27\\DLLs',
 'C:\\Python27\\lib',
 'C:\\Python27\\lib\\plat-win',
 'C:\\Python27\\lib\\lib-tk',
 'C:\\Python27',
 'C:\\Python27\\lib\\site-packages']
Unfortunately, on Windows, environment variables are typically set once and used globally. IronPython uses a different variable name because it's likely that packages designed for IronPython will cause errors when imported in CPython. But because all versions of CPython use PYTHONPATH, you can still get conflicts (for example, if you import a module written for Python 3.3 in Python 2.7, or a 64-bit PYD file into a 32-bit interpreter).

These days, sys.path is mostly managed automatically (through tools like pip and virtualenv), so there's no need to use PYTHONPATH. Because it can cause issues, we actively discourage it in all but one case. The one case where it is "allowed" is when you set it immediately before running python.exe and only for that process (like in the example above).

In PTVS, we expose this through Search Paths. When you add a Search Path to your project, we will set PYTHONPATH (or whatever variable name is provided) to those paths whenever you run or debug your project. Further (and to the annoyance of a number of people), if there is a global value for PYTHONPATH, we ignore it, since there's a good chance that it was set by an installer that should not have used it. And there's no harm in being explicit about which code you want to execute - PYTHONPATH can very easily be used to hijack your code:
C:\Python27>set PYTHONPATH=C:\Python33\Lib
C:\Python27>python.exe
  File "C:\Python33\Lib\site.py", line 173
    file=sys.stderr)
        ^
SyntaxError: invalid syntax
So when you provide the name of this variable in PTVS, we never actually read it - we write to it and the interpreter reads it. But if you've installed a package from PyPI or into your site-packages folder, there is no need for PYTHONPATH to be set at all, since those paths will be included automatically.

(Hope that answers both questions, and apologies if I'm repeating things you already know.)