Installing Python packages using pip

Nov 30, 2013 at 9:27 AM
Edited Nov 30, 2013 at 9:27 AM
Hi,

New to both Python and VS on Win 8.1. I've just installed the integrated PTVS 2.0 distribution, set up CPython 3.3 x64 and the SciPy stack.

I am trying to install the astropy package through the integrated pip installer in PTVS. This process fails with
error: Unable to find vcvarsall.bat
From what I can gather on the web, this is because the package needs to be compiled and I guess I don't have a C compiler on my system. Am I supposed to install VS 2013 express (if yes, which one?), a different version of VS or should this be working and I am doing something wrong?

Thanks for any pointers!
Coordinator
Nov 30, 2013 at 9:47 AM
You need to install the version of VS that corresponds to your Python version (i.e. the one it was built with). For CPython 3.3, this is VS 2010. Express edition should work, as well: http://go.microsoft.com/fwlink/?LinkId=323467
Dec 1, 2013 at 11:49 AM
Edited Dec 1, 2013 at 4:02 PM
Thanks.

So I have read a bit and installed the compiler in Microsoft Windows SDK for Windows 7 and .NET Framework 4, following the instructions for Cython given in its Wiki. This is the x64 VS2010 compiler. I have then used the Build Environment command line to set
C:\Program Files\Microsoft SDKs\Windows\v7.1>set DISTUTILS_USE_SDK=1
C:\Program Files\Microsoft SDKs\Windows\v7.1>setenv /x64 /release
just as the wiki above told me. When I now try to install the package astropy using pip through PTVS the compilation seems to happen ok, but the script fails while cleaning up. This is the end of the pip logfile:

  Removing temporary dir c:\users\ignacy\appdata\local\temp\pip_build_Ignacy...
Command "C:\Program Files\Python33\python.exe" -c "import setuptools;__file__='c:\\users\\ignacy\\appdata\\local\\temp\\pip_build_Ignacy\\astropy\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\ignacy\appdata\local\temp\pip-0bv146-record\install-record.txt --single-version-externally-managed failed with error code 1 in c:\users\ignacy\appdata\local\temp\pip_build_Ignacy\astropy

Exception information:
Traceback (most recent call last):
  File "C:\Program Files\Python33\lib\site-packages\pip\basecommand.py", line 134, in main
    status = self.run(options, args)
  File "C:\Program Files\Python33\lib\site-packages\pip\commands\install.py", line 241, in run
    requirement_set.install(install_options, global_options, root=options.root_path)
  File "C:\Program Files\Python33\lib\site-packages\pip\req.py", line 1298, in install
    requirement.install(install_options, global_options, *args, **kwargs)
  File "C:\Program Files\Python33\lib\site-packages\pip\req.py", line 625, in install
    cwd=self.source_dir, filter_stdout=self._filter_install, show_stdout=False)
  File "C:\Program Files\Python33\lib\site-packages\pip\util.py", line 670, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "C:\Program Files\Python33\python.exe" -c "import setuptools;__file__='c:\\users\\ignacy\\appdata\\local\\temp\\pip_build_Ignacy\\astropy\\setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record c:\users\ignacy\appdata\local\temp\pip-0bv146-record\install-record.txt --single-version-externally-managed failed with error code 1 in c:\users\ignacy\appdata\local\temp\pip_build_Ignacy\astropy
I don't really know what any of this means unfortunately. You can see the whole logfile here.

Needless to say, astropy does not appear in Python33\Lib\site-packages\

I have selected the "Run as administrator option". Also I have managed to use
pip install astropy
from the SDK Build command environment. This then makes the compiled astropy package appear in PTVS as it should.

Thanks for any hints.
Coordinator
Dec 1, 2013 at 9:17 PM
Judging by the mention of "VCVARSALL" in the failing part of the log, you may have to run "pip install" manually from command line, and using "Visual Studio Command Prompt" shortcut (which sets some environment variables that are required by the compiler). I'm not sure what the equivalent is for SDK - probably "SDK Command Prompt"?

Using VS Express is likely to be easier here as distutils will just pick it up; the only downside is that it's a bigger download, and you'll have another copy of VS around...

By the way, if you're trying to configure Python for scientific computing (which it looks like, judging from astropy), you might want to consider using one of the Python distros that come preconfigured with the most common packages. In particular, Enthought Canopy has a free Express version, and includes astropy (and numpy/scipy, and many other packages). The only disadvantage is that it is Python 2.7, not 3.3.
Dec 2, 2013 at 8:13 AM
Edited Dec 2, 2013 at 8:18 AM
Thanks for you help. I have managed to solve it, and it's probably worth spelling out what the issue is.

I am using x64 Windows with x64 Python 3.3 and need to compile the packages in 64-bit.

I've installed VS2010 express. Unfortunately, it doesn't come with the x64 compilers, so the only way to get them is to go back to installing the Win7.1 SDK (then followed by the VS2010 SP1 update and a fix for the x64 compilers which the update removes...). I can then compile C code from within VS2010 with the x64 compiler ok.

The problem is that it seems that the x64 compilers in the v7.1 SDK no longer use vcvarsall to set the environment but instead use SetEnv.cmd. Hence files such as vcvars64.bat, which are necessary for the amd64 option in vcvarsall to work are not available on the build system. If you have Vs2010 Pro or ultimate, you will not see this, since they come with the full x64 compilers and the supporting vcvars*.bat files.

However, the fix is simple. To make things work, you have to create a new batch file
C:\Program Files (x86)\Microsoft Visual Studio 10.0\vc\bin\amd64\vcvars64.bat
and put in it a redirection to SetEnv
CALL "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /release
adjusting the paths appropriately.

I haven't tested, but it's quite possible that all of this would work even without VS2010 express installed, but just with the SDK compilers. With this kludge, the pip installer called from inside PTVS downloads and compiles packages fine.

It seems that this is a problem which should be communicated to the authors of distutils.
Jan 4, 2014 at 12:51 PM
I've read this thread with interest. I've installed Python Tools for VS on VS 2013 (premium). I'm new to Python but have resolved to learn it and, hopefully, actually use it :-).

All seems well with the environment but I have difficulties bringing in libraries. Some of the libraries I'm interested in like PIL I've been able to obtain by using easy_install to install Pillow (a 3.3 conversion of PIL). I'm stuck on numpy though. Neither pip nor easy_install will work for it. I get the errors around vcvarsall.bat.

The last post seems to show a way around this:
However, the fix is simple. To make things work, you have to create a new batch file
C:\Program Files (x86)\Microsoft Visual Studio 10.0\vc\bin\amd64\vcvars64.bat and put in it a redirection to SetEnv
CALL "C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin\SetEnv.cmd" /x64 /release adjusting the paths appropriately.
However, I don't quite understand the fix or how to use it in the VS 2013 environment.

In my C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin\amd64 there's a vcvars64.bat. I presume this will work?
I don't understand what it's purpose is or how to redirect to it. Where is the SetEnv? Is it in the Python environment? Is the 'CALL' a command to the command processor or the actual text to be set?

Thanks for any help.
Jan 4, 2014 at 1:55 PM
Hi,

The issues is that Python 3.3 is compiled with VS 2010 and you need to compile the packages with the same version of the compiler. This is why you need to install VS2010 express in addition to VS 2013. Then the problem is that VS2010 does not have a 64-bit compiler, so you need to obtain that by getting the Windows 7.1 SDK as I've described above.

Finally, what I meant by the batch file is that you need to create a file vcvars64.bat in Notepad or some other text editor inside the VS2010 directory and have that batch file contain the CALL command, as I've described.
Jan 4, 2014 at 2:49 PM
Okay, thanks wigster.

Please pardon my ignorance but It seems a bit convoluted since VS 2013 has a 64 bit compiler. Might it be better to get/recompile Python 3.3 with the VS 2013 compiler? Is this something the PTVS team needs to be aware of (I'm sure they are) or, as you mention "It seems that this is a problem which should be communicated to the authors of distutils". The distutils are created by the authors of the libraries?

Again, thanks for your quick response.
Coordinator
Jan 4, 2014 at 6:57 PM
Everyone's well aware of the issue (which is fundamentally a problem with CPython, not PTVS), but there is currently no good solution. I've looked into it and there is no way to compile a working numpy with anything other than the matching compiler. The CPython team will only change compiler between major versions, so we're stuck with VS2010 for now, and they rely on a Visual Studio installation, rather than just the compiler. We're in constant discussion about how to improve it, but right now there's no universal solution.

There are other ways to get a pre-built version of packages, which we list on our installation page (get to it from Home above). The easiest is Golke's site if you've already got a Python installation. If you use easy_install and provide the full path to one of his executables you can even install them into a virtual environment.
Jan 5, 2014 at 2:46 PM
Thanks Zooba. Thanks for the heads up that you know about this issue. You guys are doing a great job with this.

As a .NET programmer it's a bit confusing delving into the open source world with all these different compilers and environments ;-)

So, I've downloaded Anaconda. It's got numpy and a whole host of other libraries that I probably will never need. It's great that PTVS has the environments feature! Makes it easy to switch Python compliers for various purposes.

So I'm on my way playing with all the cool libraries :-)
Jan 6, 2014 at 3:50 PM
One thing about the Anaconda package: the Completion DB won't successfully refresh.
Coordinator
Jan 6, 2014 at 8:38 PM
What version of Anaconda are you using? I've just tested with the latest (1.8) and it completed without any issues for me.
Jan 7, 2014 at 12:25 PM
Edited Jan 7, 2014 at 12:26 PM
Zooba, Here's the title of the zip I downloaded: Anaconda-1.8.0-Windows-x86_64. Are you using the 64 or 32 bit version? I'm using this is VS 2013 premium. -- Thanks
Coordinator
Jan 7, 2014 at 4:44 PM
I tried all the versions they have available for Windows, including that one.

There may be something else on your system that's causing a conflict. In VS, can you open Tools->Python Tools->Diagnostic Info and email the contents to ptvshelp@microsoft.com (with a link to this discussion)?
Jan 7, 2014 at 5:02 PM
Hi Zooba, okay did as you asked. I'll post any responses I get.
Coordinator
Jan 7, 2014 at 5:26 PM
Edited Jan 7, 2014 at 6:03 PM
It looks like the issue is that you've configured Anaconda manually and set its version to 3.3 - Anaconda is based on 2.7. An unfortunate circumstance of using a bundle like Anaconda is that most are based on Python 2.7 (winpython is a notable exception).

The version in our dialog doesn't affect what you have installed. It's to tell us what version to expect, so we can handle the differences. Because we are treating all of Anaconda's libraries as 3.3, there are many errors (search in that log for "parse error") and eventually an out-of-memory error that prevents us from continuing.

Changing the version of your environment to "2.7" (and making any required changes to your own code, sorry) will fix the issue. Alternatively, Anaconda has also registered itself as "Python 64-bit 2.7", so you can use that one instead.
Jan 7, 2014 at 8:09 PM
I see. Okay thanks Zooba. I guess I'll have to learn both :-|