1
Vote

Visual Studio 2010 freezes with PTVS 2.0RC

description

Visual Studio 2010 freezes for a minute or so early in each session. Most recently it occurred while typing I was typing "import exceptions" so I guess it's something to do with autocomplete.

During this time the devenv.exe process maxes out one CPU core and uses over 2GB of memory. Afterwards the memory usage drops to 1.4 GB.

I'm using a virtual environment with about 30 packages installed so I guess that's got something to do with it. I get the packages using setuptools.setup(), with the following list of required packages:
'pyramid',
'SQLAlchemy',
'transaction',
'pyramid_tm',
'pyramid_mako',
'pyramid_debugtoolbar',
'waitress',
'zope.sqlalchemy',
'redis',
'MySQL-python',
'celery',
'numpy'
Please let me know if you'd like further details.

file attachments

comments

Zooba wrote Oct 2, 2013 at 8:00 PM

Unfortunately, it's a known issue, and as you correctly surmised, it's because we're loading the completion DB into memory.

Long-term we'll fix it by switching to a better DB format, but that won't be in either 2.0 or 2.1. For 2.1 we should be able to move most of the loading to a background thread, which will at least prevent VS from freezing.

However, the memory usage you're seeing seems excessive. Could you do a pip list or screenshot the list of packages in Solution Explorer? (You said "about 30 packages" and listed 12, so I'm assuming there's more.) I'd like to try and repro that internally, in case we have some other problem there.

mplatings wrote Oct 3, 2013 at 2:00 PM

Thanks for your response.

pip list:
amqp (1.0.13)
anyjson (0.3.3)
billiard (2.7.3.32)
celery (3.0.23)
kombu (2.5.14)
mako (0.9.0)
markupsafe (0.18)
mysql-python (1.2.4)
numpy (1.7.1)
pastedeploy (1.5.0)
pip (1.4.1)
pygments (1.6)
pyramid (1.5a2)
pyramid-debugtoolbar (1.0.8)
pyramid-mako (0.2)
pyramid-tm (0.7)
python-dateutil (2.1)
redis (2.8.0)
repoze.lru (0.6)
setuptools (0.9.8)
six (1.4.1)
sqlalchemy (0.8.2)
transaction (1.4.1)
translationstring (1.1)
venusian (1.0a8)
waitress (0.8.7)
web-services (0.0)
webob (1.2.3)
zope.deprecation (4.0.2)
zope.interface (4.0.5)
zope.sqlalchemy (0.7.3)

Zooba wrote Oct 3, 2013 at 7:31 PM

Thanks. I've recreated the env and don't see the excessive memory usage. Are you using or can you upgrade to our latest dev build and see if the issue goes away? You may also want to try deleting the (hidden) .ptvs folder in your virtual environment and recreating the DB from scratch.

I also noticed that we don't provide any completions for zope (as installed using pip - did you install it a different way?) because they use tricks in the .pth files instead of just listing the directories. If you're getting completions for zope then I'll try with a normal .pth file and see if it's any different.

sgottfried wrote Nov 25, 2013 at 2:55 PM

HI Zooba,
could you please explain what you mean with zope packages using tricks in the .pth files. What is different and how this affects auto-completion?

Zooba wrote Nov 25, 2013 at 4:48 PM

Here is the contents of the zope.interface-4.0.5-py2.7-nspkg.pth file on my machine right now (may not be the latest version):
import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zope',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('zope',types.ModuleType('zope')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)
.pth files are supposed to be a list of directories. For comparison, here is setuptools.pth:
./setuptools-1.1.6-py2.7.egg
However, zope takes advantage of a workaround/feature/bug/exploit in Python that allows it to execute arbitrary code. zope appears to be using this to create a fake module and set it's __path__ variable.

PTVS does not and cannot execute this code when looking in .pth files, and so we don't know that the site-packages\zope directory is importable (since it isn't - zope has made a fake module that is importable).

The ironic thing about this is that zope could easily distribute an empty __init__.py file in the site-packages\zope directory and everything would behave identically without the .pth files (as far as I can tell), and PTVS would recognise everything correctly.

sgottfried wrote Nov 26, 2013 at 9:59 AM

Hi Zooba,
thanks for your detailed reply. I used to be a Zope/Plone developer currently using pyramid to create web applications that are supposed to run on Azure as well. Usually I use zc.buildout to setup python environments and until this question/your answer I never came across a *-nspkg.pth file.

That made me curious. A first guess for 'nspkg' was that it relates to namespace packages and that pip is handling them differently that easy_install. I remembered because I read these lines just yesterday at the RTD page for current pyramid 1.5 branch .

Why easy_install and not pip? Pyramid encourages use of namespace packages which, until recently, pip didn't permit. Also, Pyramid has some optional C extensions for performance. With easy_install, Windows users can get these extensions without needing a C compiler.

My first step was to deploy a project using default Django Application template using Django Web Site/Cloud Service Tutorial and Azure SDK 2.2. Local Debugging, Deploy to Azure Website, Deploy as Cloud Service, everything worked fine - awesome.

Just my second task was to try 'easy_install' to install 'pyramid==1.5a2' into a virtualenv WITHOUT running the command as administrator. Why this option is enabled by default??? Anyways, this is another topic.
All worked fine. And now it gets interesting.
Virtual environment was successfully created at 'c:\users\sascha\documents\visual studio 2013\Projects\PyramidTestProject1\PyramidTestProject1\env'
    Installing 'pyramid==1.5a2'
    Searching for pyramid==1.5a2
    Reading http://pypi.python.org/simple/pyramid/
    Best match: pyramid 1.5a2
    Downloading https://pypi.python.org/packages/source/p/pyramid/pyramid-1.5a2.tar.gz#md5=22ea572c42216f298342c9ce70ee688f
    Processing pyramid-1.5a2.tar.gz
    Running pyramid-1.5a2\setup.py -q bdist_egg --dist-dir c:\users\sascha\appdata\local\temp\easy_install-au6vrq\pyramid-1.5a2\egg-dist-tmp-gscifs
    Adding pyramid 1.5a2 to easy-install.pth file
easy_install does not create *-nspkg.pth files, but pip is doing that. After using easy_install to install pyramid PTVS auto-completion knows about zope.deprecation and repoze.lru - typical namespace packages - but not about zope.interface. The tooltip over the progressbar during a refresh of the completion DB shows what packages are scanned (you obviously know this :-) ). After completion the question mark button shows an error/exception for venusian - please investigate further.
Database at C:\Users\sascha\Documents\Visual Studio 2013\Projects\PyramidTestProject1\PyramidTestProject1\env\.ptvs\1839018e-9bf5-4076-9654-e27094461a29\2.7 does not contain the following modules:
venusian.tests.fixtures.subpackages.childpackage.will_cause_import_error
Summary: It is all about python package management - one of the most demanding topics for advanced python developers/tool vendors - the domain you guys are in. Currently your default seems to be using pip. Like the pyramid RTD pages claim: easy_install seems to offer some advantages if packages have optional C extensions. Besides that I do not know all details, you PTVS guys should use that as much as possible. It is a great time for tool vendors. setuptools and distribute have merged again, pip is requiring setuptools again - both now in version 1.4.1 - and the python community is trying its best (PEP 402 and successors) to offer one way to do it.



Further reading
Attachment
  • Screenshot of directory 'site-packages' in virtualenv after installing pyramid using 'easy_install' with PTVS

Zooba wrote Nov 26, 2013 at 5:14 PM

Why easy_install and not pip?
Because easy_install has been deprecated and pip has been officially named its successor. Python 3.4 is going to ship with pip included.
With easy_install, Windows users can get these extensions without needing a C compiler
The "response" to this is that Pyramid should publish a wheel and users should use pip install --use-wheel. This is obviously far from ideal, but the next version of pip will make using wheels the default. We want to support the core Python developers in this work, which is why we make pip the default rather than easy_install. (If we didn't recognize that easy_install is still necessary for Windows developers, it wouldn't be there at all :) )
... WITHOUT running the command as administrator. Why this option is enabled by default???
On some systems, easy_install.exe always runs as an administrator (possibly because of having install in the name...), so we enabled the option by default for compatibility. But we did it before the checkbox was in the dialog, so I'll change it to off by default, but you can also change it through Tools-Options-Python Tools.
easy_install does not create *-nspkg.pth files, but pip is doing that.
Actually, I suspect zope is responsible for the file (repoze does something similar). I'd have to look at its setup.py file, but pip largely does what it is told.
After completion the question mark button shows an error/exception for Venusian
I don't see this on my own machine, so we may have resolved it since 2.0 was released.
It is all about python package management
Yes it is. This is why I'm actively following (and occasionally contributing to) the distutils-sig mailing list. The focus is predominantly on venv, pip, and wheel, and I believe this is the correct direction. We're also working internally (at Microsoft), in PTVS and on python-dev to make things easier when it comes to building Python/extension modules on Windows.

sgottfried wrote Nov 27, 2013 at 6:18 PM

setuptools or pip
Of course I meant setuptools and wrote 'easy_install' because it is the name in the PTVS dialog. These names really can mix up a discussion. Since Version 1.4 pip requires setuptools. What follows is from pip 1.4 changelog

To satisfy pip’s setuptools requirement, pip now recommends setuptools>=0.8, not distribute. setuptools and distribute are now merged into one project called ‘setuptools’. (Pull #1003)
Namespace package installed by pip creates *-nspkg.pth files
Look at this screenshot attachment I provided to this issue. I used setuptools provided easy_install command to install pyramid. All packages are in 'site-packages' as self-contained eggs, there are no -nspkg.pth files or any .egg-info directories.

pip install packages in a different way creating .egg-info files and directories with same name as the package. For namespace package pip creates an additional file called -nspkg.pth. Thats why you see this file for zope. and repoze. packages. I am pretty sure about this, I did it today on Linux and Windows - behaviour as I described. I do not want to say that one is better than the other - honestly I am far from doing that. We started this conversation to explain where these -nspkg.pth files are coming from. And this is what I found out until now. So using pip install --egg would be an option, if PTVS developers do not like -nspkg.pth files because they are currently not welcome to the feature 'Auto-Completion'.
Excerpt from pip install --help

--egg             Install as self contained egg file, like easy_install does.
It is all about python package management
I am pretty sure now the PTVS team is making giant leaps. Nice to know you follow distutils-sig and you are eating your own dog food (PTVS). Keep up good work.

Zooba wrote Nov 27, 2013 at 7:03 PM

So using pip install --egg would be an option, if PTVS developers do not like -nspkg.pth
The discussion here makes it clear that it's all due to setuptools, and that pip is providing a different option to what easy_install does. The --egg option you mention is what was added to avoid passing the other option (--single-version-externally-managed). (Your screenshot clearly has an old version of pip and setuptools, but the behavior seems to be the same for me with the latest.)

However, because setuptools is responsible for creating these files, we can more reliably handle parsing them because we know the format is consistent - hopefully we can get this change in for PTVS 2.1.

Also, it's not widely advertised, but if you want to pass --egg when installing packages through PTVS, you can do that. The text box for the package name is passed directly to pip, so typing zope.interface --egg is going to work fine.

sgottfried wrote Nov 27, 2013 at 9:52 PM

Thanks for clarification.