How to use existing PYTHONPATH value

Feb 10 at 9:03 PM
It appears as of version 2.0 of PTVS, if the Search Paths are left blank, the value of the PYTHONPATH environment variable is blank, as opposed to the previous behavior of keeping the existing value. This is a breaking change for our usage.

We have a launcher application that downloads a user selected version of our in-house library (if it's not already downloaded), sets the PYTHONPATH environment variable, then launches either Visual Studio 2010 with PTVS, or opens a command prompt. We do not install in Site Packages as our users often need to change versions when testing. Since their computer may not be connected at times, we keep multiple versions locally and use PYTHONPATH to point to the desired version.

Our users need to be able to run from command prompts as well as PTVS in Visual Studio. PYTHONPATH has worked very well for this up until now. How can we accomplish this with PTVS 2.0? Setting Search Paths is not an option since our users need to be able to change versions of our library for both the command line and Visual Studio environments. Forcing two different ways to do this on them would cause much grief, confusion and support time.
Coordinator
Feb 10 at 9:39 PM
Are your users mostly working in projects/editor or using the Interactive windows?

Unfortunately, we 'fixed' that issue[1] but didn't leave any way to somehow retain the same behavior.

One solution that would work for both is using a *.pth file in the Python install - all directories listed in C:\Python27\Lib\site-packages\*.pth (or whatever the path is for you) are added to sys.path, just like PYTHONPATH. You can create one of these files and overwrite it whenever you like. Python will handle it the same as PYTHONPATH (and hence our interactive window will work), and when we refresh the completion DB for the editor we will also scan those directories.

Something else to consider is installing multiple versions into site-packages under different names, then use an import ... as statement (for example, having site-packages\mymodule1, site-packages\mymodule2 and then doing import mymodule1 as mymodule). You could also make multiple copies of the Python interpreter and set up custom environments to refer to them - this may add some training requirement but would save having to launch VS using a script. (Virtual environments are probably not a good solution since we require a project to use them, but that depends on how your users are using PTVS.)

I'm afraid there's not much more I can offer. I can go into more specifics about anything I've suggested, especially if you can give me an idea of how PTVS is being used, and we're certainly open to making changes to PTVS 2.1 to improve this scenario.

1: System-wide values of PYTHONPATH cause more problem than they solve, especially when people have multiple interpreters installed. Yours is the first genuine use case we've heard where keeping the value makes sense - most people don't set environment variables before running VS.
Coordinator
Feb 10 at 9:50 PM
Actually, there is a workaround - just set your search path to be $(PYTHONPATH) in project properties (this can't be done via "Add Folder to Search Path" command, only in the textbox on the project property page!). MSBuild will expand this to the value of the environment variable when loading the project, and PTVS will then propagate it to the child process. Solution Explorer should show the actual folders resulting from this expansion.

Note that using "Add Folder to Search Path" will cause the variable to be expanded when saving the project file, so you'll need to avoid that command if you use this workaround.
Mar 12 at 6:12 PM
To answer how our users use PTVS, it varies greatly. Some will create a project from existing files. Some spend a lot of time in the Python Interactive window. Most also do some work at the command line. Most of these users are engineers, but not software engineers. They are not very familiar with Visual Studio, so we try to keep things as simple as possible.

The workaround of putting $(PYTHONPATH) in the search path in project properties is not a good long term solution. I have confirmed that it works, but is too fragile.

Writing a .pth file in site-packages may be an option, but it seems a little heavy handed. The beauty of the environment variable is that it is easily changed, and scoped just to the process and child process where we set it. Actually, if someone wants to run two versions of our library at the same time, side by side, to do some sort of validation, this could be problematic.

The custom environments option is just digging in too deep into Visual Studio for our users, and doesn't work at all for people at the command line.

I would like to request a change to be able to retain the value of PYTHONPATH that we set before we launch Visual Studio. Perhaps a checkbox in the debug project properties page next to the search paths textbox. I personally think however it is implemented, the default behavior should be to retain the existing value of PYTHONPATH, with an option to replace it with what is set via PTVS. I think the change made in 2.0 is a bit heavy handed. However, as long as there is a reliable way to keep the existing value, that is a one time change, then we can document that for our users.
Coordinator
Mar 12 at 8:23 PM
I daresay you'll be pleased to hear that you've ignited some serious discussion within our team on how best to handle this :)

We are currently considering a few approaches, so it would be interesting to hear whether either of them would work better for you:

1. Add a PTVS option (either in the UI or registry/both) that would automatically add PYTHONPATH everywhere we use search paths.

Pros:
  • Brings back the old behaviour
  • Should silence the regular complaints/bug reports that "PTVS can't find the package I installed"
  • Applies to projects and the interactive windows
Cons:
  • Search paths are treated as part of the project, so if PYTHONPATH refers to >1000 files, updating IntelliSense is slow
  • Still leaves the issue of incompatible packages in PYTHONPATH
  • Global solution for a local problem
2. Add a project option (in the project file) to include PYTHONPATH in search paths and automatically update it.

Pros:
  • Lets users opt-in where they need it
  • Very visible in the UI
Cons:
  • Only affects interactive windows if the open project is using the same interpreter as the window
  • Requires new UI
  • Projects may behave differently on different machines
  • Search paths are treated as part of the project
3. Look for environment variable PTVS_PYTHONPATH and use its value as the default (instead of an empty string)

Pros:
  • Lets users opt-in to the packages they need
  • Scoped to the current process
Cons:
  • If package installers start setting it, we suffer from the original problem again
  • Not easy for users to discover or set
  • Search paths are treated as part of the project
4. Look for environment variables like PTVS_PYTHONPATH_27_X86 and use their values as PYTHONPATH for a matching interpreter

Pros:
  • Largely solves the compatibility issues
  • Lets users opt-in to the packages they need
  • Package installers can set just the variable that applies to their package (or preferably not at all)
  • Scoped to the current process
Cons:
  • Not easy for users to discover or set
  • Search paths are treated as part of the project
The "search paths are treated as part of the project" con exists for all of these options right now, but at some point we plan to change this so that search paths are analyzed once and the results saved (like we do for the standard library/site-packages). Once we set that up, the main remaining issue is discoverability. Even with the current behaviour, this is probably still better than when it was working for you, since you would not have had such a good IntelliSense experience.

Do any of these options sound better than the rest for you? Also, we're interested in your broader setup (who your users are, what sort of work do they do, what packages do they use, how could PTVS be more helpful for them, etc). If you've got time and feel inclined, could you drop us an email at ptvshelp@microsoft.com? (This is private to our team.) It sounds like your scenario is relevant to our next major focus, so we'd love the opportunity to discuss things in more depth.
Mar 25 at 10:03 PM
I think some sort of combination of the first two options make the most sense:
  • There should be an option on a project basis to allow users to either include or exclude the existing PYTHONPATH value in the module search path. This needs to always dynamically read the value of PYTHONPATH, and not just copy the current value. This is really the way to “play nice” with how Python, and the PYTHONPATH environment variable were designed.
  • There could be a setting (in the Tools->Options settings for PTVS) to set the default value of the project setting. This would be used for each new project. Once it is set in a project it would stick. I would vote to have the default value for this default value to be true (include PYTHONPATH).
I have a question. For the project option, you said “Only affects interactive windows if the open project is using the same interpreter as the window”. I don’t quite follow what you mean here. I also don’t know why there would be a different behavior between the Python process launched by PTVS, and the one hosted in the interactive window. Obviously, we would want consistent behavior in both.

I am not a fan of the new, custom environment variables. As you mentioned, discoverability for the use of these would be very poor.

Frankly, I think the problems you are trying to address by ignoring PYTHONPATH should not be addressed by PTVS. It should be addressed by the user of the system. If PTVS does something to avoid the problems, then what about when the users fires up the default IDLE Python GUI, or a command prompt to do something. I do this all the time for quick little tests or whatever. I wouldn’t want different modules found in those environments that in PTVS.
Coordinator
Mar 25 at 10:44 PM
Thanks for the feedback. We've decided that we will add a global option to clear PYTHONPATH through PTVS (the current behaviour), which will be on by default. Users (or an install script - anything with registry access) can disable this, which will leave the value of PYTHONPATH alone.

We have actual data from the Customer Experience Improvement Program, and well over half of our users have multiple versions of Python installed. Changing the default behaviour has the potential to affect too many people.

I know it's a big switch to make available, but I think it's the best approach right now:
  • We're on a tight schedule, and may not be able to get something more complicated ready in time for 2.1
  • You are correct, it's not really our problem to address. Users should be the ones to fix it, and those who do can let us know that PTVS does not need to protect them.
  • People whose code is currently working in PTVS will not be broken unless they actively change a setting.
  • Per-project values can be set as search paths or in the project file itself.
I'll also point out that the PYTHONPATH variable was designed to be set immediately before starting Python, and the ones misusing it are the installers that set it globally (arguably this is the Windows model vs. the Unix model, but Python came from the Unix world). PTVS has always set the value in the correct fashion based on the required search paths, and unfortunately your situation was overlooked when we decided to take that approach.

Hopefully this opt-in approach will work for you. Once we have a build available with the change I'll let you know, and give you the details on how to set the option from outside of VS.
Jul 14 at 10:58 PM
What is the status on this issue? I am performing a new install of PTVS, and currently, as far as I know, we are stuck on version 1.5 until this issue is addressed in some way.
Coordinator
Jul 14 at 11:35 PM
We added an option to globally (within PTVS) take the value of PYTHONPATH set for VS into account, so if you enable that then you should get the behaviour you're after. IIRC, it's under Tools, Options, Python Tools, General (in an airport now and can't check, sorry).

It should be taken into account everywhere, but it's possible that we've missed some places. If you try out our 2.1 Beta 2, let us know if there's anywhere it doesn't work and we'll get it fixed for the final release.
Coordinator
Jul 14 at 11:37 PM
A second thought: it may not be used for the cached database, so completions may be limited or unavailable. I'll give some thought to how we can deal with this... our current design doesn't easily support it and we can't do the planned redesign for 2.1 at this stage, but there may be a workaround/hack we can use.