Test Adapter fails from vstest.console /InIsolation


If vstest.console.exe is passed /InIsolation or /EnableCodeCoverage the PTVS Test Adater fails in TestExecutor.cs:RunTests from these sequence of events:
  • Due to Isolation being used the Test Adapter is running inside vstest.executionengine*.exe.
  • The execution engine gets passed a /parentProcessId argument, but that PID isn't Visual Studio.
  • VisualStudioApp.FromCommandLineArgs() returns a non-NULL instance of VisualStudioApp.
  • InterpreterOptionsServiceProvider.GetService() receives the non-NULL instance.
  • InterpreterOptionsServiceProvider.GetService calls app.DTE which returns NULL due to the parent process not being VisualStudio.
  • InterpreterOptionsServiceProvider.GetService passes in NULL to "new ServiceProvider" which throws an ArgumentNullException and the PTVS Test Adapter ends up crashing as a result.
If we avoid trying to create the "new ServiceProvider" then the Test Adapter successfully runs the Python test cases.

Given the various ways one could go about solving this problem it seems easiest to let someone on the project pick their way. :)



tuttb wrote Aug 30 at 2:38 AM

This is with 2.1 RC with bits downloaded today from codeplex.


tuttb wrote Sep 2 at 4:24 PM

This is particularly annoying because /EnableCodeCoverage forces isolation and my unit tests utilize win32com to perform some COM smoke testing of my C# COM library.

I could write up a pull request that would adjust FromCommandLineArgs to acquire DTE just after construction if you like and return NULL if DTE ends up being NULL.


Zooba wrote Sep 2 at 4:56 PM

Thanks for the analysis. There are some changes we can make to support running outside of VS, though it will cause some restrictions.

Projects that use custom Python environments will not work (nor will virtual environments created from custom environments), as without a running VS instance we can't get to the saved settings that specify them. There's really no way around this problem other than forbidding running outside of VS.

Another issue is that the changes required to prevent the crash you're seeing but still work for the settings we can find are too significant for us to take in PTVS 2.1. At best, you'll get an early error message saying that tests cannot be run outside of VS, which is not really much better than the crash (though it does explain the reason). The pull request you suggest wouldn't be sufficient, as even though it looks like the later calls will be fine with app == null, they won't actually work correctly (these checks are mainly there for our unit tests, where we can mock certain things out that are necessary for actual use).

And frankly, the main reason we haven't spent much time trying to support this is that Python will do a better job of running the tests if you run them directly. Running python -m unittest discover -v in the tests directory of your project will run all of the tests more efficiently than our test adapter does and provide equally detailed results. You don't get a TRX file of results, but you do get a non-zero exit code if anything failed, which should be sufficient for useful automation (though not ideal, I agree).

What we would really appreciate is a test runner (in Python) that can communicate with another process over a socket. Then we can execute tests more efficiently by keeping the process running and getting individual results back out of the runner. We are also considering having a test metadata file (auto-generated by VS, but user editable and suitable for checking into source control if desired) that would make running tests outside of VS far easier to set up. This sort of change would not be in 2.1 though, so I do think the best immediate solution for you is to try and launch Python directly for the Python tests.

tuttb wrote Sep 2 at 6:12 PM

Hrm. I hacked up a change to PTVS to deal with the NULL DTE and my tests now run correctly. I guess I must be lucky because I only have one Python environment installed on my machine atm.
(2.7 32bit) (Although now that I write this I'm not sure if I verified that code coverage was actually collected in the vstest.console.exe scenario...)

For my use case (needing C# code coverage called form a Python win32com test case) the test metadata file generated by VS to supply the environment details the Test Adapter is missing seems like the best idea.

I know TRXs are slower, but you can do some really useful things with them: Coverage, perf counters, etc....
It also reduces the number of CI test log parsing modules someone (or myself) needs to write.

I totally hear you about running the unit tests directly from python when feasible being much more efficient.
If I didn't need code coverage the vstest.console.exe way or the convenience of only having one XML log file to parse I would definitely just launch python.exe myself.

I don't know much about the environment settings for python so I can't comment on whether or not it would make sense to check the files in. I could envision you only wanting to check some of them in. (like checking in entire virtualenvs or something, but not checking in the Python 2.7.x global environment maybe)

Thanks again,