Customizing unit test discovery

Mar 19 at 5:28 PM
Edited Mar 19 at 5:33 PM
The UnitTest integration is very useful.

However, it places several restrictions on how test code should be organized.

Could there be a way of customizing the parameters of the test discovery, e.g. by specifying (optional) values for the file name pattern or the test search path, like options -p and -s from unittest discovery respectively ?
Coordinator
Mar 19 at 5:38 PM
As far as I can tell, those options would place more restrictions on test discovery than we currently have. This may be beneficial for performance, but probably at a cost to reliability (i.e. number of bugs affecting test discovery).

What restrictions are you facing with PTVS right now? We may need to address these in some other way.
Mar 20 at 6:45 PM
Hi Zooba!

Thank you for your reply.

Actually, I asked about this to extend the scope of testing, e.g. to include files, which for some reason (third party or legacy code) doesn't follow the standard unittest naming conventions.

But restricting could also be useful, e.g. to exclude long running tests from Visual Studio.

Regards,
Petr
Coordinator
Mar 20 at 7:03 PM
Edited Mar 20 at 7:04 PM
I see. The examples given in the unittest documentation are restrictions compared to what PTVS does - we already look at all files in the project regardless of name. We don't really want to maintain yet-another-list-of-paths that reference test files - there's already enough of a mess with environments/search paths/references/etc.

The best way to approach this in PTVS is to create another .pyproj file for your tests and have two projects in your solution (i.e. have them both open at once). If you add a search path from the test project back to the main project, all the tests should work just as well as if they were in the same project.

Another approach that could be used is to create a test in your project that runs the third-party test. Deriving from another test class should make all of the tests visible (provided they also appear in IntelliSense - we use the same code analysis to find tests). Something like:
class TestWrapper(unittest.TestCase, ThirdPartyTestClassWithTests):
    pass
should work, but you could also redirect to individual tests with def test_a(self): ThirdPartyTestClassWithTests.runtest_a(self) if that naming convention needs fixing.

Neither us nor the unittest options let you configure the method name prefix (and we currently don't detect runTest based tests, which is a separate issue), so I don't think anything would be gained here.

It may be interesting to try and handle the @skip decorators and hide them from the test list in VS. In general, we prefer to use the standard approaches to these things and keep them visible outside of VS; an annotation in the test file is preferable to a separate "playlist" that only works in VS. For example, if third-party code doesn't follow the unittest naming conventions, people on your team who aren't using VS will face the same issue. A wrapper class like I mentioned above will also help them, while a VS-specific setting won't.
Mar 21 at 12:53 PM
OK, fair enough. Thank you for your detailed answer.