2
Vote

Tests not discovered after upgrade to 2.1 beta

description

After upgrading from PTVS 2.0 to PTVS 2.1 I miss most of the unit tests. At the first look it looks like tests derived directly from unittest.TestCase are discovered, and tests derived from django.test.TestCase are missing (but Selenium tests derived from django.test.LiveServerTestCase are discovered).

All test files are named as test<filename>.py, all test classes are named as Test<class_name>, all test methods are named as test<method-name>().

Also when I add new test case from template, that means derived from unittest.TestCase, it mostly did not appear in the test list (but ocassionaly do, I did not find the rule yet).

Diagnostic info can be found in the attachment. I`m a bit surprised, that no one of my files is mentioned there, only Django and Python files.

Any idea how to fix or at least investigate it?

Thanks.

file attachments

comments

Zooba wrote Jul 9 at 11:50 AM

There may be some other issue that's causing test discovery to fail. Do you see any messages in the Tests view of the Output window? It may also be worth looking in your Application event log in case it's crashing quietly. It's also possible that we've failed to refresh the completion DB properly and no longer realise that Django's test class is a subclass of the standard one. If you don't get completions for the assert members, then that's what has happened.

Haven't looked at your diagnostic info yet (on my phone currently) but I wouldn't expect to see too many details about your open projects - it's mostly for the installed interpreters.

brano wrote Jul 15 at 1:22 PM

Good tips. Thanks.
There is a funny think in Tests Output window - it reports 97 tests discovered, but in the Text Explorer there are still only 71. When running the Django test framework, it finds 306 tests. There are few lines like "A test with the same name 'libs\parsley\tests\tests.py::CharFieldTest::test_basic' already exists. This test is not added to the test window.", if I count them, this are the missing tests (97-71).
There is one more line in Tests Output window Im not sure. It reports "File has an invalid module name: E:\Projects\T3C\Source\code\server\web\webSite\mainconf.template.py". We use this file only as a template, it is not really used to run or include as a Python file. After removing the file, the line is gone, but no more tests are discovered.
I
m not sure what to check in the Event viewer, I found few entries 'Microsoft-Windows-WebDeploy/Debug', but with no event in it.

Zooba wrote Jul 15 at 4:17 PM

Are you sure you haven't got tests with identical names? Python allows it, obviously, and other test frameworks may think that you have separate tests even though it's running the same one multiple times. If you run from the command-line, how many tests run?

You can ignore the "invalid module name" warning - it's because of the extra period in the name that prevents it from being imported, which (IIRC) we need to run it with our test harness.

brano wrote Jul 16 at 7:44 AM

Well, what is meant by test name? Most of my test cases are organized as follows. The test methods are in separate base classes (derived from object or django.tests.TestCase). The TestCase class contains more test methods, which share setup & tearDown and test simillar functionality, some of the test methods are directly in derived class, some are defined in parent class or trait (derived from object). Regarding the question of name that means, that if we consider test name to be only method name, then yes, I have some test names used multiple times. But - also those methods with unique names are not discovered. If we consider test name to be somethink larger - like test method name plus final derived class name, as I would expect, then I have test names unique.
Another strange think is, that the tests mentioned in the Ouput window are discovered. The missing tests are not mentioned in the Ouput window at all. Few other tests, all those derived directly from unittest.TestCase are found.

I`m not sure, what you mean by running in the command line, but I tried following:
  1. run python manage.py test (Django test runned) - 306 tests
  2. run python -m unittest (PyUnit test runner) - 306 tests
  3. run python -m unittest discover (PyUnit test discovery) - 306 tests
Command line directory was set to project root, the one with manage.py for Django projects.

Anyway, in PyVS 2.0 I did not notice any problems. It starts after upgrade to 2.1Beta 2. Were there any significant changes in the way, how tests are discovered? I`m a bit catched in the idea, that the tests derived from Django test case are missing, but I can not believe, that no one except me would report, if that will happend. I guess, there is something special in this case.

Should the following code find two tests in the class MyTestCase?

class TestBase(django.tests.TestCase):
def testA(self):
  pass
class TestTrait(object):
def testB(self):
pass
class MyTestCase(TestBase, TestTrait):
pass

Zooba wrote Jul 16 at 12:58 PM

We should certainly find testA in that example, but I'm not sure about how we'll handle the mixin. I'm pretty sure we handle inheritance correctly based on the MRO, but it's possible that something has changed.

The biggest code change since 2.0 is a significant refactoring so we can share code with Node.js Tools for VS. I didn't do that work, so I'll have to take some time to check this out. I should get to it today.

brano wrote Jul 16 at 1:42 PM

I created a simple new solution to check this, and tests defined in mixins are not discovered, the ones in derived classes are discovered. Attaching the source code and screenshot to save some work for reproducing this issue.
from django.test import TestCase

class TestBase(TestCase):
    def testA(self):
        pass

class TestMixin(object):
    def TestB(self):
        pass

class TestDerived(TestBase, TestMixin):
    def testC(self):
        pass
finds tests:
TestBase::testA
TestDerived::testA
TestDerived::testC
When I try to use multiple inheritance like in example bellow, testB is still not discovered, same if I change the order of classes.
from django.test import TestCase

class TestBase(TestCase):
    def testA(self):
        pass

class TestMixin(TestCase):
    def TestB(self):
        pass

class TestDerived(TestMixin, TestBase):
    def testC(self):
        pass
Anyway, I still think, there is something more in my case, I see that also TestDerived::testC is not discovered for some reason in few cases here. I`ll try to simulate in sample project and come back later. Thanks for help to understand since now.

brano wrote Jul 16 at 2:17 PM

Ok, I have the other issue. It is related to inheritance in different files. The simplest case to reproduce is:

file: testBase.py
from django.test import TestCase

class TestBase(TestCase):
    def testBase(self):
        pass
file: testDerived.py:
from app.tests.testBase import TestBase

class TestDerived(TestBase):
    def testDerived(self):
        pass
The tests discovered:
TestBase::testBase

Tests not discovered:
TestDerived::testBase
TestDerived::testDerived

I`m attaching the sample project in the attachment to save you some work while reproducing.

brano wrote Jul 16 at 2:19 PM

It looks, that this two issues can explain all the not discovered tests I have in my project.

Zooba wrote Jul 16 at 3:20 PM

Okay, so the problem with the mixin is that we use a case sensitive comparison for the "test" part of the method name. If I rename "TestB" to "testB" then it is discovered correctly in both cases. FWIW, unittest.main() does not find "TestB" either, and that's what we've modelled our behaviour on.

The second case also worked fine for me, though admittedly I'm using a slightly later version than 2.1b2. I tested with Django 1.6.5 in a Python 2.7 32-bit virtual environment and a Python 3.4 32-bit virtual environment with the completion databases all up to date and the project fully saved (Ctrl+Shift+S). Anything sound different from your setup?

brano wrote Jul 17 at 8:08 AM

Ouch, typo. :) Shame on me. Sorry. Yes, the first sample - mixins - works for me after correcting the case typo. The second still - different files - do not. I also tried to verify, if this helps in the my regular project - I moved the tests from separate files to the one, where the base classes are defined, and the tests started to appear right after saving the file. So that could be the reason, why I don`t see tests here.

Regarding the configuration - I run here Win 7, 64-bit, VS 2012, Python 3.3.3., Django 1.6.1. I use PyVS 2.1b2, I tried to upgrade, and the installer told me, that I can change or repair. I also use 'Save all', or run the project, or close VS and restart to force start of test finder.

brano wrote Jul 17 at 8:46 AM

I don`t know if this could be related, but in the testDerived.py, the import of TestBase class is marked green by Intelisense complaining: "Unable to resolve 'app.tests.TestBase'. Intellisense may be missing for this module.".

brano wrote Jul 17 at 11:51 AM

Btw. the Python is 32-bit version.

Zooba wrote Jul 17 at 1:09 PM

The green squiggle is almost certainly related, since it means we won't have the full inheritance hierarchy for your tests. What does your file structure look like (specifically the file with the import and the one you're expecting to import)? If you add a Search Path to the folder containing app does it work?

brano wrote Jul 17 at 2:16 PM

Hm. I see the green squiggle also in the sample project uploaded before - DjangoTestDiscovery2.zip, in the file app/tests/testDerived.py . The TestBase class is unable to resolve. This project is generated from the standard Django project template, so i guess there is nothing special, or at least I don`t see anything right now. In the working project right now, colleges of my renamed the 'app' folder to 'webAdmin', but I guess that should not play the role. Screenshot of Solution Explorer attached.
I tried to add the directory to the folder containing app (in my case 'E:\Projects_PlayGround\DjangoTestDiscovery\DjangoTestDiscovery\'), but also to the 'app\' folder, and 'app\tests\' folder to the Search Paths in project options. After saving the file, test finder launches, which is indicated by the moving green progressbar in the Test Explorer, but no more tests are discovered and squiggle remains.

brano wrote Jul 17 at 2:23 PM

In the real project, it looks like all imports from my projects are unresolved. The structure of the project is shown in the attached screenshot, or I can send the project itself with some removed file content if it will help.

The struture is like:
  • web
    • webSite
    • webAdmin
    • model
    • templates
    • views
    • tests
      • lots of test, mostly not resolved, all derived from some custom classes in webAdmin module
      • unitTests
        • few other tests, all resolved correctly, but derived directly from unittest.TestCase,

brano wrote Jul 17 at 2:24 PM

Ouch, sorry, the indentation gets corrupted. See the attached image instead.

Zooba wrote Jul 17 at 4:06 PM

I think there are two separate issues in the two images.

In the sample structure, the tests folder is missing __init__.py and that's why the testBase class can't be imported.

In the actual structure, I can't see whether all the __init__.py files are there, but I suspect they are. It looks like we don't think webAdmin is a top-level package (which is probably true according to normal Python rules, but Django does some tricks to break those).

In the testInstitutionDetailsBase.py file, can you import tests or import web (or the name of the folder containing the project - typically it's the same, but not always) without getting the warning? If so, it may mean that there's an extra __init__.py in the root of your project or one missing from webAdmin.

brano wrote Jul 18 at 8:56 AM

Wow, cool. Yes, you are right, adding the init.py helped to the sample case. We also had init.py in the web root folder, so webAdmin was not the root folder. Removing it helped. Somehow pyUnit test discovery tool was able to handle it, maybe it does not check, if the test class is derived from some unit test base class. The number of tests discovered by pyUnit runner and pyVS test discovery tool still differs, I need some time to find which tests are missing.
Thanks for help, wish you a nice weekend.

Zooba wrote Jul 18 at 4:42 PM

pyUnit will be checking the base class, but it will be doing it by actually running your code. Because Django's trick to make imports work despite the extra __init__.py file only applies at runtime, the class will be resolved correctly. Our code tries to analyze it without actually running the code, which is why we don't work well when sys.path is modified by the program.

That said, we keep seeing Django projects with __init__.py files at the top level, so we really need to find a sensible way to deal with these - presumably they are added for a reason, but we're not quite sure what that reason is and whether telling people to delete it is a good idea.