Debugging Unit tests

Dec 20, 2011 at 10:06 AM

Firstly, thank you for this fantastic set of tools - everything  has worked pretty much straight out the box for me.

My problem: I have noticed that breakpoints in my unit tests (created with the standard unittest package) are not triggered when debugging.

I followed the advice on the unittest documentation site and ran my tests using TestSuite.debug() (instead of TestSuite.run() ), but alas, no breakpoints are triggered.

I am using :

Python Tools 1.1.41206.0

Python 2.6

Visual Studio 10.0.40219.1 SP1Rel

Editor
Dec 20, 2011 at 5:16 PM

Thanks for the kind words and the bug report. Just wanted to let you know that we'll be a little slower responding to this than usual, as most of the team is on vacation for the holidays.

Coordinator
Jan 3, 2012 at 8:47 PM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Jan 3, 2012 at 8:47 PM

I've created a bug for this and will look into it - http://pytools.codeplex.com/workitem/626

Coordinator
Jan 3, 2012 at 10:49 PM

Are you doing something along the lines of:

import unittest

from unittest import TestSuite

class MyTestCase(unittest.TestCase):
    def runTest(self):
        self.test_one()
        self.test_two()

    def test_one(self):
        raise Exception()

    def test_two(self):
        print('hi')

 TestSuite(tests = (MyTestCase(), )).debug()

 

I was able to run this and hit breakpoints, but I'm not quite certain if I've correctly reproduced the issue - I'm more used to seeing something more like:

import unittest

from unittest import TestSuite

class MyTestCase(unittest.TestCase):
    def test_one(self):
        raise Exception()

So far I haven't identified anything that should break the debugger when using TestSuite.debug though (I was concerned it might hoook sys.settrace, but it doesn't look like the standard TestSuite class does this).

    def test_two(self):
        print('hi')
 

if __name__ == '__main__':   
    unittest.main()

Coordinator
Jan 4, 2012 at 5:43 PM

I've also tried the version which is closer to what the unittest documentation has, but still slightly different.  I still wasn't able to repro the issue though.

import random
import unittest

class TestSequenceFunctions(unittest.TestCase):

    def setUp(self):
        self.seq = range(10)

    def test_shuffle(self):
        # make sure the shuffled sequence does not lose any elements
        random.shuffle(self.seq)
        self.seq.sort()
        self.assertEqual(self.seq, range(10))

        # should raise an exception for an immutable sequence
        self.assertRaises(TypeError, random.shuffle, (1,2,3))

    def test_choice(self):
        element = random.choice(self.seq)
        self.assertTrue(element in self.seq)

    def test_sample(self):
        with self.assertRaises(ValueError):
            random.sample(self.seq, 20)
        for element in random.sample(self.seq, 5):
            self.assertTrue(element in self.seq)

if __name__ == '__main__':
    suite = unittest.TestLoader().loadTestsFromTestCase(TestSequenceFunctions)
    suite.debug()


 

Jan 12, 2012 at 3:28 PM

@dinov: The code that you posted above works for me as well (breakpoints are hit), but when I move TestSequenceFunctions into a seperate module, they are not.

Coordinator
Jan 12, 2012 at 7:55 PM

Unfortunately I'm still having trouble reproing this, I've uploaded my repro project here http://pytools.codeplex.com/Download/AttachmentDownload.ashx?ProjectName=pytools&WorkItemId=626&FileAttachmentId=327544 

This splits the TestSequenceFunctions into it's own module - in fact I've tried a module, a package, and module in a package, as well as loading the the test cases using several different mechanisms (loadTestsFromName, testTestsFromModule, etc...). 

Does this project repro for you?  If not, could you upload the entire project which does repro?  Thanks!

Jan 13, 2012 at 2:17 PM

Thanks, but I am unable to reproduce the problem with your project either, meaning that there must be something in my project and/or environment which is causing this. Unfortunately I don't think my employer will be too thrilled with me uploading the project here - I will need to spend some time getting this to repro on a smaller project (may take a while...)

Coordinator
Jan 13, 2012 at 4:18 PM

Understandable - you might be able to diff the two project files and see if there's any obvious differences.  Otherwise I await your reply and thanks for being so helpful w/ getting to the bottom of this.

Coordinator
Jan 14, 2012 at 12:38 AM

Oh, another interesting thing might be where the projects are located.  If the project you're working on where it repro'd is significantly different than the path where you saved the downloaded project them maybe there's something about the full path that is messing up the debugger.  I tried a couple of different variations in that area but couldn't repro it that way. 

Jan 16, 2012 at 10:27 AM

OK I'm able to reproduce it now: If you add a subdirectory to your project, but without an __init__.py, and place your tests in there (also adding the subdirectory to the project search path), then the unit test will be found and executed, but without breakpoints being hit). If this fails for you, something else to experiment with is omitting the subdirectory from the project search path, and instead having a drive substitution such as O:\ --> project-source-directory, and having O:\test-subdir included in the global PYTHONPATH.

Coordinator
Jan 16, 2012 at 5:24 PM

Could this be related to .pyc files that remain on disk after renaming a module?  I just tried these steps and I was able to get the repro - but I went to change Program.py to make sure it was actually getting executed and my change didn't show up.  So I opened the directory and I still had a Program.pyc that was sitting around next to my PythonApplication#.py file.  That .pyc file is getting picked up before the .py file that lives in the search paths and therefore we're not hitting the breakpoints.

If this is the cause then maybe we should enable the -B command line option by default although that seems a little heavy handed.