1

Resolved

multiprocessing in a unittest doesn't work

description

Running following unittest from within Visual Studio (or any other code using multiprocessing) gives errors (at least in Python 2.7):
    def test_pool(self):
        from multiprocessing import Pool

        pool = Pool()
        
        asyncresults = [pool.apply_async(self.square, t) for t in range(10)]
    
        results = [p for r in asyncresults for p in r.get()]
        expected = [t*t for t in range(10)]
        pool.close()

        self.assertListEqual(expected, results)
Unit test result:

Test Name: test_pool
Test Outcome: Failed
Result StandardError:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
File "C:\Python27\lib\multiprocessing\forking.py", line 495, in prepare
'__parents_main__', file, path_name, etc
File "C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 11.0\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON TOOLS FOR VISUAL STUDIO\2.0\visualstudio_py_testlauncher.py", line 28, in <module>
parser = OptionParser(prog = 'visualstudio_py_testlauncher', usage = 'Usage: %prog [<option>] <test names>... ', version = '%prog ' + PTVS_VER)
NameError: name 'PTVS_VER' is not defined
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
File "C:\Python27\lib\multiprocessing\forking.py", line 495, in prepare
'__parents_main__', file, path_name, etc
File "C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 11.0\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON TOOLS FOR VISUAL STUDIO\2.0\visualstudio_py_testlauncher.py", line 28, in <module>
parser = OptionParser(prog = 'visualstudio_py_testlauncher', usage = 'Usage: %prog [<option>] <test names>... ', version = '%prog ' + PTVS_VER)
NameError: name 'PTVS_VER' is not defined
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python27\lib\multiprocessing\forking.py", line 380, in main
prepare(preparation_data)
File "C:\Python27\lib\multiprocessing\forking.py", line 495, in prepare
'__parents_main__', file, path_name, etc
File "C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 11.0\COMMON7\IDE\EXTENSIONS\MICROSOFT\PYTHON TOOLS FOR VISUAL STUDIO\2.0\visualstudio_py_testlauncher.py", line 28, in <module>
parser = OptionParser(prog = 'visualstudio_py_testlauncher', usage = 'Usage: %prog [<option>] <test names>... ', version = '%prog ' + PTVS_VER)
NameError: name 'PTVS_VER' is not defined


Version Info
Projects:
Project: dataprep.pyproj
    Kind: Python
    InterpreterId: 9a7a9026-48c1-4688-9d5d-e5699d47d074
    InterpreterVersion: 2.7
    WorkingDirectory: .
    PublishUrl: <undefined>
    SearchPath: 
    CommandLineArguments: <undefined>
    InterpreterPath: <undefined>

    Interpreter: Python 64-bit 2.7
        Id: 9a7a9026-48c1-4688-9d5d-e5699d47d074
        Version: 2.7
Environments:
Microsoft.PythonTools.Interpreter.ConfigurablePythonInterpreterFactoryProvider
Microsoft.PythonTools.Interpreter.CPythonInterpreterFactoryProvider
    Id: 9a7a9026-48c1-4688-9d5d-e5699d47d074
    Factory: Python 64-bit 2.7
    Version: 2.7
    Arch: Amd64
    Prefix Path: C:\Python27
    Path: C:\Python27\python.exe
    Windows Path: C:\Python27\pythonw.exe
    Lib Path: C:\Python27\lib
    Path Env: PYTHONPATH

    Id: 9a7a9026-48c1-4688-9d5d-e5699d47d074
    Factory: Python 64-bit 3.3
    Version: 3.3
    Arch: Amd64
    Prefix Path: C:\Python33
    Path: C:\Python33\python.exe
    Windows Path: C:\Python33\pythonw.exe
    Lib Path: C:\Python33\lib
    Path Env: PYTHONPATH

comments

Zooba wrote Jan 14 at 3:46 PM

I assume we need to change the visualstudio_py_testlauncher.py script to be safely importable (that is, move most of the code into an if __name__ == '__main__' block). I'm not sure whether multiprocessing will be able to handle things from there, but it's certainly a starting point.

Samuel, if you want to try this then you can modify visualstudio_py_testlauncher.py yourself (it's in our install directory). We'll happily take a contribution if all the fixes are easily done in that file. Otherwise, we may not get to look at this ourselves for our upcoming release.

samuelbosch wrote Jan 15 at 7:49 AM

Putting the entire content of visualstudio_py_testlauncher.py into an if __name__ == '__main__' block fixes my issue. With the following code you can reproduce the issue and verify that it is fixed after changing visualstudio_py_testlauncher.py.
import unittest

def square(x):
    return x*x

class Test_multiprocessing(unittest.TestCase):
    def test_pool(self):
        from multiprocessing import Pool

        pool = Pool()
        
        asyncresults = [pool.apply_async(square, [t]) for t in range(10)]
    
        results = [r.get() for r in asyncresults]
        expected = [t*t for t in range(10)]
        pool.close()

        self.assertListEqual(expected, results)

if __name__ == '__main__':
    unittest.main()