DRAFT: Doctest fails in Python 3 with wxPython

Published on 2019-08-14.

This is a work in progress that will change. Like to see it finished? Let me know by sending me an email.

We are working on porting Timeline to Python 3 and ran into a problem where a doctest failed under certain circumstances. I managed to create a small example that reproduced the error. I describe the example below and show how we solved the problem.

The example consists of a test runner and two test cases. The test runner is a slimmed down version of the one we use in Timeline:

  1. testrunner.py
import doctest
import sys
import unittest

def load_test_cases_from_module_name(suite, module_name):
    __import__(module_name)
    module = sys.modules[module_name]
    module_suite = unittest.defaultTestLoader.loadTestsFromModule(module)
    suite.addTest(module_suite)

def load_doc_tests_from_module_name(suite, module_name):
    __import__(module_name)
    module = sys.modules[module_name]
    try:
        module_suite = doctest.DocTestSuite(module)
    except ValueError:
        # No tests found
        pass
    else:
        suite.addTest(module_suite)

if __name__ == "__main__":
    suite = unittest.TestSuite()
    load_test_cases_from_module_name(suite, "test_wx")
    load_doc_tests_from_module_name(suite, "test_doc")
    print(unittest.TextTestRunner().run(suite))

It creates a test suite with test cases from two modules: one with a unit test and one with a doctests. It then runs the tests in the suite.

The first test is a unit test that needs an instance of wx.App:

  1. test_wx.py
import contextlib
import unittest

import wx

class WxTest(unittest.TestCase):

    def test_wx(self):
        with self.wxapp() as app:
            # Test something that requires a wx.App
            pass

    @contextlib.contextmanager
    def wxapp(self):
        app = wx.App()
        try:
            yield app
        finally:
            app.Destroy()

The example doesn't test anything, but this is enough to reproduce the error.

The second test is a doctest that asserts that something is printed:

  1. test_doc.py
"""
>>> print_fun_stuff()
This is fun!
"""

def print_fun_stuff():
    print("This is fun!")

When I run this example, the result is unexpected:

$ python3 testrunner.py
.This is fun!
F
======================================================================
FAIL: test_doc ()
Doctest: test_doc
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.7/doctest.py", line 2196, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for test_doc
  File "/home/rick/rickardlindberg.me/writing/draft-timeline-py3-segfault/test_doc.py", line 0, in test_doc

----------------------------------------------------------------------
File "/home/rick/rickardlindberg.me/writing/draft-timeline-py3-segfault/test_doc.py", line 2, in test_doc
Failed example:
    print_fun_stuff()
Expected:
    This is fun!
Got nothing


----------------------------------------------------------------------
Ran 2 tests in 0.074s

FAILED (failures=1)
<unittest.runner.TextTestResult run=2 errors=0 failures=1>

What appears to happen is that the expected string is written to the console (or perhaps stderr) instead of being captured by doctest.

The machine I run the example on is running Fedora 31, Python 3.7.3, and wxPython 4.0.4:

$ uname -a
Linux localhost.localdomain 5.0.9-301.fc30.x86_64 #1 SMP Tue Apr 23 23:57:35
UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ python3
Python 3.7.3 (default, Mar 27 2019, 13:36:35)
[GCC 9.0.1 20190227 (Red Hat 9.0.1-0.8)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import wx
>>> wx.version()
'4.0.4 gtk3 (phoenix) wxWidgets 3.0.4'

But the example does not always fail. On this machine, it fails most of the time, but sometimes it succeeds. When I run the example on a different machine, it always succeeds. That machine is running Fedora 26, Python 3.6.5, and wxPython 4.0.1:

$ uname -a
Linux x220 4.16.11-100.fc26.x86_64 #1 SMP Tue May 22 20:02:12 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
rick@x220 | ~/rickardlindberg.me/writing/draft-timeline-doctest-wxpython
$ python3
Python 3.6.5 (default, Apr  4 2018, 15:09:05)
[GCC 7.3.1 20180130 (Red Hat 7.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import wx
>>> wx.version()
'4.0.1 gtk3 (phoenix)'

Also, if I change the test so that it doesn't use a context manager, it always succeeds:

def test_wx(self):
    app = wx.App()
    try:
        # Test something that requires a wx.App
        pass
    finally:
        app.Destroy()

Also, if I run the doctest in isolation it works. So something in the wx test must interfere with it. My guess is that instantiating a wx.App has some effects on streams and redirection. But shouldn't the app.Destroy() call reset any such effects? It would seem reasonable. But what if the wx.App is not completely destroyed when the doctest is run? I tried to force a garbage collection after the app.Destroy() call like this:

import gc; gc.collect()

This seems to get rid of the error.

Perhaps the context manager has some effect on when objects are garbage collected.

If you have an idea of why this example fails some times, I would be interested to know.


Site proudly generated by Hakyll.