Published on 18 April 2023.
Ron is working on an Asteroids game in Python and also writing about it. I’m interested in his workflow, so I follow along.
He recently published the code on Github.
I thought I would have a look.
I clone the repo and see a bunch of Python files and an .idea
folder.
I’ve never been a fan of IDEs. Perhaps I should learn one properly. In any case I find it useful to be able to run commands from the command line as well.
First, I want to see if I can get this game running:
$ python game.py
AttributeError: 'pygame.math.Vector2' object has no attribute 'copy'
I suspect I’m using a different version of pygame that lacks the copy method on vectors.
I try to run the test to see if I get the same failure there. How to run the tests? I think Ron mentioned that he uses pytets. I try:
$ pytest
===================================================================== test session starts =====================================================================
platform linux -- Python 3.9.10, pytest-6.2.2, py-1.11.0, pluggy-0.13.1
rootdir: /home/rick/downloads/python-asteroids-1
collected 3 items / 3 errors
...
I see the same error about the copy method of vector and some more in the same style.
I read about the copy method in the pygame manual and conclude that it was added in a later version.
I think I’ve installed pygame via Fedora’s package manager. That doesn’t have a more recent version of pygame.
I try to install it using pip instead:
$ pip install --user pygame
Requirement already satisfied: pygame in /usr/lib64/python3.9/site-packages (2.0.3)
I add --user
because I don’t want to install anything globally using pip. I suppose I should create a virtual environment, but I haven’t worked much with them. This will do.
It indeed tells me that I already have pygame installed. How do I upgrade? Ah, the --upgrade
flag:
$ pip install --user --upgrade pygame
Requirement already satisfied: pygame in /usr/lib64/python3.9/site-packages (2.0.3)
Collecting pygame
Downloading pygame-2.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.8 MB)
|████████████████████████████████| 13.8 MB 692 kB/s
Installing collected packages: pygame
Successfully installed pygame-2.3.0
Sometimes I hesitate to install Python packages via pip. Especially when they are not pure Python packages (like pygame which depends on SDL and C libraries). Mostly because it hasn’t worked so well for me in the past. Maybe things are better now. And maybe it depends on the library. Let’s see how this works now.
I try running the game again:
$ python game.py
pygame 2.3.0 (SDL 2.24.2, Python 3.9.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
Success! Or, I don’t get any errors at least. But it exits right away. Am I running the wrong file?
Ah, there is a main.py
. Let’s try that.
It works!
I’m quite familiar with both Python and pygame, so it was not that difficult for me to get started. But I think we can improve.
One idea that I got from James Shore’s writing about a zero friction development is that you should have scripts for doing common tasks like running your tests.
Let’s see if Ron likes that as well. I add one script to test
#!/usr/bin/env bash
set -e
pytest
and one to run the application
#!/usr/bin/env bash
exec python main.py
Should the way to run tests or the application change, only those files need to be changed, and the usage of the developer stays the same.
]]>Published on 2019-09-28.
When working on porting Timeline to Python 3, I ran into a problem where a test caused a segfault. I managed to create a small example that reproduces the failure. I describe the example below and show how I solved the test failure.
The example consists of a test that stores an instance of a custom wx event in a mock object:
from unittest.mock import Mock import unittest import wx import wx.lib.newevent CustomEvent, EVT_CUSTOM = wx.lib.newevent.NewEvent() class WxTest(unittest.TestCase): def test_wx(self): mock = Mock() mock.PostEvent(CustomEvent()) if __name__ == "__main__": unittest.main()
When I run this example, I get the following error:
$ python3 test_wx.py . ---------------------------------------------------------------------- Ran 1 test in 0.001s OK Segmentation fault (core dumped)
If I instead run it through gdb, I can see the C stacktrace where the error happens:
$ gdb python3 GNU gdb (GDB) Fedora 8.2.91.20190401-23.fc30 ... (gdb) run test_wx.py ... . ---------------------------------------------------------------------- Ran 1 test in 0.001s OK Program received signal SIGSEGV, Segmentation fault. dict_dealloc (mp=0x7fffe712f9d8) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/dictobject.c:1901 1901 /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/dictobject.c: No such file or directory. Missing separate debuginfos, use: dnf debuginfo-install fontconfig-2.13.1-6.fc30.x86_64 libXcursor-1.1.15-5.fc30.x86_64 libgcrypt-1.8.4-3.fc30.x86_64 libxkbcommon-0.8.3-1.fc30.x86_64 lz4-libs-1.8.3-2.fc30.x86_64 python3-sip-4.19.17-1.fc30.x86_64 (gdb) bt #0 dict_dealloc (mp=0x7fffe712f9d8) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/dictobject.c:1901 #1 0x00007fffea02e1cc in wxPyEvtDict::~wxPyEvtDict (this=0x5555556f3ee8, __in_chrg=<optimized out>) at ../../../../src/pyevent.h:48 #2 wxPyEvent::~wxPyEvent (this=0x5555556f3e90, __in_chrg=<optimized out>) at ../../../../src/pyevent.h:96 #3 sipwxPyEvent::~sipwxPyEvent (this=0x5555556f3e90, __in_chrg=<optimized out>) at ../../../../sip/cpp/sip_corewxPyEvent.cpp:56 #4 0x00007fffea02e24d in sipwxPyEvent::~sipwxPyEvent (this=0x5555556f3e90, __in_chrg=<optimized out>) at ../../../../sip/cpp/sip_corewxPyEvent.cpp:56 #5 0x00007fffea02dfd2 in release_wxPyEvent (sipCppV=0x5555556f3e90, sipState=<optimized out>) at ../../../../sip/cpp/sip_corewxPyEvent.cpp:261 #6 0x00007fffe72ff4ce in ?? () from /usr/lib64/python3.7/site-packages/sip.so #7 0x00007fffe72ff51d in ?? () from /usr/lib64/python3.7/site-packages/sip.so #8 0x00007ffff7c14869 in subtype_dealloc (self=<_Event at remote 0x7fffea4ddd38>) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/typeobject.c:1256 #9 0x00007ffff7b8892b in tupledealloc (op=0x7fffe712cda0) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/tupleobject.c:246 #10 0x00007ffff7b8892b in tupledealloc (op=0x7fffea4d7620) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/tupleobject.c:246 #11 0x00007ffff7c14869 in subtype_dealloc (self=<_Call at remote 0x7fffea4d7620>) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/typeobject.c:1256 #12 0x00007ffff7b882ae in list_dealloc (op=0x7fffe70ba228) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/listobject.c:324 #13 0x00007ffff7c14869 in subtype_dealloc (self=<_CallList at remote 0x7fffe70ba228>) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/typeobject.c:1256 #14 0x00007ffff7b8c813 in free_keys_object (keys=0x555555855080) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/gcmodule.c:776 #15 dict_dealloc (mp=0x7fffe712f630) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/dictobject.c:1913 #16 subtype_clear (self=<optimized out>) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Objects/typeobject.c:1101 #17 delete_garbage (old=<optimized out>, collectable=<optimized out>) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/gcmodule.c:769 #18 collect (generation=2, n_collected=0x7fffffffd230, n_uncollectable=0x7fffffffd228, nofail=0) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/gcmodule.c:924 #19 0x00007ffff7c4ac4e in collect_with_callback (generation=generation@entry=2) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/gcmodule.c:1036 #20 0x00007ffff7ca7331 in PyGC_Collect () at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/gcmodule.c:1581 #21 0x00007ffff7caaf03 in Py_FinalizeEx () at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Python/pylifecycle.c:1185 #22 0x00007ffff7cab048 in Py_Exit (sts=sts@entry=0) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Python/pylifecycle.c:2278 #23 0x00007ffff7cab0ff in handle_system_exit () at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Python/pythonrun.c:636 #24 0x00007ffff7cab1e6 in PyErr_PrintEx (set_sys_last_vars=1) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Python/pythonrun.c:646 #25 0x00007ffff7cab651 in PyRun_SimpleFileExFlags (fp=<optimized out>, filename=<optimized out>, closeit=<optimized out>, flags=0x7fffffffd410) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Python/pythonrun.c:435 #26 0x00007ffff7cad864 in pymain_run_file (p_cf=0x7fffffffd410, filename=<optimized out>, fp=0x5555555a20d0) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/main.c:427 #27 pymain_run_filename (cf=0x7fffffffd410, pymain=0x7fffffffd520) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/main.c:1627 #28 pymain_run_python (pymain=0x7fffffffd520) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/main.c:2877 #29 pymain_main (pymain=0x7fffffffd520) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/main.c:3038 #30 0x00007ffff7cadc0c in _Py_UnixMain (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/python3-3.7.3-1.fc30.x86_64/Modules/main.c:3073 #31 0x00007ffff7e12f33 in __libc_start_main (main=0x555555555050 <main>, argc=2, argv=0x7fffffffd678, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd668) at ../csu/libc-start.c:308 #32 0x000055555555508e in _start ()
Somewhere in the middle, there is a call to PyGC_Collect
followed, a bit higher up, by a call to release_wxPyEvent
. This indicates that the error occurs during garbage collection of the custom wx event.
The machine I run the example on is running Python 3.7.3, and wxPython 4.0.4:
$ python3 Python 3.7.3 (default, Mar 27 2019, 13:36:35) [GCC 9.0.1 20190227 (Red Hat 9.0.1-0.8)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import wx >>> wx.version() '4.0.4 gtk3 (phoenix) wxWidgets 3.0.4'
To solve this problem, I replaced the mock function PostEvent
with one that simply discards its input like this:
mock.PostEvent = lambda x: None
This way there is no custom wx event to garbage collect. In Timeline's case, it was not important to store the event in the mock object anyway.
If you have any idea why this example causes a segfault, I would be interested to know. It feels like an error in the wxPython wrapper.
]]>Published on 2019-08-31.
When working on porting Timeline to Python 3, I ran into a problem where a doctest failed under certain circumstances. I managed to create a small example that reproduces the failure. I describe the example below and show how I solved the test failure.
The example consists of a test runner and two test cases. The test runner is a slimmed down version of the one used in Timeline:
import doctest import sys import unittest def load_test_cases_from_module_name(suite, module_name): __import__(module_name) module = sys.modules[module_name] module_suite = unittest.defaultTestLoader.loadTestsFromModule(module) suite.addTest(module_suite) def load_doc_tests_from_module_name(suite, module_name): __import__(module_name) module = sys.modules[module_name] try: module_suite = doctest.DocTestSuite(module) except ValueError: # No tests found pass else: suite.addTest(module_suite) if __name__ == "__main__": suite = unittest.TestSuite() load_test_cases_from_module_name(suite, "test_wx") load_doc_tests_from_module_name(suite, "test_doc") print(unittest.TextTestRunner().run(suite))
It creates a test suite with test cases from two modules: one with a unit test and one with a doctests. It then runs the tests.
The first test is a unit test that needs an instance of wx.App
:
import contextlib import unittest import wx class WxTest(unittest.TestCase): def test_wx(self): with self.wxapp() as app: # Test something that requires a wx.App pass @contextlib.contextmanager def wxapp(self): app = wx.App() try: yield app finally: app.Destroy()
This example doesn't test anything, but is enough to reproduce the failure.
The second test is a doctest that asserts that a function prints a string:
""" >>> print_fun_stuff() This is fun! """ def print_fun_stuff(): print("This is fun!")
When I run this example, I get the failure:
$ python3 testrunner.py .This is fun! F ====================================================================== FAIL: test_doc () Doctest: test_doc ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib64/python3.7/doctest.py", line 2196, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for test_doc File "test_doc.py", line 0, in test_doc ---------------------------------------------------------------------- File "test_doc.py", line 2, in test_doc Failed example: print_fun_stuff() Expected: This is fun! Got nothing ---------------------------------------------------------------------- Ran 2 tests in 0.074s FAILED (failures=1) <unittest.runner.TextTestResult run=2 errors=0 failures=1>
What appears to happen is that the expected string in the doctest is written to the console (or perhaps stderr) instead of being captured by doctest. When I run the doctest in isolation, it passes, so there is nothing wrong with the test itself. It is the sequence of these two tests that causes the problem.
My guess is that something in the wx test interferes with the doctest. Perhaps instantiating a wx.App
has some effects on streams and redirection. But shouldn't the app.Destroy()
call reset any such effects? It would seem reasonable. But what if the wx.App
is not completely destroyed when the doctest is run? To test this, I modify the example to force a garbage collection after the app.Destroy()
call like this:
import gc; gc.collect()
This gets rid of the failure and the tests pass consistently. This is also the solution that I adopted for Timeline.
The machine I run the example on is running Fedora 30, Python 3.7.3, and wxPython 4.0.4:
$ uname -a Linux localhost.localdomain 5.0.9-301.fc30.x86_64 #1 SMP Tue Apr 23 23:57:35 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux $ python3 Python 3.7.3 (default, Mar 27 2019, 13:36:35) [GCC 9.0.1 20190227 (Red Hat 9.0.1-0.8)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import wx >>> wx.version() '4.0.4 gtk3 (phoenix) wxWidgets 3.0.4'
But the example doesn't always fail. On the Fedora 30 machine, it fails most of the time, but sometimes it succeeds. When I run the example on a machine that is running Fedora 26, Python 3.6.5, and wxPython 4.0.1, it always succeeds:
$ uname -a Linux x220 4.16.11-100.fc26.x86_64 #1 SMP Tue May 22 20:02:12 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux rick@x220 | ~/rickardlindberg.me/writing/draft-timeline-doctest-wxpython $ python3 Python 3.6.5 (default, Apr 4 2018, 15:09:05) [GCC 7.3.1 20180130 (Red Hat 7.3.1-2)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import wx >>> wx.version() '4.0.1 gtk3 (phoenix)'
Also, if I change the test so that it doesn't use a context manager, it always succeeds:
def test_wx(self): app = wx.App() try: # Test something that requires a wx.App pass finally: app.Destroy()
Perhaps the context manager has some effect on when objects are garbage collected.
If you have any idea why this example sometimes fails, I would be interested to know. It seems illogical that a forced garbage collection should be needed to get a correct program.
]]>Published on 11 March 2017.
Today I found a bug in a piece of Python code that I had written. The buggy code was the result of not taking into consideration how generators in Python work. It looked like this:
def main():
try:
"Processing items")
logging.info(= get_the_items()
items except:
"Could not get items")
logging.exception(else:
for item in items:
process_item(item)
Can you spot the error? What could possible go wrong with this code?
When I was debugging it, it was printing the info message, but did not log an exception. Execution continued in the else-clause, but suddenly an exception was raised when looping over the items (outside of process_item
). How can that happen? The except-clause should catch all exceptions. And just iterating over a collection should not raise an exception.
The answer is that get_the_items
returned a generator. It looked like this:
def get_the_items():
for item in read_items_from_disk():
if item.is_good():
yield item
The exception actually came from read_items_from_disk
, but since this code creates a generator, it is not executed until the collection is accessed (which happened in the else-clause). So the exception was actually raised when starting looping over items.
To ensure that an exception from the generator is caught, the main
function could be written like this:
def main():
try:
"Processing items")
logging.info(for item in get_the_items():
process_item(item)except:
"Could not get items") logging.exception(
I don’t like this version because it also catches exceptions from process_item
. I like the try-except-else syntax because it allows narrower exception regions in a nice looking way.
I made get_the_items
return a generator mainly because I thought it read better. The alternative I came up with looked like this:
def get_the_items():
= []
items for item in read_items_from_disk():
if item.is_good():
items.append(item)return items
I didn’t like the temporary items
variable. And it is two lines longer than the generator version.
Another way to write get_the_items
, avoiding the temporary variable, is to use list comprehensions. It would look like this:
def get_the_items():
return [
itemfor item in read_items_from_disk()
if item.is_good()
]
I find this code reads as good as the generator version (even though it is two lines longer). This is what I ended up using. But get_the_items
was a bit more complicated than in the example, so I had to divide it into two list comprehensions.
One argument for using generators is that they consume less memory. The whole collection of items do not have to fit in memory at once, only the one currently being processed.
In my case I had already read all items into memory in a previous step. The get_the_items
function was mainly used to transform and filter the items. So I would not have used much less memory by using generators. Also, my collections were small, so having them in memory was not a problem.
Generators have some nice properties that can be useful. However, after being bitten by them I now think they should only be used if those properties are absolutely needed. I will favor list comprehensions over generators if I can afford to keep the whole collection in memory.
]]>Published on 2015-06-27.
What is the precision of datetime in Python? The documentation says
Return the current local date and time. If optional argument tz is None or not specified, this is like today(), but, if possible, supplies more precision than can be gotten from going through a time.time() timestamp (for example, this may be possible on platforms supplying the C gettimeofday() function).
It goes on further to say
Note that even though the time is always returned as a floating point number, not all systems provide time with a better precision than 1 second.
So, the answer is that it depends.
Let's try to figure out what it looks like on Windows using Python 3.4. For reference:
import sys
print(sys.version)
Let's create a series of datetime objects that we can analyze to find out how far apart they are:
import datetime
dates = []
for _ in range(10000000):
dates.append(datetime.datetime.now())
Let's load them into Pandas so we can analyze them:
import pandas as pd
date_series = pd.Series(dates)
date_series.head()
date_series.describe()
Let's remove all duplicates:
uniq_date_series = date_series.drop_duplicates()
uniq_date_series.describe()
Now let's figure out the delta between all uniqe dates:
deltas = uniq_date_series - uniq_date_series.shift(1)
deltas.describe()
And the smallest delta is
deltas.min()
This means that the smallest increment between two consecutive dates is 1ms. So we can not use the datetime on Windows to measure events that occur more frequently than 1ms.
And that is in the best case. This number will vary depending on how many other processes are running and what the Python code does in between two measurements.
Published on 3 November 2014.
In Python we can put an expression in an if statement that is not a boolean. For example:
a_list = [1, 2, 3]
if a_list:
# do something
The expression will evaluate to either true or false. Some examples of expressions that will evaluate to false:
[]
(empty list)""
(empty string)0
(the number 0)None
(the null value)Some examples of expressions that will evaluate to true:
[1, 2]
(non-empty list)"hello"
(non-empty string)88
(the number 88)So if we are only interested in knowing if a value is truthy, we do not need to make an explicit comparison in the if statement. The above example with an explicit comparison would look like this:
if a_list != []:
# do something
We can argue that the first example read better because there is less cruft in the expression, but there is one real danger in being implicit. Consider a function that returns either a number or None if no number could be returned. We want to run some code only if we get a number back:
number = give_me_a_number()
if number:
# do something
This works fine for most numbers:
Except for:
The number 0 is a number, so we would like to do something with it. But on the other hand, the number 0 evaluates to false. So with an implicit check, it is not considered truthy, and we will not enter the if block. What we should have done instead was this:
if number is not None:
# do something
I have made this mistake more than once, and I’m starting to think that explicit if statements should always be used except in special cases.
]]>