Published on 7 February 2024.
This is what I’ve been up to in January 2024:
Published on 2 January 2024.
This is what I’ve been up to in December 2023:
This month I’ve been all consumed by Advent of Code. This year was the first year that I tried to solve all puzzles from beginning to end. At the time of this writing, I have completely solved day 1-20 and part 1 of day 21.
I solved all puzzles in my experimental programming language. One reason I did that was to see how good a fit the language was and also get feedback on how the language could be improved to be a better fit for a broader range of problems.
The Advent of Code experience has been as expected: fun, challenging, frustrating, and stressful. The most fun thing I learned was an algorithm for filling polygons. It was useful for two problems, but only fast enough for one.
The experience of developing my language at the same time has also been fun, but the pressure of completing the puzzles has given me less focus on the language itself. But I managed to add some features to the language that were direct needs that came up when solving the puzzles. One was a simple, built-in test framework. Another was nicer syntax. Most of the time though, it was not limitations in the language that made solving the puzzles hard. However, sometimes the solution would have been more straightforward in a better suited language.
Published on 12 December 2023.
This is what I’ve been up to in November 2023:
I continued work on my experimental programming language which is based on RLMeta.
I did more research on the paradigm of the Linda programming language and how it would fit together with RLMeta.
I started evolving the RLMeta implementation to support the paradigm of Linda. I realized that RLMeta could itself be implemented in this paradigm and started to evolve the code base in that direction.
I plan on solving Advent of Code puzzles using this language to see how problems from different domains can be expressed.
I really should document this language and my work on it better.
Published on 6 November 2023.
This is what I’ve been up to in October 2023:
Somehow videos from Computer History Museum showed up in my Youtube feed and I once again got sucked into the world of Alan Kay and Smalltalk. Here are (some of?) the videos that I watched:
I found the Byte Magazine Volume 06 Number 08 - Smalltalk online which covers many topics from Smalltalk. So interesting.
I played with some Smalltalks from the Smalltalk zoo, and also with Squeak.
I played with RLMeta and worked on a simpler base version.
Revisiting Alan Kay, I once again was inspired by his ideas. What triggered it this time was re-watching Joe Armstrong & Alan Kay - Joe Armstrong interviews Alan Kay. In it, he mentions the Linda programming language and how it relates to META II (which I’ve previously explored in RLMeta). I decided to try to explore those ideas further in a new project: https://github.com/rickardlindberg/linda-meta-oop. Perhaps more to come there soon.
I watched Is Software Engineering Real Engineering? • Hillel Wayne • YOW! 2023. What I remember from it now is that other engineering disciplines envy that software engineering has version control.
I had the realization that creating a DSL might be like science. You observe how a certain problem is solved or how a certain thing works. Then you try to describe that phenomena in a new language, the DSL. Then you make it run by writing a compiler. The new language allow people to solve problems that they might not have been able to do before. It allows them to think in a different way. The group of people who can create a DSL is most likely smaller than the group of people who can do useful work with one. Similar to how many people can use math today to solve useful problems, but might not have been able to invent math itself.
Published on 10 October 2023.
This is what I’ve been up to in September 2023:
I wrote another devlog about my Raspberry Pi game console.
I watched Continued Learning: The Beauty of Maintenance - Kent Beck - DDD Europe 2020. What stuck in my mind was the metaphor that software development is like a swan. You can watch the beauty as it moves across the water. That represents the behavior of the software system. The features. But to make that happen, lots of things is going on under the surface that you don’t see. The swan paddles its feet (?) to go forward, poop comes out, it is messy. That is the structural changes needed in software to make the features possible. Refactoring.
I started reading Modern Software Engineering by Dave Farley. So far, I’ve gotten a few useful ideas out of it.
When I started reading about software engineering, Alan Kay came to mind again. I revisited some of his talks, trying to find things about science and engineering in particular. I admire Alan’s work, and I also like the Agile concepts that Dave talks about. I’m interested in figuring out if they contradict, or if I can happily continue to admire both.
When reading about Alan Kay again, Bret Victor came to mind, and I watched his talk The Humane Representation of Thought. I feel like his and Kay’s work are worth revisiting from time to time. For inspiration for better ways of doing things. And for inspiration of things to try to do differently.
Published on 10 September 2023.
It is time to revisit the balloon shooter. I’m interested in building a “game console PC” so that my son can more easily play the balloon shooter and other games. Until now we have played all games on my laptop.
This will involve two main steps I think. The first is to get a Raspberry Pi and install all games on it. The second involves auto starting a custom application that can be used to select which game to play by using the gamepad. Ideally, you should not need to use a mouse or a keyboard. My plan for this custom application is to build it using the framework that we have in the balloon shooter.
Let’s get started.
At first, I’m not sure what hardware to get for this game console PC. I look around a bit, and then eventually settle on a Raspberry Pi starter kit.
I am bit concerned that it will not be powerful enough to play games. But it is relatively cheap, and if it can’t play all games, perhaps my son (or me) can have some fun with it in another way.
The starter kit comes with everything you need to get started. That’s also one reason that I went with it. I’m not that interested in selecting hardware. I’m more interested in quickly prototyping this game console PC. If it turns out the Pi is not powerful enough, but the game console PC concept is a hit, we can look for better hardware. However, if the game console PC is not a hit, we have not wasted that much time or money.
And look. Apparently Raspberry Pis need heat sinks and fans nowadays. When I last played with a Pi, many, many years ago, I don’t remember that being the case. Let’s hope that means that they are more powerful now.
I assemble the kit in about 15 minutes. Then I boot it up and install the operating system that comes preconfigured. I let it do its thing, and come back once it is installed.
I want to install SuperTux and the balloon shooter on the Pi.
It seems like the version of SuperTux is older than what I have on my laptop. And my laptop is old. Furthermore, Python 2 seems to be the default Python. I learn that when trying to install all requirements for the balloon shooter. I also have to install a newer version of Pygame and for that I need to install some SDL build dependencies. Perhaps getting a newer operating system would be nice.
Eventually, I get everything working:
The versions might be a little old. The performance might be so so. But we have something setup that we can experiment with.
Me and my son try to play SuperTux on the setup. It feels a little different. Part of it might be that it is slightly different version of the game. Part of it might be that the Pi has worse performance. We try to run the game at a lower resolution, and it seems to help a bit. We can probably try different things to get better performance, but this is absolutely fine for now. My son is still having fun playing.
To start SuperTux on the Pi you first have to start the Pi and then you have to select SuperTux from the menu with the mouse. The balloon shooter is even more complicated to start. First you need to open a terminal and then run a command.
I don’t think that is good enough for a game console PC. I want to be able to operate it using the gamepad only.
The first tiny step in that direction is to configure SuperTux as the startup application. If we can do that, then SuperTux can be started and played without using the keyboard or mouse.
Once we have that working, we can work on our own startup application that let us select the game, and then we can start that one instead.
I search the internet for how to configure a startup application for the Pi.
I find an article that says that you can put a file in the autostart directory. I try this:
$ cat /etc/xdg/autostart/game_console_start.desktop
[Desktop Entry]
Name=Game console start
Exec=supertux2
I restart the Pi, and SuperTux actually starts automatically and you can start playing it using the gamepad. Fantastic!
Let’s move on to our custom startup application. Here is the idea that I have for it:
while True:
subprocess.call(StartupApplication.create().run())
This code runs the startup application in a loop. Its run
method should return the command to run. (The game to play or shutdown command.)
I think we can test drive the StartupApplication
and then we can hook it up in the loop above.
Perhaps we should even test drive the loop.
We’ll see.
I start with this in a new file:
class StartupApplication:
"""
I draw an application select screen:
>>> events = StartupApplication.run_in_test_mode(
... events=[
... [GameLoop.create_event_user_closed_window()],
... ]
... )
"""
I create the bare minimum that the test complains about and get this:
class StartupApplication:
...
@staticmethod
def run_in_test_mode(events=[]):
loop = GameLoop.create_null(
events=events+[
[GameLoop.create_event_user_closed_window()],
]
)
events = loop.track_events()
StartupApplication(loop).run()
return events
def __init__(self, loop):
self.loop = loop
def run(self):
self.loop.run(self)
def event(self, event):
pass
def tick(self, dt):
pass
Now it doesn’t complain, but it seems to hang in an infinite loop.
I modify event
to this:
class StartupApplication:
...
def event(self, event):
if event.is_user_closed_window():
raise ExitGameLoop()
And we’re green. Let’s commit.
$ git commit -a -m 'Emryo to new startup application.'
[main a55d17e] Emryo to new startup application.
2 files changed, 39 insertions(+)
create mode 100644 startup.py
The test is not yet fleshed out. It doesn’t test what it says it tests. But it drove out the skeleton of the application.
It’s been a while since I worked on the balloon shooter. What do I think when I work in this design again?
I got stuck in an infinite loop. That happens because we have a while True:
in our game loop somewhere. I’ve always found testing infinite loops difficult. That’s one reason why I hesitated testing the loop for the startup application. But now I get another idea. What if we create the loop like this instead?
while self.loop_condition.active():
...
Then we can create different versions of the loop condition maybe something like this:
class InfiniteLoopCondition:
def active(self):
return True
class TestLoopCondition:
def __init__(self, iterations):
self.counter = 0
self.iterations = iterations
def active(self):
flag = self.counter >= self.iterations
self.iterations += 1
return flag
Let’s see if we can try this out in the startup application. If it works out well, perhaps we can port it to the game loop as well?
The test that we wrote does not assert anything on the events. Let’s fix that. I comment out the assignment of events
and paste the expected test output:
"""
I draw an application select screen:
>>> StartupApplication.run_in_test_mode(
... events=[
... [GameLoop.create_event_user_closed_window()],
... ]
... )
GAMELOOP_INIT =>
resolution: (1280, 720)
fps: 60
GAMELOOP_QUIT =>
"""
This startup application should run in an infinite loop. In each iteration it should init the game loop and show the game selection screen. Once the selection has been made, it should quit the game loop and run the command. Then it starts all over.
Let’s try the looping thing.
I start by TDDing the loop conditions:
class InifiteLoopCondition:
def active(self):
"""
>>> InifiteLoopCondition().active()
True
"""
That fails. Fix by return true. The other:
class FiteLoopCondition:
def __init__(self, iterations):
self.iterations = iterations
self.count = 0
def active(self):
"""
>>> condition = FiteLoopCondition(iterations=2)
>>> condition.active()
True
>>> condition.active()
True
>>> condition.active()
False
"""
flag = self.count < self.iterations
self.count += 1
return flag
I actually got the condition wrong here at first. I’m glad I wrote a test for it. The previous example, TestLoopCondition
, above is actually wrong. Even for really simple code like this, having tests is nice.
Let’s see if we can use a loop condition and have the test show us that two loops are actually made.
I change
class StartupApplication:
...
@staticmethod
def run_in_test_mode(events=[]):
...
StartupApplication(loop).run()
...
to
class StartupApplication:
...
@staticmethod
def run_in_test_mode(events=[]):
...
StartupApplication(
loop=loop,
loop_condition=FiteLoopCondition(2)
).run()
...
I also notice that i misspelled finite. I fix that and then add the parameter to the class. Test passes. Let’s add an actual loop:
class StartupApplication:
...
def run(self):
while self.loop_condition.active():
self.loop.run(self)
This, expectedly, output another loop which I add to the assertion. Perfect!
GAMELOOP_INIT =>
resolution: (1280, 720)
fps: 60
GAMELOOP_QUIT =>
We are not yet using the InfiniteLoopCondition
. Let’s change that by adding a create
method:
class StartupApplication:
...
@staticmethod
def create():
"""
>>> isinstance(StartupApplication.create(), StartupApplication)
True
"""
return StartupApplication(
loop=GameLoop.create(),
loop_condition=InifiteLoopCondition()
)
I also add this:
if __name__ == "__main__":
StartupApplication.create().run()
And when I run
$ python startup.py
It indeed creates a new window every time I close it.
$ git commit -a -m 'Add startup entry point and have it loop.'
[main aadd1a2] Add startup entry point and have it loop.
1 file changed, 60 insertions(+), 5 deletions(-)
What is the simplest possible solution for selecting a game?
I imagine that the display shows an icon for each game that can be selected. Then you move a cursor over it and press a key to select it.
I start by getting some games on the screen:
def tick(self, dt):
self.loop.clear_screen()
self.loop.draw_text(Point(x=100, y=100), text="SuperTux")
self.loop.draw_text(Point(x=100, y=200), text="Balloon Shooter")
It looks like this:
I think we also need a cursor:
def tick(self, dt):
self.loop.clear_screen()
self.loop.draw_text(Point(x=100, y=100), text="SuperTux")
self.loop.draw_text(Point(x=100, y=200), text="Balloon Shooter")
self.loop.draw_circle(Point(x=500, y=500), radius=20, color="pink")
It looks like this:
Now I think two things are missing. The first is that at the press of a button, the game closest to the cursor should start. The second is that you also need to be able to move the cursor.
I think working on movement is secondary. It is more important to be able to start one game instead of nothing. So let’s work on that first.
I want to write a test for the new behavior, but I find that testing at the top level is tedious and error prone. I would therefore like to start by refactoring and extracting a StartupScene
maybe that has an interface that is easier to test. I end up with this:
class StartupScene:
def event(self, event):
if event.is_user_closed_window():
raise ExitGameLoop()
def draw(self, loop):
loop.draw_text(Point(x=100, y=100), text="SuperTux")
loop.draw_text(Point(x=100, y=200), text="Balloon Shooter")
loop.draw_circle(Point(x=500, y=500), radius=20, color="pink")
I’m sure this refactoring works because I have tests to cover it.
Commit!
Now, let’s see if we can write a test:
"""
When XBOX_A is pressed, I start the game that is closest to the cursor:
>>> scene = StartupScene()
>>> scene.event(GameLoop.create_event_joystick_down(XBOX_A))
SuperTux
"""
I make it pass like this:
class StartupScene:
...
def event(self, event):
...
elif event.is_joystick_down(XBOX_A):
print("SuperTux")
This is obviously faking it. It is not supposed to print the name of the game, it is supposed to run it, or, wait a minute. This class is not supposed to run it, the top-level class is.
Let’s scratch this and start over.
Let’s have a look at the top-level test:
"""
I draw an application select screen:
>>> StartupApplication.run_in_test_mode(
... events=[
... [],
... [GameLoop.create_event_user_closed_window()],
... [],
... [GameLoop.create_event_user_closed_window()],
... ]
... )
GAMELOOP_INIT =>
resolution: (1280, 720)
fps: 60
CLEAR_SCREEN =>
DRAW_TEXT =>
x: 100
y: 100
text: 'SuperTux'
DRAW_TEXT =>
x: 100
y: 200
text: 'Balloon Shooter'
DRAW_CIRCLE =>
x: 500
y: 500
radius: 20
color: 'pink'
GAMELOOP_QUIT =>
GAMELOOP_INIT =>
resolution: (1280, 720)
fps: 60
CLEAR_SCREEN =>
DRAW_TEXT =>
x: 100
y: 100
text: 'SuperTux'
DRAW_TEXT =>
x: 100
y: 200
text: 'Balloon Shooter'
DRAW_CIRCLE =>
x: 500
y: 500
radius: 20
color: 'pink'
GAMELOOP_QUIT =>
"""
This shows our game loop runs twice, but there is no mention that a command is run. Let’s modify
class StartupApplication:
...
def run(self):
while self.loop_condition.active():
self.loop.run(self)
to
class StartupApplication:
...
def run(self):
while self.loop_condition.active():
self.loop.run(self)
print(f"TODO: run {self.startup_scene.get_command()}")
It complains that get_command
does not exist. Let’s add it:
class StartupScene:
def get_command(self):
return ["supertux2"]
...
We are now getting a somewhat expected test failure:
Differences (ndiff with -expected +actual):
+ TODO: run ['supertux2']
+ TODO: run ['supertux2']
GAMELOOP_INIT =>
resolution: (1280, 720)
fps: 60
CLEAR_SCREEN =>
I was thinking to fake this and postpone running the actual command. To do it properly we need an infrastructure wrapper for running commands. I’ll just do it.
Here is a first faked version:
class Command(Observable):
@staticmethod
def create():
return Command()
@staticmethod
def create_null():
return Command()
def run(self, command):
self.notify("COMMAND", {"command": command})
Instead of printing the command, it sends a notification so that we can assert that the event happens at the right time in the test. That is, we can assert that a command is run after the game loop is quit:
...
GAMELOOP_QUIT =>
COMMAND =>
command: ['supertux2']
...
This works. Let’s commit:
$ git commit -a -m 'Run command from StartupScene when game loop is quit.'
[main 4c47b18] Run command from StartupScene when game loop is quit.
1 file changed, 31 insertions(+), 5 deletions(-)
For this to actually do something, we need to flesh out Command
. Here is what I end up with:
class Command(Observable):
"""
>>> Command.create().run(["echo", "hello"])
>>> Command.create().run(["command-that-does-not-exist"])
Traceback (most recent call last):
...
FileNotFoundError: [Errno 2] No such file or directory: 'command-that-does-not-exist'
>>> Command.create_null().run(["command-that-does-not-exist"])
"""
@staticmethod
def create():
return Command(subprocess=subprocess)
@staticmethod
def create_null():
class NullSubprocess:
def run(self, command):
pass
return Command(subprocess=NullSubprocess())
def __init__(self, subprocess):
Observable.__init__(self)
self.subprocess = subprocess
def run(self, command):
self.notify("COMMAND", {"command": command})
self.subprocess.run(command)
When the startup application is run and then quit, SuperTux is actually started.
This is actually some real progress.
$ git commit -a -m 'Command actually runs commands.'
[main 270440e] Command actually runs commands.
1 file changed, 23 insertions(+), 2 deletions(-)
Let’s review the StartupScene
:
class StartupScene:
def get_command(self):
return ["supertux2"]
def event(self, event):
if event.is_user_closed_window():
raise ExitGameLoop()
def draw(self, loop):
loop.draw_text(Point(x=100, y=100), text="SuperTux")
loop.draw_text(Point(x=100, y=200), text="Balloon Shooter")
loop.draw_circle(Point(x=500, y=500), radius=20, color="pink")
We have higher-level tests in place that checks that whatever get_command
returns is run when the game loop quits.
I think it should now be fairly easy to write tests for selection behavior. Let’s first modify the event handler to also exit the game loop when XBOX_A
is pressed:
class StartupScene:
...
def event(self, event):
"""
>>> StartupScene().event(GameLoop.create_event_user_closed_window())
Traceback (most recent call last):
...
gameloop.ExitGameLoop
>>> StartupScene().event(GameLoop.create_event_joystick_down(XBOX_A))
Traceback (most recent call last):
...
gameloop.ExitGameLoop
"""
if event.is_user_closed_window() or event.is_joystick_down(XBOX_A):
raise ExitGameLoop()
Now let’s think about what get_command
should return. It should return the command of the game that is closest to the cursor. Let’s write two tests for that:
class StartupScene:
...
def get_command(self):
"""
>>> scene = StartupScene()
>>> scene.move_cursor(x=100, y=100)
>>> scene.get_command()
['supertux2']
>>> scene.move_cursor(x=100, y=200)
>>> scene.get_command()
['python', '/home/.../agdpp/agdpp.py']
"""
It complains that move_cursor
does not exist. I add it like this:
class StartupScene:
def __init__(self):
self.cursor = Point(x=500, y=500)
def move_cursor(self, x, y):
self.cursor = Point(x=x, y=y)
...
I also modify the drawing code to use this point for the cursor.
Now the second test case fails:
Failed example:
scene.get_command()
Differences (ndiff with -expected +actual):
- ['python', '/home/.../agdpp/agdpp.py']
+ ['supertux2']
I make a quick and dirty fix, because I want to go quickly to green so that I can refactor and generalize the solution:
def get_command(self):
if self.cursor.y == 200:
return ["python", "/home/.../agdpp/agdpp.py"]
return ["supertux2"]
And this is my favorite state of programming. This is actually where some design happens. I have the safety net of the tests and I can push code around until I think it looks good and the next thing is easy to add.
Here is what I come up with this time:
class StartupScene:
def __init__(self):
self.cursor = Point(x=500, y=500)
self.games = [
Game(
name="SuperTux",
position=Point(x=100, y=100),
command=["supertux2"],
),
Game(
name="Balloon Shooter",
position=Point(x=100, y=200),
command=["python", "/home/.../agdpp/agdpp.py"],
),
]
def get_command(self):
return min(
self.games,
key=lambda game: game.distance_to(self.cursor)
).command
def draw(self, loop):
for game in self.games:
game.draw(loop)
loop.draw_circle(self.cursor, radius=20, color="pink")
...
And here is the Game
class:
class Game:
def __init__(self, name, position, command):
self.name = name
self.position = position
self.command = command
def draw(self, loop):
loop.draw_text(self.position, text=self.name)
def distance_to(self, point):
return self.position.distance_to(point)
This implementation still passes all tests and is also generalized. Nice!
$ git commit -a -m 'Run the command closest to the cursor.'
[main 921c71f] Run the command closest to the cursor.
1 file changed, 64 insertions(+), 7 deletions(-)
Next I want to work on cursor movement so that we can actually select different games.
I’m not quite sure how to write a low-level test for this in GameScene
, so I write a top-level test instead:
"""
>>> StartupApplication.run_in_test_mode(
... events=[
... [],
... [GameLoop.create_event_joystick_motion(axis=1, value=1.0)],
... [GameLoop.create_event_user_closed_window()],
... ],
... iterations=1
... ).filter("DRAW_CIRCLE")
DRAW_CIRCLE =>
x: 500
y: 500
radius: 20
color: 'pink'
DRAW_CIRCLE =>
x: 500
y: 501
radius: 20
color: 'pink'
"""
We assert that the cursor is drawn in two different positions given a joystick motion event.
The gist of the implementation is here:
class StartupScene:
...
def event(self, event):
...
elif event.is_joystick_motion():
if event.get_axis() == 0:
self.dx = event.get_value()
elif event.get_axis() == 1:
self.dy = event.get_value()
def update(self, dt):
delta = Point(x=self.dx, y=self.dy)
if delta.length() > 0.05:
self.cursor = self.cursor.add(delta.times(dt))
The update
method did not exist on StartupScene
before. The pattern how it is called is here:
class StartupApplication:
...
def event(self, event):
self.startup_scene.event(event)
def tick(self, dt):
self.loop.clear_screen()
self.startup_scene.update(dt)
self.startup_scene.draw(self.loop)
So the scene will receive these calls in order:
event
update
draw
This represents one game loop cycle. If this pattern becomes more permanent, we can move the top-level test down to StartupApplication
and have that test call event
+ update
and assert that the cursor moved. But for now, I want the confidence that the high-level test gives, that everything is actually working together.
I also test this in game to fist of all make sure that I got the axis right and also to tweak numbers so that speed feels good. The length check is needed because joystick movement events rarely return a value of 0. If we only move the joystick a tiny bit, we don’t want the cursor to move.
Also, we should probably add constant names for the axis to not compare to numbers. Maybe XBOX_AXIS_Y
for example.
Anyway, when I try this out, it actually works. I can move the cursor around, and when I press XBOX_A
the game closest to the cursor is started.
I want to visualize the game that is closest to the cursor. Let’s do it with another color.
class StartupScene:
...
def draw(self, loop):
for game in self.games:
game.draw(loop, self.game_closest_to_cursor())
def game_closest_to_cursor(self):
return min(
self.games,
key=lambda game: game.distance_to(self.cursor)
)
class Game:
...
def draw(self, loop, closest):
loop.draw_text(
self.position,
text=self.name,
color="lightblue" if closest is self else "black"
)
I modify tests to assert the correct color. This works perfectly.
Next I want to fix the games that are configured. I want them to display evenly on the screen, and I want to have a “QUIT” game that runs a shutdown command to shut down the Pi.
Here it is:
class StartupScene:
def __init__(self):
self.cursor = Point(x=400, y=300)
self.games = [
Game(
name="SuperTux",
position=Point(x=100, y=100),
command=["supertux2"],
),
Game(
name="Balloon Shooter",
position=Point(x=400, y=300),
command=["python3", "agdpp.py"],
),
Game(
name="QUIT",
position=Point(x=1000, y=600),
command=["shutdown", "now"],
),
]
And it looks like this:
I change the startup script, /etc/xdg/autostart/game_console_start.desktop
, to this:
[Desktop Entry]
Name=Game console start
Exec=/home/pi/game_console_pc.sh
Where /home/pi/game_console_pc.sh
is this:
#!/usr/bin/env bash
exec > /home/pi/game_console_pc.log
exec 2>&1
cd /home/pi/agdpp
for retry in 1 2 5 10 giveup; do
if [ $retry = giveup ]; then
echo giving up
break
elif git pull --ff-only; then
break
else
echo Retrying in $retry
sleep $retry
fi
done
python3 startup.py
And it works beautifully.
Why did I not test drive this startup script? Good question. I for sure spend some time debugging the loop, which, by the way, is needed to give the Pi time to connect to the wireless network before it can download the latest version of the startup application and balloon shooter.
pi@raspberrypi:~ $ cat game_console_pc.log
fatal: unable to access 'https://github.com/rickardlindberg/agdpp.git/': Could not resolve host: github.com
Retrying in 1
fatal: unable to access 'https://github.com/rickardlindberg/agdpp.git/': Could not resolve host: github.com
Retrying in 2
fatal: unable to access 'https://github.com/rickardlindberg/agdpp.git/': Could not resolve host: github.com
Retrying in 5
Already up to date.
I feel like this script is maybe not part of the game itself. So that is one reason why I just “hacked” it together on the Pi. But I’m not entirely happy that it exists only there, and not in some repo, and doesn’t have any tests.
However, for now, it works fine, but there is another problem. It is not possible to quit the balloon shooter with the gamepad. So once you start it, you are stuck in it.
I modify GameScene
by adding a check for XBOX_START
:
class GameScene:
...
def event(self, event):
if event.is_user_closed_window() or event.is_joystick_down(XBOX_START):
raise ExitGameLoop()
...
And by printing events, I figure out the value of XBOX_START
:
XBOX_START = 7
Finally, I have the first version of the setup that I had in mind.
I find it a little difficult to document all my thinking in this DevLog format. I feel like I make hundreds of decisions every minute when programming, and writing about all of them seems impossible. I think one solution would be to cover smaller changes in each DevLog. Your questions and commends are very welcome.
Even if these DevLogs are not valuable to anyone else, they are valuable to me because I get to practice writing and explaining my thinking.
See you next time!
]]>Published on 4 September 2023.
This is what I’ve been up to in August 2023:
I continued writing DevLogs. All about the development of my video editor.
I wrote the perhaps final article in my Agile Game Development with Python and Pygame series. I will most like continue the series in the form of DevLogs.
I decided that I need to use more of my free time for other things than programming. I still want to continue writing though but, most likely, program a bit less.
Published on 23 August 2023.
I have managed to edit some footage using my own video editor. When I tried to export it, it took forever and eventually crashed. In this DevLog, we will investigate why that might be.
When we press the export button, the following code is run:
class Project:
...
def export(self):
path = "export.mp4"
producer = self.split_into_sections().to_mlt_producer(
profile=self.profile,
cache=ExportSourceLoader(profile=self.profile, project=self)
)
def work(progress):
consumer = mlt.Consumer(self.profile, "avformat")
consumer.set("target", path)
consumer.connect(producer)
consumer.start()
while consumer.is_stopped() == 0:
progress(producer.position()/producer.get_playtime())
time.sleep(0.5)
self.background_worker.add(
f"Exporting {path}",
lambda result: None,
work,
)
It creates an MLT producer with the real clips, and not the proxy clips. The work
function is called in a thread, and this code does the actual export:
consumer = mlt.Consumer(self.profile, "avformat")
consumer.set("target", path)
consumer.connect(producer)
consumer.start()
while consumer.is_stopped() == 0:
progress(producer.position()/producer.get_playtime())
time.sleep(0.5)
As I remember, this is the code that takes forever and eventually crash. I also think its memory consumption steadily increase.
There is not much Python code in here. Just the loop that queries the consumer. So my guess is that something in MLT consumes memory and eventually crashes. We had a similar problem, I think, before when we created proxies using MLT in this way. On the other hand, it seems unlikely that MLT would crash when exporting a “small” project.
What I want to try today is to export my project as an MLT XML file and try to render it using melt. It should do roughly the same thing as my Python code, but will avoid using the Python binding for MLT.
If there is something wrong with MLT, which I doubt, the export will fail here as well. If not, well, then I don’t know what is wrong, but we can at least rule out MLT (core).
We have this code that enables us to export MLT XML:
if sys.argv[1:2] == ["--export-melt"]:
path = sys.argv[2]
print(f"Exporting {path}")
project = Project.load(args=sys.argv[3:])
consumer = mlt.Consumer(project.profile, "xml")
consumer.set("resource", path)
consumer.connect(project.get_preview_mlt_producer())
consumer.start()
while consumer.is_stopped() == 0:
time.sleep(0.5)
print("Done")
return
However, it creates the preview MLT producer which uses the proxy clips.
Since this is just a test, not intended to be committed, I modify this code to instead create an MLT producer with the real clips.
from rlvideolib.domain.project import ExportSourceLoader
producer = project.split_into_sections().to_mlt_producer(
profile=project.profile,
cache=ExportSourceLoader(profile=project.profile, project=project)
)
consumer.connect(producer)
Now we can export the XML like this:
$ rlvideo --export-melt test.xml devlog-009.rlvideo
Exporting test.xml
...
Done
I verify that the XML file has references to the real clips. It does. Perfect!
We can now do the equivalent export with this command:
mlt-melt test.xml -consumer avformat target=export.mp4
And now, it’s just to wait and see what happens.
The memory consumption seems to be quite stable. Unless there is a memory leak, this is what I expect. If the memory consumption keeps increasing for every frame that is exported, that would mean that you can only export longer videos by getting more memory. That does not seem right.
I should probably also verify that the export in the application keeps increasing memory consumption. If it does, then there might be a memory leak in the Python binding for MLT. Or I might use the binding incorrectly.
Using threads (which is used in the export) has also been problematic. I’ve experienced that the Python threads interfere with the MLT threads. I’m don’t understand the problem fully, it’s just a feeling. So that might be something to look into. Try the export with threading disabled.
I might have mistaken. The memory consumption seems to keep increasing. However, the export finish without crashing and the final result looks fine.
It seems that MLT consumes more and more memory the longer the exported video. To confirm this, I should probably do some more precise measures. Maybe using something like psrecord? However, memory consumption might not be problematic in itself. Perhaps it allocates more memory to speed things up, but will not allocated more than what is available. Perhaps the crash that I experienced before was not related to memory.
We have learned something today, and this knowledge will make us better prepared for the future.
Here are a few things I think of as possible next steps in this area:
We’ll see if we work on any of these the next time or something else.
]]>Published on 21 August 2023 in Agile Game Development with Python and Pygame.
When I started this series, my intention was to document my journey of creating a game using agile methods. I think I have mostly succeeded in this regard, but at the moment I’ve done some development that I have not documented. Furthermore, I did that development many months ago, so documenting it gets harder and harder because I forget what I was thinking when I did the development.
Recently though, I’ve experimented with a new format which I call DevLog. It is basically the same thing but a little less polished. I write a DevLog while doing the development, so there is no risk of falling behind. I write about what is going on in my mind as I do the development. Also, I’m not strict about documenting everything in a DevLog. It’s OK to do something and not write about it.
In this post I will briefly mention the development that I’ve done on the balloon shooter but not documented and then talk a little about future plans for this project.
I polish the game a little by adding a particle effect system that I use to render a splashing animation when a balloon is hit.
It looks a little something like this (although it is hard to show in a single image):
The most interesting piece of code is this:
class Balloon:
...
def get_hit_particles(self):
number_of_particles = random.randint(4, 8)
return [
BalloonParticle(
position=self.position.move(
dx=random.randint(0, self.radius),
dy=random.randint(0, self.radius)
),
radius=self.radius*(random.randint(30, 70)/100),
velocity=Angle.fraction_of_whole(random.random()).to_unit_point().times(self.speed*2)
)
for x
in range(number_of_particles)
]
It generates a list of particles when a balloon is hit. The particles have a randomized position, radius, and velocity. The radius keeps decreasing as time passes, and when it reaches a low enough value, the particle is removed.
The complete diff for this change can be seen on GitHub.
Me and my son record sound effects that are played when a balloon is hit. We go to the store, buy some balloons, rig up the mic, and pop them. It is much fun.
The code for integrating the sound can be seen on GitHub.
This change include adding the load_sound
method to GameLoop
:
class GameLoop(Observable):
...
def load_sound(self, path):
return Sound(self.pygame.mixer.Sound(path))
Does it really make sense that you load a sound from the game loop? I’m not sure. The game loop is the only abstraction that we have for accessing pygame. That’s why it ended up there. But the design here feels a little off to me. Something to keep in mind for the future. Next time we touch this area of the code, we might feel the same thing again and have an idea about how to improve.
When I ask my son what he wants the game to do next, he says that he wants to get a medal for every 100 balloon that you shoot down.
I add a fun little particle effect again for the animation when you get a medal:
The medals stack up in the upper left corner like this:
The complete diff for this change can be seen on GitHub.
Testing the medal particle effect is tedious. You have to shoot down 100 balloons, then you can see the effect for a split second, and then you have to shoot down 100 more.
When I have done that enough times, I come up with a better idea. And that is to allow the game to be started in “test mode” where we can trigger the animation with a press of a button.
We can do it like this:
$ ./make.py rundev test-scene-score
Instead of starting the game, it starts a test scene:
if __name__ == "__main__":
if sys.argv[1:] == ["test-scene-score"]:
scene = TestSceneScore()
else:
scene = None
BalloonShooter.create(scene).run()
This test scene is only used for test purposes and looks like this:
class TestSceneScore:
def __init__(self):
self.score = Score()
def event(self, event):
if event.is_user_closed_window():
raise ExitGameLoop()
elif event.is_keydown(KEY_SPACE):
self.score.add_points(100)
def update(self, dt):
self.score.update(dt)
def draw(self, loop):
self.score.draw(loop)
It uses the score object (which is used in the real game) and adds 100 points when we press the space key.
It looks like this:
This way, I can quickly exercise the animation and validate that it looks good.
This project has been inactive for a few months. With my hobby projects, I follow my interest. And my interest has lately been about writing my own video editor. And also, after the medals in place, my son said that the game was finished.
However, right now, I have two ideas that I’m interested in doing. One is trying a decentralized design that Ron has been writing about and doing in his Asteroids Python series.
Another is to create a “game console PC” where I customize this game to first show a start screen where the game to play can be selected. One game will be the balloon shooter. Another will be SuperTux (which me and my son have played a lot).
This post probably marks the end of this series in the current format. When I continue this project, it will be in the form of a DevLog. See you there!
]]>Published on 6 August 2023.
I’ve added a few more timeline edit operations to the video editor. For example, it is now possible to change the speed of a cut with ctrl+drag on the right hand side and modify the in point with drag on the left hand side.
However, changing the out point of a cut by dragging the right hand side does not yet work. It prints the following in the console:
TODO: implement move_right!
It is a bit trickier to get working than changing the in point as we will see in a second.
Here is roughly what happens when you drag the right hand side of a cut:
transaction = project.new_transaction()
transaction.modify(cut_id, lambda cut: cut.move_right(delta))
transaction.commit()
Here is Transaction.modify
:
def modify(self, cut_id, fn):
self.project.set_project_data(self.project.project_data.modify_cut(cut_id, fn))
Here is ProjectData.modify_cut
:
def modify_cut(self, cut_id, fn):
return self._replace(cuts=self.cuts.modify(cut_id, fn))
Here is Cuts.modify
:
def modify(self, cut_id, fn):
...
old_cut = self.cut_map[cut_id]
new_cut = fn(old_cut)
new_cuts = dict(self.cut_map)
new_cuts[cut_id] = new_cut
return self._replace(
cut_map=new_cuts,
region_to_cuts=...,
)
And it is here that the lambda gets called to modify the cut.
The problem is that when we modify the out point, we can’t place it outside the length of the source. And the cut itself does not know how long the source is. It just has a source id where it can be looked up, but only in the ProjectData
structure, which is two levels above.
Let’s have a look at the data structures and what they contain:
class ProjectData(namedtuple("ProjectData", "sources,cuts")):
class Sources(namedtuple("Sources", "id_to_source")):
class FileSource(namedtuple("FileSource", "id,path,length")):
class TextSource(namedtuple("TextSource", "id,text")):
class Cuts(namedtuple("Cuts", "cut_map,region_to_cuts,region_group_size")):
class Cut(namedtuple("Cut", "source,in_out,position,id,mix_strategy,volume,speed")):
Put in a more hierarchical format:
To make sure that a cut’s out point does not exceed the length of the source, we have to make the check in ProjectData since that is the only structure that has both the source information and the cut information.
Let’s have a look at ProjectData.modify_cut
again:
def modify_cut(self, cut_id, fn):
return self._replace(cuts=self.cuts.modify(cut_id, fn))
How about if we did something like this:
def modify_cut(self, cut_id, fn):
def wrapper(cut):
return self.sources.get(cut.source).limit_out_point(fn(cut))
return self._replace(cuts=self.cuts.modify(cut_id, wrapper))
That is, we let the original lambda modify the out point beyond the length of the source. Then in the wrapper above we get the source of the clip and have it adjust the out point to not exceed the length.
I think this will actually work.
When first thinking about this problem I had a much more complicated solution in mind. I was annoyed that the cut itself did not know about the maximum length. I was thinking that Cut.modify
somehow has to be passed a length so that it could do the limiting itself.
Then I started writing about it, and I thought that each data structure should be responsible for validating itself. Since a cut has no information about length, it is ok to specify any length. But when a cut is put into a ProjectData
and is associated with a source, the validation must happen.
This makes a lot of sense to me, and I feel like a made a breakthrough.
You could argue that the design of the data structure is wrong. Perhaps a cut should have more information about its source so that it can do more validation.
But when it looks as it does, I think this will be fine.
Let’s see if we can test this.
Here is the test that I come up with:
"""
A cut's out point is adjusted if going outside the limit:
>>> data = ProjectData.empty()
>>> data = data.add_source(FileSource(id="source_a", path="a.mp4", length=5))
>>> data = data.add_cut(Cut.test_instance(name="source_a", start=0, end=3, id="cut_a"))
>>> data = data.modify_cut("cut_a", lambda cut: cut.move_right(10))
>>> data.get_cut("cut_a").in_out
Region(start=0, end=5)
"""
We create project data with one source and one cut. The source is of length 5 and the cut is of length 3. We can extend it two more frames before we have reached the end of the source.
Then we modify the cut by trying to extend it by 10 frames.
Then we assert that the end point is limited to 5.
This fails with this:
Failed example:
data = data.modify_cut("cut_a", lambda cut: cut.move_right(10))
Differences (ndiff with -expected +actual):
+ TODO: implement move_right!
I implement Cut.move_right
like this:
def move_right(self, amount):
return self._replace(
in_out=self.in_out.move_end(amount),
)
Then we get this failure:
Failed example:
data.get_cut("cut_a").in_out
Differences (ndiff with -expected +actual):
- Region(start=0, end=5)
? ^
+ Region(start=0, end=13)
? ^^
I expected this. We don’t do any limiting yet.
Let’s modify ProjectData.modify_cut
to what we had in mind. I write this:
def modify_cut(self, cut_id, fn):
def wrapper(cut):
return self.get_source(cut.source.source_id).limit_in_out(fn(cut))
return self._replace(cuts=self.cuts.modify(cut_id, wrapper))
We now get this error:
AttributeError: 'TextSource' object has no attribute 'limit_in_out'
This is also to be expected. Now we need to implement limit_in_out
on every type of source. At the moment those are TextSource
and FileSource
. Let’s see if we have coverage for both. We get a failure for TextSource
now, so let’s start there.
A text source does not have a length. It is infinite. So limit_in_out
just becomes this:
def limit_in_out(self, cut):
return cut
Now we get the same error for the file source.
I implement FileSource.limit_in_out
like this:
def limit_in_out(self, cut):
return cut.limit_out(self.length)
The test now complains about this:
AttributeError: 'Cut' object has no attribute 'limit_out'
I implement it like this:
def limit_out(self, max_out):
return self._replace(in_out=self.in_out.limit_end(max_out))
And Region.limit_end
like this:
def limit_end(self, max_end):
return self._replace(end=min(self.end, max_end))
And wow, that actually works.
$ ./make.py commit -m 'ProjectData.modify_cut ensures that in_out is withing source limit.'
...................................................................
----------------------------------------------------------------------
Ran 67 tests in 3.925s
OK
[main a9eb857] ProjectData.modify_cut ensures that in_out is withing source limit.
4 files changed, 29 insertions(+), 3 deletions(-)
Right above modify_cut
I see add_cut
which also has a TODO:
def add_cut(self, cut):
# TODO: assert that source id exists (even for json loading)
return self._replace(cuts=self.cuts.add(cut))
Now that we have touched this area of the code, let’s have a closer look if we can make something cleaner with our new insights.
The add_cut
could probably also benefit from having the in and out points limited.
However, it is not used by the JSON loading mechanism.
I move the JSON loading part of the comment to here:
@staticmethod
def from_json(json):
# TODO: validate the cuts point to valid sources and that they have
# valid in/out points.
return ProjectData(
sources=Sources.from_json(json["sources"]),
cuts=Cuts.from_json(json["cuts"])
)
I’m not sure that we want to adjust cuts that are invalid. We could remove cuts that don’t have a corresponding source, and we could adjust in and out points of cuts with valid sources. But that would change the project. So a load + save will save something else without the user having done any changes. Unless manually modified, a JSON export should never have these problems. So validation should be ok. But I said should. If we make a mistake somewhere, we could export invalid JSON. So a load that fixes bad input it probably a good idea. However, in such cases the user should probably be informed about the changes made and a backup file with the old contents should probably be written. I think this work is for a later time. Not really prioritized now.
Let’s go back to ProjectData.add_cut
. It is only used when the user actively adds a cut somehow. At that point the cut does not exists yet, and if we modify the in and out points, there is no obvious change.
Let’s modify it guided by this test:
"""
In/Out is modified according to source:
>>> ProjectData.empty(
... ).add_source(
... FileSource(id="source_a", path="a.mp4", length=5)
... ).add_cut(
... Cut.test_instance(name="source_a", start=0, end=10, id="cut_a")
... ).get_cut("cut_a").in_out
Region(start=0, end=5)
"""
It fails with this message:
Differences (ndiff with -expected +actual):
- Region(start=0, end=5)
? ^
+ Region(start=0, end=10)
? ^^
We fix it in a similar way to before:
return self._replace(cuts=self.cuts.add(self.get_source(cut.source.source_id).limit_in_out(cut)))
That passes all the tests.
I also noticed an issue with the limiting for cuts that had a changed speed. I modify FileSource.limit_in_out
to take speed into account:
def limit_in_out(self, cut):
"""
>>> source = FileSource(id="source_a", path="a.mp4", length=5)
>>> source.limit_in_out(Cut.test_instance(start=0, end=10)).in_out
Region(start=0, end=5)
>>> source.limit_in_out(Cut.test_instance(start=0, end=20, speed=0.5)).in_out
Region(start=0, end=10)
"""
return cut.limit_out(int(self.length/cut.speed))
We added a feature to the application. It is now possible to move the out point of a cut and it is properly limited to not exceed the length of the underlying source.
I was surprised at how elegant the solution came out. The realisation that made this possible was that validation should happen at the point where all data exists. Each data entity validates itself. If parent attributes are needed for the validation, do the validation higher up the hierarchy.
This also makes me wonder if the limit of in point should also be done by the source. Right now the cut assumes that in point >= 0 is ok. It doesn’t need to know anything about the source. But it makes assumptions about the source. I think this assumption is always correct, but I don’t think it hurts to not assume anything and let the source do the decision.
I will probably try that refactoring out. My suspicion is that the code base will be a little cleaner then.
But not in this session. This is it for now. See you next time!
]]>Published on 5 August 2023.
This is what I’ve been up to in July 2023:
I continued work on my video editor. Here is the initial blog post about it: Writing my own video editor.
I started writing DevLogs. DevLogs is an experiment to try to document development that I do on various projects. I will try to write what is going on in my head as I do various development tasks. So far, they have all been about development of the video editor.
I wrote a blog post called How to get fast feedback on graphical code? I share a technique that I started using while developing the video editor.
Published on 3 August 2023.
I try to edit some footage with my video editor. Actually, it is footage from DevLog 009 that I hope to put together. Everything is going quite well. After I add a split-cut-at-playhead operation to the editor, in addition to the previously added ripple delete, I am actually able to do some useful edits.
However, after a while I notice that a cut does not seem to render the correct frame. I decide to restart the application, and then it happens. Segfault!
This time, the segfault reproduces consistently. I’m excited to debug this and see how we can resolve it. I’ve got my cup of coffee, and I’m ready to go.
Because this is not the first time I see segfaults in this application, I have added a command to run the application in GDB. Here is how to use it:
$ ~/rlvideo/make.py gdb devlog-009.rlvideo
...
Starting program: /usr/bin/python3 /home/rick/rlvideo/rlvideo.py devlog-009.rlvideo
...
Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
..
(gdb) bt
#0 0x00007ffff7a64474 in pthread_mutex_lock () at /lib64/libpthread.so.0
#1 0x00007fffe96866af in XrmQGetResource () at /lib64/libX11.so.6
#2 0x00007fffe9667fca in XGetDefault () at /lib64/libX11.so.6
#3 0x00007fffe9a5ae8a in _cairo_xlib_surface_get_font_options () at /lib64/libcairo.so.2
...
The segfault seems to happen inside some Cairo drawing code. That is most likely happening because GTK is trying to show a widget that tries to draw itself. I think GTK calls were further down in the backrace.
I find it very unlikely that this can happen from the Python GTK bindings. My suspicion is that this has something to do with MLT. Why? Because the segfault only happens for some projects.
I know that many MLT calls return status codes that I never check. Perhaps I should.
There is also a way to serialize an MLT producer to an XML file which can then be played with melt
. That way we can see if MLT has the same problems as we are having given the same MLT producer.
This might be useful for other types of debugging as well.
Let’s see if we can implement that XML export and see if melt
segfaults as well or if that works.
I add this to the main function:
if sys.argv[1:2] == ["--export-melt"]:
path = sys.argv[2]
print(f"Exporting {path}")
project = Project.load(args=sys.argv[3:])
consumer = mlt.Consumer(project.get_preview_profile(), "xml")
consumer.set("resource", path)
consumer.connect(project.get_preview_mlt_producer())
consumer.start()
while consumer.is_stopped() == 0:
time.sleep(0.5)
print("Done")
return
We can run it like this:
$ ./make.py rundev --export-melt test.xml
Exporting test.xml
Done
Then we can feed it to melt
like this:
$ mlt-melt test.xml
When I do, I get this:
[producer_xml] parse fatal: Input is not proper UTF-8, indicate encoding !
Bytes: 0xC0 0xF3 0x68 0x0E
row: 3 col: 25
[producer_xml] parse fatal: invalid character in attribute value
...
There seems to be an encoding issue. I look at the file and see that the profile description looks weird.
I fix it manually, and then get this:
$ mlt-melt test.xml
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
|1=-10| |2= -5| |3= -2| |4= -1| |5= 0| |6= 1| |7= 2| |8= 5| |9= 10|
+-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+ +-----+
+---------------------------------------------------------------------+
| H = back 1 minute, L = forward 1 minute |
| h = previous frame, l = next frame |
| g = start of clip, j = next clip, k = previous clip |
| 0 = restart, q = quit, space = play |
+---------------------------------------------------------------------+
Segmentation fault (core dumped)
Hmm. Now I’m not using the project that I had problems with. Now I’m just using the default test project which works fine otherwise.
When I look closer at the profile in the XML file, other things seem off as well. The width and height don’t seem to be correct either. I try to use the project profile instead of the preview profile in the XML export code. This works better. However, the player only shows a couple of frames where there should be more. What is going on?
Then I notice this at the end of the XML file:
<playlist id="playlist0">
<entry producer="playlist1" in="" out=""/>
<entry producer="producer4" in="0" out="0"/>
</playlist>
</mlt>
The first item in the playlist, which is another playlist, seems to lack in and out arguments. If I change to the following, the file plays ok:
<entry producer="playlist1" in="0" out="43"/>
I print in and out points for all playlists that we create, and they seem to have valid numbers. Time to dig into the MLT XML export code.
I find this:
char *mlt_properties_get_time(mlt_properties self, const char *name, mlt_time_format format)
{
mlt_profile profile = mlt_properties_get_data(self, "_profile", NULL);
if (profile) {
double fps = mlt_profile_fps(profile);
mlt_property value = mlt_properties_find(self, name);
property_list *list = self->local;
return value == NULL ? NULL : mlt_property_get_time(value, format, fps, list->locale);
}
return NULL;
}
The mlt_properties_get_time
functions seems to be used in the XML export. And it seems to work only if there is a profile.
My playlists don’t have profiles.
I add it like this:
diff --git a/rlvideolib/domain/section.py b/rlvideolib/domain/section.py
index 4c50d6d..78a0683 100644
--- a/rlvideolib/domain/section.py
+++ b/rlvideolib/domain/section.py
@@ -33,7 +33,7 @@ class Sections:
return canvas
def to_mlt_producer(self, profile, cache):
- playlist = mlt.Playlist()
+ playlist = mlt.Playlist(profile)
for section in self.sections:
playlist.append(section.to_mlt_producer(profile, cache))
assert playlist.get_playtime() == self.length
@@ -71,7 +71,7 @@ class PlaylistSection:
return canvas
def to_mlt_producer(self, profile, cache):
- playlist = mlt.Playlist()
+ playlist = mlt.Playlist(profile)
for part in self.parts:
part.add_to_mlt_playlist(profile, cache, playlist)
assert playlist.get_playtime() == self.length
Now the export works fine!
Let’s export the XML file for the project that segfaults.
I examine the XML file and notice the same problem for mlt.Tractor
. It is also missing in and out arguments. I add profiles to those as well.
$ ./make.py commit -m 'Pass profile to mlt.Tractor so that XML export works properly with in/out points.'
...........................................................
----------------------------------------------------------------------
Ran 59 tests in 3.024s
OK
[main a5db808] Pass profile to mlt.Tractor so that XML export works properly with in/out points.
1 file changed, 1 insertion(+), 1 deletion(-)
The export works fine and it plays fine in the melt
player.
I think that the fixes we made for the XML export only affects the XML export. But it is nice that we now have the ability to play our projects with melt
. I suspect it might come in handy in the future as well.
So there doesn’t seem to be anything wrong with the producer that we create. Melt can play it just fine. That is good news, I guess, but what to do next?
I mentioned in the beginning that the reason that I restarted the application was that I thought a cut rendered the wrong frame.
I see this problem when playing the XML file with melt as well.
This is most likely something wrong in our code. However, it doesn’t seem to contribute to the segfault.
I add a TODO in the code in a place where I think the problem is. Let’s deal with that later. We are on the hunt for segfault reasons now.
We have concluded that the producer that we create is probably fine.
My suspicion is that there is something in the combination of MLT and GTK that causes the segfault. MLT and GTK are running in the same process, so it might be possible that they interfere with each other somehow. The backtrace got segfaulted inside the pthread library. So perhaps this is also timing related.
Let’s try a few things out.
The thing that connects MLT and GTK is the player. We start an MLT SDL consumer and have it display it’s output in a GTK window.
I try to remove the player like this:
#mlt_player = MltPlayer(self.project, preview.get_window().get_xid())
class MockPlayer:
def position(self):
return 0
mlt_player = MockPlayer()
And now the application starts!
But of course it doesn’t work properly.
However, it tells me that there is something about this combination that causes the segfault.
I then try this:
diff --git a/rlvideolib/gui/gtk.py b/rlvideolib/gui/gtk.py
index cb13bef..3feaf87 100644
--- a/rlvideolib/gui/gtk.py
+++ b/rlvideolib/gui/gtk.py
@@ -160,6 +160,9 @@ class MltPlayer:
# TODO: figure out why SDL consumer seems to produce brighter images (black -> grey)
self.project = project
os.putenv("SDL_WINDOWID", str(window_id))
+ GLib.idle_add(self.init_player)
+
+ def init_player(self):
self.consumer = mlt.Consumer(self.project.get_preview_profile(), "sdl")
self.consumer.start()
self.producer = None
That is, I create the MLT consumer a little later, once GTK has had time to start up a bit more.
And wow, this actually works!
I though about this idea because I had come across this comment in the Flowblade source code:
# SDL 2 consumer needs to created after Gtk.main() has run enough for window to be visible
#if editorstate.get_sdl_version() == editorstate.SDL_2: # needs more state consideration still
# print "SDL2 timeout launch"
# global sdl2_timeout_id
# sdl2_timeout_id = GLib.timeout_add(1500, create_sdl_2_consumer)
The comment was for SDL2, and we are using SDL1, but I thought it was worth a try anyway.
Here is one reason that I think it is valuable documenting my work. I was able to get an idea from Flowblade. From a comment written in the source code. That was valuable to me. Maybe others will find similar value in what I write about. Maybe.
I try the idle_add
solution a couple of times, and it seems like I was too fast to declare victory. It seems like it still segfaults sometimes.
Then I try to take the SDL consumer out of the picture by replacing it with this:
class DummyConsumer:
def disconnect_all_producers(self):
print("Dummy disconnect")
def connect(self, producer):
print("Dummy connect")
And it still segfaults sometimes.
I’m thinking that we need to delay all MLT operations until GTK is properly initialized.
I try to get this to work, but I don’t manage. The code is too tangled together.
Many hours pass, and I don’t seem to be making any progress.
I’m thinking that this segfault might have to do with the bug I talked about in the beginning about the wrong frame being rendered.
I find the problem in the code, write a test that exposes the bug, and then fix it.
That was good, but it did not resolve the segfault.
I keep scratching my head, thinking of things to try. Hours pass. Then I have a breakthrough.
But some lucky guess, I find out that the segfault only happens when we have overlapping clips in our project. I decide to comment out transitions (the code that merges multiple, overlapping frames together), and suddenly, the reproducible segfault goes away. The problem seems to be with the qtblend
transition. There is another one called frei0r.cairoblend
which works as well for our purposes. I switch to that one and write this comment in the code about it.
# 'qtblend' that was first used first seems to give problems
# when used in a GTK context. The application segfaults when
# started.
#
# Steps to reproduce:
#
# 1. ./make.py rundev foo.rlvideo resources/*mp4
#
# 2. Move a cut so that there is a overlap somewhere
#
# 3. ./make.py rundev foo.rlvideo
#
# Boom! Stacktrace:
#
# (gdb) bt
# #0 0x00007ffff7a64474 in pthread_mutex_lock () at /lib64/libpthread.so.0
# #1 0x00007fffe96866af in XrmQGetResource () at /lib64/libX11.so.6
# #2 0x00007fffe9667fca in XGetDefault () at /lib64/libX11.so.6
# #3 0x00007fffe9a5ae8a in _cairo_xlib_surface_get_font_options () at /lib64/libcairo.so.2
# ...
#
# frei0r.cairoblend seems to work better.
#
# TODO: How to fix this problem? Is qtblend just incompatible?
I am extremely satisfied that we found the reason for the segfault and were able to fix it.
In the process we also found a couple of other bugs that we fixed and added the XML export for easier debugging.
After fixing the segfault I continue to edit. Unfortunately, I get other segfaults now. This time not reproducible, but more random. I conclude that I must learn better the internals of MLT to figure out what I’m doing wrong in the Python code. And after the things I learned from this session, I’m more prepared.
]]>Published on 3 August 2023.
As a try to edit some footage with my video editor, I get annoyed by a timeline scrubbing issue.
Scrubbing the timeline means clicking and dragging the playhead and then the frame at that position will play. This works fine today if you click and drag, but if you only click, nothing happens:
Sometimes I just want to place the playhead at a certain position. And then I just want to click.
That’s what we’ll work on fixing today.
Here is the scrub action:
class ScrubAction(Action):
def __init__(self, player, scrollbar):
self.player = player
self.scrollbar = scrollbar
self.mouse_up()
def left_mouse_down(self, x, y):
self.x = x
def mouse_up(self):
self.x = None
def mouse_move(self, x, y):
if self.x is not None:
self.player.scrub(
int(round(
self.scrollbar.content_start
+
x/self.scrollbar.one_length_in_pixels
))
)
We can see that the scrubbing is happening only when we move the mouse, not if we just left click.
The solution seems obvious: make sure to scrub on the click as well.
Let’s see how we can move slowly and carefully and pay attention to design as we go along. Let’s start with a test.
This is the test that I come up with:
"""
I scrub the player when clicked:
>>> class MockPlayer:
... def scrub(self, position):
... print(f"scrub {position}")
>>> class MockScrollbar:
... content_start = 0
... one_length_in_pixels = 1
>>> action = ScrubAction(player=MockPlayer(), scrollbar=MockScrollbar())
>>> action.simulate_click(x=10)
scrub 10
"""
The left_mouse_down
currently takes both the x and y coordinates. In this test, we only care about the x coordinate. That’s why I introduced Action.simulate_click
. The idea is that it should simulate the calls that GTK does when a left click happens. My idea is to extend this further with something like Action.simulate_drag
which will fire left_mouse_down
, mouse_move
, and mouse_up
in the same way that GTK would do it.
I implement it like this:
def simulate_click(self, x=0, y=0):
self.left_mouse_down(x=x, y=y)
To make the test pass, I call self.player.scrub
in the left_mouse_down
event as well. I extract it to a common method to remove the duplication.
This passes the tests, and when I try it in the application, it works as intended.
Are we done?
Let’s take a moment to think about some design issues.
One thing that worry me is that Action.simulate_click
does not actually simulate clicks in the right way. That is, when we hook this up with GTK, the same kinds of events will not be generated.
Let’s have a look at how it works today.
Here is how *_mouse_down
is handled:
timeline = Gtk.DrawingArea()
timeline.connect("button-press-event", timeline_button)
timeline.add_events(
timeline.get_events() |
Gdk.EventMask.SCROLL_MASK |
Gdk.EventMask.BUTTON_PRESS_MASK |
Gdk.EventMask.BUTTON_RELEASE_MASK |
Gdk.EventMask.POINTER_MOTION_MASK
)
def timeline_button(widget, event):
# TODO: clarify what translate_coordinates do
if event.button == 1:
self.timeline.left_mouse_down(*timeline.translate_coordinates(
main_window,
event.x,
event.y
))
elif event.button == 3:
self.timeline.right_mouse_down(*timeline.translate_coordinates(
main_window,
event.x,
event.y
), GtkGui(event))
This code exists in a method which has a bunch of other GTK setup code and is quite long.
Let’s see if we can extract a GTK widget that has all the mechanisms for custom drawing and event handling.
I slowly start to extract pieces, and eventually end up with this:
class CustomDrawWidget(Gtk.DrawingArea):
def __init__(self, main_window, custom_draw_handler):
Gtk.DrawingArea.__init__(self)
self.add_events(
self.get_events() |
Gdk.EventMask.SCROLL_MASK |
Gdk.EventMask.BUTTON_PRESS_MASK |
Gdk.EventMask.BUTTON_RELEASE_MASK |
Gdk.EventMask.POINTER_MOTION_MASK
)
self.connect("draw", self.on_draw)
self.connect("button-press-event", self.on_button_press_event)
self.connect("button-release-event", self.on_button_release_event)
self.connect("motion-notify-event", self.on_motion_notify_event)
self.rectangle_map = RectangleMap()
self.custom_draw_handler = custom_draw_handler
self.down_action = None
self.main_window = main_window
def on_draw(self, widget, context):
self.rectangle_map.clear()
self.custom_draw_handler(context, self.rectangle_map)
def on_button_press_event(self, widget, event):
x, y = self.get_coordinates_relative_self(event)
if event.button == 1:
self.down_action = self.rectangle_map.get(x, y, Action())
self.down_action.left_mouse_down(x, y)
elif event.button == 3:
self.down_action = self.rectangle_map.get(x, y, Action())
self.down_action.right_mouse_down(GtkGui(event))
def on_motion_notify_event(self, widget, event):
x, y = self.get_coordinates_relative_self(event)
if self.down_action:
self.down_action.mouse_move(x, y)
else:
self.rectangle_map.get(x, y, Action()).mouse_move(x, y)
def on_button_release_event(self, widget, event):
if self.down_action:
self.down_action.mouse_up()
self.down_action = None
def get_coordinates_relative_self(self, event):
return self.translate_coordinates(
self.main_window,
event.x,
event.y
)
The timeline is then created like this:
timeline = CustomDrawWidget(
main_window=main_window,
custom_draw_handler=timeline_draw,
)
This part of the code base does not have many tests. I therefore moved slowly and tested my changes manually after each small step.
Let’s discuss some aspects of this and what we have done:
The CustomDrawWidget
now owns the rectangle map. (The timeline gets a reference to it, but there it only one instance, and it is created by CustomDrawWidget
.)
The CustomDrawWidget
can handle clearing of the rectangle map on redraw, something that the timeline previously did.
The CustomDrawWidget
can handle mouse events and take the appropriate action by using the rectangle map.
The timeline widget no longer knows about mouse events. It just has a rectangle map that it can fill with actions to be performed.
When I look at this, I feel like there are so many more things to improve. However, I will practice stopping here and think that I made a bit of improvement.
We can now see a bit more clearly the connection between GTK events, the rectangle map, and what methods are called on the action. And, if we need a second component that does custom drawing and handles events with a rectangle map, we can re-use CustomDrawWidget
and do not need to duplicate as much.
We improved the application a tiny bit by allowing click on the timeline to position the playhead. We also cleaned up the code base in the area we touched. It now reflects a little better the ideas that we have about the code. I’m happy with this progress.
]]>Published on 2 August 2023.
As I sit down this morning to continue work on my video editor, I don’t feel motivated at all. I browse some code that I worked on yesterday, see problems with it, but can’t really see how to improve it. Everything feels complicated, and I don’t feel like programming at all.
What to do?
Well, this is just a hobby project of mine. I could just do something else today. But let’s pretend that it’s not. After all, I still have a desire to make progress on this project.
Generally speaking, I can make two types of changes to the code:
Yesterday, I spent most of the day refactoring and designing. I still feel that the design needs improvements in the area that I worked on, but I find that extra hard to motivate myself to work on today.
And perhaps that is also the wrong thing to do? Yesterday I improved the design a little to the point where fixing an actual problem was easier. Shouldn’t that be enough?
If we keep improving a little bit for every feature we work on, we never have to exclusively work on refactoring.
When I write that, it makes sense to me. I should practice feeling content with having made some improvements. I should practice not striving for perfection.
So how can we improve the product? What is something that we can add that makes it easier, more pleasant, or more efficient for me to edit footage?
Yesterday I was annoyed by proxy clip loading time. I am still annoyed by that, but I have a feeling it will be a little difficult to fix. And I don’t feel up for it this morning.
Is there something easier that we can work on?
Yes, there is!
One common thing that happens when I shoot is that some clips turn out to be complete garbage. I might have pressed the record button by mistake or I might have an out of focus shot. In those cases I just want to discard the clip.
Say that clip C0015.MP4
below is out of focus.
I want to open up the context menu for that cut and choose “ripple delete”. It should remove that cut from the timeline and move all cuts to the right of it left to fill up the space.
If we add this feature, I can actually start editing some footage. Because that is how I usually edit videos. I drop all clips on the timeline and then I cut things apart and make it shorter. With this new feature, I still can’t make any cuts, but I can discard clips.
I keep mixing the words clip and cut. A clip means a file on disk. When a clip is added to the timeline, a cut is created that spans the whole region of the clip. So in the beginning, the clip and the cut is of equal length. However, the cut can change in and out points of the clip, making it shorter.
I will try to go slowly when working on this feature and pay attention to the design as I go along.
I will try to make small refactorings to improve the design along the way, but the focus will still be to implement this feature.
Let’s get started.
Let’s start with the context menu for a cut.
class CutAction(Action):
...
def right_mouse_down(self, x, y, gui):
def mix_strategy_updater(value):
def update():
with self.project.new_transaction() as transaction:
transaction.modify(self.cut.id, lambda cut:
cut.with_mix_strategy(value))
return update
gui.show_context_menu([
MenuItem(label="over", action=mix_strategy_updater("over")),
MenuItem(label="under", action=mix_strategy_updater("under")),
])
Aha, we are back to the CutAction
that we worked on in the previous DevLog. This is an opportunity to make design improvements to it while still focusing on the new ripple delete feature.
Making design improvements is always easier when we have tests, and it is many times my preferred way of adding new functionality. So let’s start there. This is what I come up with:
"""
I show a menu item for ripple delete:
>>> project = None
>>> cut = None
>>> scrollbar = None
>>> action = CutAction(project, cut, scrollbar)
>>> gui = TestGui()
>>> action.right_mouse_down(x=None, y=None, gui=gui)
>>> gui.print_context_menu_items()
over
under
ripple delete
"""
At this point, I just want to assert that we have a ripple delete menu item.
I null out any parameters that are not used.
To make this test run, I also have to extend TestGui
with print_context_menu_items
:
diff --git a/rlvideolib/gui/testing.py b/rlvideolib/gui/testing.py
index aaba74d..0df0dda 100644
--- a/rlvideolib/gui/testing.py
+++ b/rlvideolib/gui/testing.py
@@ -4,7 +4,12 @@ class TestGui:
self.click_context_menu = click_context_menu
def show_context_menu(self, menu):
+ self.last_context_menu = menu
for item in menu:
if item.label == self.click_context_menu:
item.action()
return
+
+ def print_context_menu_items(self):
+ for item in self.last_context_menu:
+ print(item.label)
Let’s make it pass:
@@ -290,6 +305,7 @@ class CutAction(Action):
gui.show_context_menu([
MenuItem(label="over", action=mix_strategy_updater("over")),
MenuItem(label="under", action=mix_strategy_updater("under")),
+ MenuItem(label="ripple delete", action=lambda: None),
])
def mouse_up(self):
Just enough to make the test pass. We now have a context menu item that will do nothing when we click on it.
$ ./make.py commit -m 'New ripple delete context menu item that does nothing.'
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 2.893s
OK
[main 2659383] New ripple delete context menu item that does nothing.
2 files changed, 21 insertions(+)
The TestGui
that we had to modify for this test lives in the rlvideolib.gui.testing
module. The rlvideolib.gui
package looks like this:
rlvideolib/gui
├── framework.py
├── generic.py
├── gtk.py
├── __init__.py
└── testing.py
We recently extracted the framework module. It contains framework related GUI code that does not depend on GTK and does not depend on our application. It makes sense for a framework to include facilities to help testing, right?
Let’s get rid of the testing module and move its contents to the framework module.
$ ./make.py commit -m 'Move TestGui to rlvideolib.gui.framework and get rid of the testing module.'
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 2.896s
OK
[main 3ea95fc] Move TestGui to rlvideolib.gui.framework and get rid of the testing module.
4 files changed, 21 insertions(+), 3 deletions(-)
rename rlvideolib/gui/{testing.py => framework.py} (58%)
We have made small progress towards the ripple delete feature and made the code base a little cleaner by indicating that test helpers are part of the GUI framework. Nice!
I feel much more motivated now than when I got started. But before I move on, I will take a break and have some breakfast.
Let’s go back to the test. This is what we have:
"""
>>> project = None
>>> cut = None
>>> scrollbar = None
>>> action = CutAction(project, cut, scrollbar)
>>> gui = TestGui()
>>> action.right_mouse_down(x=None, y=None, gui=gui)
>>> gui.print_context_menu_items()
over
under
ripple delete
"""
That feels like a lot of set up to me. And many of the parameters are None
.
I take a closer look at the x and y coordinates. As far as I can tell, no action is using those in the right_mouse_down
method. Let’s get rid of them.
$ ./make.py commit -m 'Get rid of x and y coordinates in Action.right_mouse_down since they are never used.'
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 3.403s
OK
[main 2c4c80e] Get rid of x and y coordinates in Action.right_mouse_down since they are never used.
3 files changed, 4 insertions(+), 4 deletions(-)
Let’s further refactor the test to this:
"""
I show cut menu items on right click:
>>> gui = TestGui()
>>> action = CutAction(project=None, cut=None, scrollbar=None)
>>> action.right_mouse_down(gui=gui)
>>> gui.print_context_menu_items()
over
under
ripple delete
"""
This indicates that the showing of the menu does not depend on the project, cut, or scrollbar. I think that it reads quite nicely.
$ ./make.py commit -m 'Change cut action test to be assertion for menu items shown.'
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 3.411s
OK
[main 1708e0c] Change cut action test to be assertion for menu items shown.
1 file changed, 2 insertions(+), 5 deletions(-)
Now we need a new test for clicking the ripple delete menu item.
I write this test:
"""
I ripple delete:
>>> gui = TestGui(click_context_menu="ripple delete")
>>> action = CutAction(project=None, cut=None, scrollbar=None)
>>> action.right_mouse_down(gui=gui)
do ripple delete
"""
That is, I assert that “do ripple delete” is printed when we press that menu item. Baby steps.
I make it pass like this:
@@ -299,10 +306,12 @@ class CutAction(Action):
transaction.modify(self.cut.id, lambda cut:
cut.with_mix_strategy(value))
return update
+ def ripple_delete():
+ print("do ripple delete")
gui.show_context_menu([
MenuItem(label="over", action=mix_strategy_updater("over")),
MenuItem(label="under", action=mix_strategy_updater("under")),
- MenuItem(label="ripple delete", action=lambda: None),
+ MenuItem(label="ripple delete", action=ripple_delete),
])
$ ./make.py commit -m 'Add non-empty action for ripple delete.'
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 2.909s
OK
[main 4c4e272] Add non-empty action for ripple delete.
1 file changed, 10 insertions(+), 1 deletion(-)
Let’s take the next step and assert that it actually does a ripple delete.
I modify the test to this:
"""
I ripple delete:
>>> from rlvideolib.domain.project import Project
>>> project = Project.new()
>>> with project.new_transaction() as transaction:
... hello_id = transaction.add_text_clip("hello", length=10, id="A")
... _ = transaction.add_text_clip("there", length=10, id="B")
>>> project.split_into_sections().to_ascii_canvas()
|<-A0-----><-B0----->|
>>> CutAction(
... project=project,
... cut=project.project_data.get_cut(hello_id),
... scrollbar=None
... ).right_mouse_down(
... gui=TestGui(click_context_menu="ripple delete")
... )
>>> project.split_into_sections().to_ascii_canvas()
|<-B0----->|
"""
This got quite messy. Let’s see if we can break it down. First we setup a new project with two clips next to each other. Then we simulate that the ripple delete menu item is clicked and assert that the first clip is removed and the second clip is moved to the beginning.
The setup of the project is kind of messy. For example, we have to do the import in the doctest to prevent a circular import. And we reach in to grab the project data to get the cut.
There are many things to improve here.
But I think I want to move on and get it to pass. We’ll get back to the issues above. I promise.
I make this change:
def ripple_delete():
- print("do ripple delete")
+ self.project.ripple_delete(self.cut.id)
gui.show_context_menu([
That tells me that ‘Project’ object has no attribute ‘ripple_delete’.
I add it like this:
class Project:
...
def ripple_delete(self, cut_id):
with self.new_transaction() as transaction:
transaction.ripple_delete(cut_id)
That tells med that ‘Transaction’ object has no attribute ‘ripple_delete’.
We’re getting closer.
I can’t come up with the general solution for ripple delete, so I hard code a solution for the particular case where we only have two cuts in the project:
class Transaction:
...
def ripple_delete(self, cut_id):
data = self.project.project_data
data = data.remove_cut(cut_id)
data = data.modify_cut(list(data.cuts.cut_map.keys())[0], lambda cut: cut.move(-10))
self.project.set_project_data(data)
And also make a quick and dirty version of remove_cut
.
$ ./make.py commit -m 'Quick and dirty version of ripple delete that works in one case.'
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 3.902s
OK
[main 7ba45c9] Quick and dirty version of ripple delete that works in one case.
2 files changed, 43 insertions(+), 5 deletions(-)
Here is the example project:
If we try to ripple delete the first clip in the GUI, we get this:
Not quite right, but it shows progress in the right direction.
I think we can leave the cut action test alone for a while now. It is fine. Now we need to turn our attention to the ripple delete method in the transaction and make it work as intended.
The project has a hierarchy of classes representing the different parts. The ripple delete only affects the cuts. I make that clear by just forwarding ripple_delete
calls until we get to Cuts
.
Transaction forwards to project data:
class Transaction:
...
def ripple_delete(self, cut_id):
self.project.set_project_data(self.project.project_data.ripple_delete(cut_id))
And project data forwards to cuts:
class ProjectData(namedtuple("ProjectData", "sources,cuts")):
...
def ripple_delete(self, cut_id):
return self._replace(cuts=self.cuts.ripple_delete(cut_id))
And finally, the hard coded ripple delete is here:
class Cuts(namedtuple("Cuts", "cut_map,region_to_cuts,region_group_size")):
...
def ripple_delete(self, cut_id):
data = self
data = data.remove(cut_id)
data = data.modify(list(data.cut_map.keys())[0], lambda cut: cut.move(-10))
return data
$ ./make.py commit -m 'Move ripple_delete down to Cuts.'
.....................................................
----------------------------------------------------------------------
Ran 53 tests in 3.397s
OK
[main c5deb77] Move ripple_delete down to Cuts.
2 files changed, 10 insertions(+), 7 deletions(-)
If we can just get that one working properly, I think our feature is done.
Implementing the ripple delete was much more difficult than I expected. I feel like I hit problem after problem that I need to solve before I can actually get to the ripple delete. I get tired and demotivated again.
Let’s recap what’s left on our imaginary TODO list.
First there is the issue of the messy cut action test, where it was particularly painful to setup a project.
And then there is the ripple delete in cuts that is not fully implemented.
I actually think that is it.
Not as much as I felt it was. Writing it down helped me realize that.
I think both of them might be a little difficult and take some time. Instead of documenting them in detail, I will just report on the status once done.
I managed to generalize ripple delete and add some tests for it.
I’m not sure it’s perfect, but we can improve it later with the help of test. I’m sure it is not harmful at least.
However, it brings up another question. What if we do a ripple delete on the wrong clip? How to recover from that? The answer right now is that we can’t. Therefore, I think an undo function is high on the priority list. It should be relatively straight forward to implement thanks to the immutable data structures.
When it comes to the project setup, I just didn’t have the energy to do anything about it. I know this project setup is done in a few test, so I’m sure we will come across it later. Hopefully I have a better idea for how to improve it then. And some more energy. I’m OK leaving it like this. I don’t think we have made things worse. So much for a promise to get back to it.
This change took longer than expected in part because the design was not clean enough in a few places, and in part because the project was lacking methods for modifying cuts because there had been no need for it.
We cleaned up the design in a few places add added a bit more functionality for project editing operations.
Next time we work in this area, I think we can move faster.
Is this evolutionary design?
This also gets me thinking about stories and estimating stories and how that does not make sense in this context. If things get easier and easier to implement over time, that would also mean that time to complete a story takes less and less time. So you can’t really estimate multiple stories, because the estimate changes once the previous story is completed. At least if stories somewhat overlap it terms of changes in the code base.
]]>Published on 1 August 2023.
In this session I will select what to work on next in my video editor by trying to use it to edit some footage and see where I get stuck.
I’ve previously managed to create a project which has some footage imported and proxy clips generated. I can open that project like this:
$ rlvideo my-project.rlvideo
When I do that, two things happen that annoy me.
First of all, there are lots of exceptions printed to the console:
Traceback (most recent call last):
File "/home/rick/rlvideo/rlvideolib/gui/gtk.py", line 80, in timeline_draw
self.timeline.draw_cairo(
File "/home/rick/rlvideo/rlvideolib/gui/generic.py", line 200, in draw_cairo
self.draw_scrollbar(context, area, playhead_position)
File "/home/rick/rlvideo/rlvideolib/gui/generic.py", line 287, in draw_scrollbar
self.rectangle_map.add(Rectangle(
File "/home/rick/rlvideo/rlvideolib/graphics/rectangle.py", line 19, in __init__
raise ValueError("Width must be > 0.")
ValueError: Width must be > 0.
And second of all, it seems like it’s loading proxy clips again even though they are already generated:
Which one should I work on? Should I work on something else? What is most important?
Let’s do an analysis of why the two problems occur.
The exception when drawing the scrollbar happens because there are too many clips in a too small window, so the width of the scrollbar handle gets smaller than 1 pixel. It can be worked around by zooming out a bit so that a larger portion of the timeline is visible.
This is obviously not good, but not the end of the world.
The fix probably involves setting a minimum width on the handle.
What about proxies?
Actually, proxies are not created again, but in order to find the correct proxy for a clip, the clip’s md5 sum has to be calculated. This is much faster than generating the proxy, but still takes some time, delaying me when I want to edit clips.
The fix probably involves storing the path of the proxy clip in the project file.
It is also not the end of the world. I can open the editor, go make some coffee, and maybe when I’m back, it’s done.
So which should I work on?
If you work in an agile fashion, doing evolutionary design, what should happen is that it should get easier and easier to work with the code base and add new features. I learned that from James Shore.
Say I start working on the scrollbar exception now. When I’m done with that, it should be easier to fix the proxy loading issue than it was before, assuming that the areas that need change overlap.
With that kind of thinking, it doesn’t matter that much what we choose to work on as long as we think it is somewhat important. Just pick one and the next thing will be easier.
It almost sounds too good to be true, but I believe in it. For this to work though, we need to practice evolutionary design. We’ll do that today.
Let’s pick the scrollbar issue.
The error happens in draw_scrollbar
which looks like this:
def draw_scrollbar(self, context, area, playhead_position):
x_start = self.scrollbar.region_shown.start / self.scrollbar.whole_region.length * area.width
x_end = self.scrollbar.region_shown.end / self.scrollbar.whole_region.length * area.width
playhead_x = playhead_position / self.scrollbar.whole_region.length * area.width
# TODO: add callback mechanism in rectangle map
x, y, w, h = (
area.x+x_start,
area.y,
x_end-x_start,
area.height
)
rect_x, rect_y = context.user_to_device(x, y)
rect_w, rect_h = context.user_to_device_distance(w, h)
self.rectangle_map.add(Rectangle(
x=int(rect_x),
y=int(rect_y),
width=int(rect_w),
height=int(rect_h)
), "position")
context.rectangle(area.x, area.y, area.width, area.height)
context.set_source_rgba(0.4, 0.9, 0.4, 0.5)
context.fill()
scroll_box = Rectangle(x, y, w, h)
context.rectangle(scroll_box.x, scroll_box.y, scroll_box.width, scroll_box.height)
context.set_source_rgba(0.4, 0.9, 0.4, 0.5)
context.fill()
# Playhead
context.set_source_rgb(0.1, 0.1, 0.1)
context.move_to(playhead_x, area.top)
context.line_to(playhead_x, area.bottom)
context.stroke()
context.set_source_rgb(0.1, 0.1, 0.1)
scroll_box.draw_pixel_perfect_border(context, 2)
When I look at this, it’s difficult for me to see what is going on. It is just too long and does too much. It doesn’t clearly represent what I had in mind when I wrote it.
If we are going to do evolutionary design, we have to pay more attention to design. All the time.
It’s fine that I didn’t pay too much attention last time I modified this method, but now that we are here again, let’s give it some extra love so that it is easier to work with next time.
The error happens when creating the rectangle in the following piece of code:
# TODO: add callback mechanism in rectangle map
x, y, w, h = (
area.x+x_start,
area.y,
x_end-x_start,
area.height
)
rect_x, rect_y = context.user_to_device(x, y)
rect_w, rect_h = context.user_to_device_distance(w, h)
self.rectangle_map.add(Rectangle(
x=int(rect_x),
y=int(rect_y),
width=int(rect_w),
height=int(rect_h)
), "position")
Look, there is even a TODO comment there. Now that we are touching this piece of code again, perhaps it’s time to deal with it.
The rectangle map is used to store areas of the screen that the user can interact with. You can put objects at a given rectangle and retrieve them by position. Here is an example:
"""
>>> r = RectangleMap()
>>> r.add(Rectangle(x=0, y=0, width=10, height=10), "item")
>>> r.get(5, 5)
'item'
>>> r.get(100, 100) is None
True
"""
In the timeline area, each cut puts itself in a rectangle, allowing a context menu to be shown when it is right clicked like this:
def right_mouse_down(self, x, y, gui):
cut = self.rectangle_map.get(x, y)
if isinstance(cut, Cut):
# show context menu
The TODO comment that I wrote suggests that we should instead store objects that can handle right_mouse_down
events for example so that we don’t need to check instances at the outermost event handler.
Let’s see if we can do it.
I sketch this:
class Action:
def left_mouse_down(self, x, y):
pass
def right_mouse_down(self, x, y, gui):
pass
def mouse_move(self, x, y):
pass
def mouse_up(self):
pass
class ScrollbarDragAction(Action):
def __init__(self, timeline, scrollbar):
self.timeline = timeline
self.scrollbar = scrollbar
self.mouse_up()
def left_mouse_down(self, x, y):
self.x = x
def mouse_up(self):
self.x = None
def mouse_move(self, x, y):
if self.x is not None:
self.timeline.set_scrollbar(
self.scrollbar.move_scrollbar(
x - self.x
)
)
Let’s see if we can use it.
I modify right_mouse_down
and all the other event handlers to this:
def right_mouse_down(self, x, y, gui):
item = self.rectangle_map.get(x, y)
if isinstance(item, Action):
item.right_mouse_down(x, y, gui)
self.down_item = item
return
...
This is special handling for the case where the entry in the rectangle map is an Action
. Eventually, we want there to be only actions in there, and then the instance check can be removed.
Next I change what we put into the rectangle map for the scrollbar to this:
self.rectangle_map.add(Rectangle(
x=int(rect_x),
y=int(rect_y),
width=int(rect_w),
height=int(rect_h)
), ScrollbarDragAction(self, self.scrollbar))
Boom! Test failure:
Failed example:
timeline.rectangle_map # doctest: +ELLIPSIS
Differences (ndiff with -expected +actual):
...
Rectangle(x=0, y=0, width=300, height=20):
scrub
Rectangle(x=0, y=77, width=300, height=23):
- position
+ <rlvideolib.gui.generic.ScrollbarDragAction object at 0x7fd1f8891d00>
There is now another object in the rectangle map. Let’s modify the test to assert that instead.
The question now is, will it work in the application?
This behavior I think lacks tests, so let’s try.
Nothing happens.
I review the code and find that I had forgotten the mouse_move
event:
def mouse_move(self, x, y):
if self.down_item:
self.down_item.mouse_move(x, y)
return
...
And that actually works!
$ ./make.py commit -m 'Add a ScrollbarDragAction instead of position string.'
...................................................
----------------------------------------------------------------------
Ran 51 tests in 3.405s
OK
[main 87c9b07] Add a ScrollbarDragAction instead of position string.
1 file changed, 59 insertions(+), 2 deletions(-)
I make the same change for the remaining actions.
Here is the one for scrubbing the timeline:
class ScrubAction(Action):
def __init__(self, player, scrollbar):
self.player = player
self.scrollbar = scrollbar
self.mouse_up()
def left_mouse_down(self, x, y):
self.x = x
def mouse_up(self):
self.x = None
def mouse_move(self, x, y):
if self.x is not None:
self.player.scrub(
int(round(
self.scrollbar.content_start
+
x/self.scrollbar.one_length_in_pixels
))
)
And here is the one for moving a cut and opening the context menu for a cut:
class CutAction(Action):
def __init__(self, project, cut, scrollbar):
self.project = project
self.cut = cut
self.scrollbar = scrollbar
self.mouse_up()
def left_mouse_down(self, x, y):
self.transaction = self.project.new_transaction()
self.x = x
def right_mouse_down(self, x, y, gui):
def mix_strategy_updater(value):
def update():
with self.project.new_transaction() as transaction:
transaction.modify(self.cut.id, lambda cut:
cut.with_mix_strategy(value))
return update
gui.show_context_menu([
MenuItem(label="over", action=mix_strategy_updater("over")),
MenuItem(label="under", action=mix_strategy_updater("under")),
])
def mouse_up(self):
self.transaction = None
self.x = None
def mouse_move(self, x, y):
if self.transaction is not None:
self.transaction.rollback()
self.transaction.modify(self.cut.id, lambda cut:
cut.move(int((x-self.x)/self.scrollbar.one_length_in_pixels)))
At this point, we only put actions into the rectangle map, and we can simplify the event handlers to this:
def left_mouse_down(self, x, y):
self.down_action = self.rectangle_map.get(x, y, Action())
self.down_action.left_mouse_down(x, y)
def right_mouse_down(self, x, y, gui):
self.down_action = self.rectangle_map.get(x, y, Action())
self.down_action.right_mouse_down(x, y, gui)
def mouse_move(self, x, y):
if self.down_action:
self.down_action.mouse_move(x, y)
else:
self.rectangle_map.get(x, y, Action()).mouse_move(x, y)
def mouse_up(self):
if self.down_action:
self.down_action.mouse_up()
self.down_action = None
$ ./make.py commit -m 'Timeline assumes there are Actions in rectangle map.'
...................................................
----------------------------------------------------------------------
Ran 51 tests in 3.381s
OK
[main 3c8e9b9] Timeline assumes there are Actions in rectangle map.
Date: Mon Jul 31 14:32:06 2023 +0200
2 files changed, 14 insertions(+), 64 deletions(-)
Everything seems to work fine. However, I notice that the committing of the transaction has disappeared.
This is not tested anywhere, missed my manual tests, and is pretty severe.
Let’s see if we can make the code a little more reliable. I write this test:
"""
>>> project = Project.new()
>>> transaction = project.new_transaction()
>>> transaction = project.new_transaction()
Traceback (most recent call last):
...
ValueError: transaction already in progress
"""
I make it pass, and I am now more confident that this error will show up when testing in the application.
$ ./make.py commit -m 'Ensure there can be only one transaction active at a time.'
....................................................
----------------------------------------------------------------------
Ran 52 tests in 3.402s
OK
[main 2b36bdb] Ensure there can be only one transaction active at a time.
2 files changed, 34 insertions(+), 7 deletions(-)
And sure enough, it does. The second time I try to drag a cut, I get the “transaction already in progress” error.
Nice!
The fix:
def mouse_up(self):
+ if self.transaction:
+ self.transaction.commit()
self.transaction = None
self.x = None
$ ./make.py commit -m 'Ensure CutAction transaction is commited at mouse_up.'
....................................................
----------------------------------------------------------------------
Ran 52 tests in 3.406s
OK
[main 5fd460d] Ensure CutAction transaction is commited at mouse_up.
1 file changed, 4 insertions(+), 1 deletion(-)
Normally you use a transaction like this:
with project.new_transaction() as transaction:
_ = transaction.add_text_clip("hello", length=30)
x = transaction.add_text_clip("world", length=35)
_ = transaction.add_text_clip("end", length=20)
_ = transaction.add_text_clip("end", length=20)
transaction.modify(x, lambda cut: cut.move(-10))
In that case a commit/rollback is guaranteed.
However, when dealing with mouse events, we can not use the context manager and instead have to deal with mouse events.
The new check that prevents multiple transactions ensures that everything stops working if we forget to close a transaction.
But I would like to come up with a nicer pattern for ensuring that transactions close.
I’ll add a TODO for it and maybe we can come up with a nicer solution later.
In order to satisfy Python’s import mechanism, I put Action
and MenuItem
in the rlvideolib.domain.cut
module.
They obviously don’t belong there.
Here is what the gui package looks like now:
rlvideolib/gui/
├── generic.py
├── gtk.py
├── __init__.py
└── testing.py
Previously Action
and MenuItem
were defined in generic
. That makes sense. But now we have a dependency on them from rlvideolib.domain.cut
. Should a domain object depend on GUI? Maybe that is ok.
I think what I’ll do is create another module inside the gui package called framework
. It will contain generic GUI elements that do not depend on GTK or our application.
$ ./make.py commit -m 'Move generic framework GUI code to new rlvideolib.gui.framework.'
....................................................
----------------------------------------------------------------------
Ran 52 tests in 3.393s
OK
[main b1a8f5d] Move generic framework GUI code to new rlvideolib.gui.framework.
5 files changed, 23 insertions(+), 19 deletions(-)
create mode 100644 rlvideolib/gui/framework.py
Back to this code where we started:
# TODO: add callback mechanism in rectangle map
x, y, w, h = (
area.x+x_start,
area.y,
x_end-x_start,
area.height
)
rect_x, rect_y = context.user_to_device(x, y)
rect_w, rect_h = context.user_to_device_distance(w, h)
self.rectangle_map.add(Rectangle(
x=int(rect_x),
y=int(rect_y),
width=int(rect_w),
height=int(rect_h)
), ScrollbarDragAction(self, self.scrollbar))
Ah, the TODO is actually done now.
$ ./make.py commit -m 'Remove completed TODO about callback mechanism for rectangle map.'
....................................................
----------------------------------------------------------------------
Ran 52 tests in 3.392s
OK
[main b757e3a] Remove completed TODO about callback mechanism for rectangle map.
1 file changed, 1 deletion(-)
We still haven’t made any progress on the exception problem though. But we have fixed design issues in related areas.
Let’s focus again on the exception.
I think the following pattern exists in all places where we add actions to the rectangle map:
rect_x, rect_y = context.user_to_device(x, y)
rect_w, rect_h = context.user_to_device_distance(w, h)
self.rectangle_map.add(Rectangle(
x=int(rect_x),
y=int(rect_y),
width=int(rect_w),
height=int(rect_h)
), ...)
What about if we add a method to RectangleMap
like this:
def add_from_context(self, x, y, w, h, context, item):
rect_x, rect_y = context.user_to_device(x, y)
rect_w, rect_h = context.user_to_device_distance(w, h)
self.add(Rectangle(
x=int(rect_x),
y=int(rect_y),
width=int(rect_w),
height=int(rect_h)
), item)
We can use that method to add both the scroll action and the scrub action.
However, the cut action looks slightly different:
rect_x, rect_y = context.user_to_device(rectangle.x, rectangle.y)
rect_w, rect_h = context.user_to_device_distance(rectangle.width, rectangle.height)
if int(rect_w) > 0 and int(rect_h) > 0:
rectangle_map.add(Rectangle(
x=int(rect_x),
y=int(rect_y),
width=int(rect_w),
height=int(rect_h)
), CutAction(project, self.get_source_cut(), scrollbar))
It actually has the check that we also need for the scrollbar. That is, we only add the rectangle to the map if it has a width and height.
Let’s add those checks to add_from_context
:
def add_from_context(self, x, y, w, h, context, item):
rect_x, rect_y = context.user_to_device(x, y)
rect_w, rect_h = context.user_to_device_distance(w, h)
if int(rect_w) > 0 and int(rect_h) > 0:
self.add(
Rectangle(
x=int(rect_x),
y=int(rect_y),
width=int(rect_w),
height=int(rect_h)
),
item
)
$ ./make.py commit -m 'Extract RectangleMap.add_from_context which does width/height checks.'
....................................................
----------------------------------------------------------------------
Ran 52 tests in 3.498s
OK
[main cd38e3e] Extract RectangleMap.add_from_context which does width/height checks.
3 files changed, 17 insertions(+), 25 deletions(-)
And this actually resolves the exception problem when I open my project.
I don’t have much experience doing evolutionary design. My feeling right now is that I need to spend much more time designing than what I am currently doing. I feel like I need to do at least 60% designing and only 40% adding new features. If you are reading this and have any experience with evolutionary design, feel free to share it with me. I should probably also re-read the chapters in James’ book to refresh my memory.
]]>