newsletter

May 2, 2025 ∞

Newsletter April 2025: projects2

This month I’ve done a lot of programming. I ended up working more on my own code hosting platform. I call it projects2. Why two? Because it’s my second attempt at implementing this idea. Second attempt in recent times at least.

In my 2017 blog post, A new home for Timeline, I wrote

My suggested way forward is therefore to develop a new platform whose core features are registration free discussions and pull requests. In addition, it would need features common to many platforms like hosting of releases and a project web page.

In my previous attempt I focused on registration free discussions. This time, I decided to instead focus on creating the minimal possible software that allowed us to move away from SourceForge for Timeline and also get rid of the Jenkins instance that I run. That way, the infrastructure for running Timeline would not depend on proprietary systems or “complicated” third party software (Jenkins) which is overkill for our needs.

What follows is a demo of the current state of projects2.

Requirements

To use projects2 we need the following:

A machine running Fedora Linux that we have root access to
A domain that resolves to that machine
An SSL certificate for that domain

I use DNSimple to purchase domains and SSL certificates and Linode to provision Fedora servers.

Initial setup

projects2 is implemented as a single Python script, projects2.py, which is used to configure a single Fedora Linux machine to act as a code hosting platform.

We configure our server in a config.ini file. Let’s use this for the demo:

[Global]
InstanceName = projectsdemo
Domain = projectsdemo.rickardlindberg.me
Title = A demo site for projects2.
Description = This site showcases the project2 code hosting platform.

[User:admin]
DisplayName = Rickard
SshKeys = <my public ssh key>
Projects = *

[WildcardCertificate]
pem = <my ssl certificate>
key = <my ssl private key>

Next we run the bootstrap command, which should only be run once on a fresh Fedora install:

$ path/to/projects2.py bootstrap
ssh root login prompt
...
Ensuring user scm...
Ensuring passwordless sudo for scm...
Ensuring folder /home/scm/.ssh...
Ensuring authorized keys...
Ensuring folder /opt/projectsdemo...
Ensuring folder /opt/projectsdemo/web/artifacts...
Ensuring myself...
Ensuring myself api...
Ensuring config.ini...
Ensuring folder /home/scm/.ssh...
Ensuring authorized keys...
Ensuring SSH configured...
Ensuring sshd is restarted...
Ensuring hostname is projectsdemo.rickardlindberg.me...
Ensuring folder /opt/projectsdemo/web...
Ensuring folder /opt/projectsdemo/web/artifacts...
Ensuring folder /opt/projectsdemo/web/scm...
Ensuring folder /opt/projectsdemo/events...
Ensuring pem...
Ensuring key...
Setting up tools...
Setting up CI...
Setting up timezone...
Building /opt/projectsdemo/web/index.html...

We now have our Fedora server configured as a code hosting platform! But it looks a little empty at the moment:

The initial projectsdemo website which shows only
the instance title, descriptions, and empty placeholders for projects and
events.

Adding a project

Let’s add a project to our config.ini and also fix two typos that I made in the title and description:

[Global]
...
Title = A demo site for projects2
Description = This site showcases the projects2 code hosting platform.

[Project:demo]
Scm = git
Description = A demo project.

To apply these changes, we run the update command:

$ path/to/projects2.py update
Ensuring myself...
Ensuring myself api...
Ensuring config.ini...
Ensuring folder /home/scm/.ssh...
Ensuring authorized keys...
Ensuring SSH configured...
Ensuring hostname is projectsdemo.rickardlindberg.me...
Ensuring folder /opt/projectsdemo/web...
Ensuring folder /opt/projectsdemo/web/artifacts...
Ensuring folder /opt/projectsdemo/web/scm...
Ensuring folder /opt/projectsdemo/events...
Ensuring pem...
Ensuring key...
Setting up tools...
Setting up CI...
Setting up timezone...
Configuring project demo...
Ensuring pre-receive hook...
Ensuring post-update hook...
Building /opt/projectsdemo/web/demo.html...
Building /opt/projectsdemo/web/index.html...

And now the demo project appears on the website along with the fixed texts:

The projectsdemo website which now also shows the
projects placeholder filled in with the demo project.

Pushing code to the project

We have an empty git project setup up. Let’s push some code to it:

$ git init demo
$ cd demo
$ git branch -m main
$ vim README.md
$ git add README.md
$ git commit -m 'Add readme.'
$ git remote add origin scm@projectsdemo.rickardlindberg.me:demo.git
$ git push -u origin main
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 239 bytes | 239.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Hello from projects2 pre-receive hook!
remote: Ensuring /opt/projectsdemo/web/demo gone...
remote: Building /opt/projectsdemo/web/index.html...
remote: Building /opt/projectsdemo/web/demo.html...
To projectsdemo.rickardlindberg.me:demo.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

The website updates to show that we pushed some code to the demo project:

The projectsdemo website which now alos shows the
recent events placeholder filled in with one entry.

We can make more changes as usual and push them:

$ ...make changes...
$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 281 bytes | 281.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Hello from projects2 pre-receive hook!
remote: Ensuring /opt/projectsdemo/web/demo gone...
remote: Building /opt/projectsdemo/web/index.html...
remote: Building /opt/projectsdemo/web/demo.html...
To projectsdemo.rickardlindberg.me:demo.git
   b81aad4..6f7b10a  main -> main

And the diff will appear in the event log:

The projectsdemo website which now shows two
events for the demo project where the most recent event also includes a diff.

CI

You might have noticed that the push log includes lines like these:

remote: Building /opt/projectsdemo/web/index.html...
remote: Building /opt/projectsdemo/web/demo.html...

When we push code, projects2, intercepts that push to update the website. This mechanism is also used for Continuous Integration (CI).

In our repo, we can add files called Dockerfile*.ci. Those define Docker images in which the CI scripts are run. Let’s add Dockerfile.py312.ci to our demo project:

FROM python:3.12

CMD ["python3.12", "build.py"]

It says that the CI command to run is python3.12 build.py. Here is what build.py looks like:

#!/usr/bin/env python3

import json
import os
import sys

binary_path = "binary"
site_root = "html"

with open(binary_path, "w") as f:
    f.write("the compiled binary")

os.makedirs(site_root)
with open(os.path.join(site_root, "index.html"), "w") as f:
    f.write("hello from demo site")

with open("Dockerfile.py312.ci.files", "w") as f:
    f.write(json.dumps({
        "artifacts": [
            {
                "source": binary_path,
                "destination": "binary",
            },
        ],
        "site": site_root,
    }))

It simulates building a binary which looks like this:

$ cat binary
the compiled binary

And it builds a project website that looks like this:

$ cat html/index.html
hello from demo site

It tells projects2 about these artifacts via the file Dockerfile.py312.ci.files that looks like this:

{
  "artifacts": [
    {
      "source": "binary",
      "destination": "binary"
    }
  ],
  "site": "html"
}

When we push this change, we can see the following addition in the log:

remote: Building Dockerfile.py312.ci...
remote: Running Dockerfile.py312.ci...

The Dockerfile.py312.ci.files is parsed by projects2 and the binary file has been saved as an artifact along with the project website. The link to the binary is shown in the event log:

The projectsdemo website which shows a link to an
artifact in the most recent event.

And we can verify that it is correct like this:

$ curl https://projectsdemo.rickardlindberg.me/artifacts/demo/binary
the compiled binary

The project website is also published at <domain>/demo:

The demo project website which shows the
placeholder text.

This CI workflow is so nice. In my opinion, it is also much better than Jenkins'. It implements real CI. We just push code as we normally do. If the build brakes, the code will not get pushed and we can try again.

Summary

projects2 is now complete enough that I can start using it for my projects. For Timeline, we can replace all infrastructure from SourceForge and my Jenkins instance with projects2. Almost. We still use the mailing list from SourceForge. And maybe we will continue doing that. I’m not sure that registration free discussions and pull requests are as important to me as I thought. Mostly because there are not many contributors to my projects. But I might incorporate some kind of communication mechanism into projects2.

I couldn’t have implemented projects2 say 5 years ago. I wasn’t as good at Agile development and couldn’t have implemented the simplest thing that could possible work. I have many prior projects to thank for that. In particular I did the simplest thing that could possibly work. Here’s what happened. and Agile Game Development with Python and Pygame. I also couldn’t have come up with this solution for CI without prior reading, thinking, and prototyping solutions. I think the takeaway here is that you need to do many projects. From each project you will learn something that you can incorporate into your next project. Most projects will fail, but you will learn something. And eventually you will have a success. I have a feeling that projects2 might be a success. It is successful now in the sense that I actually use it. Time will tell for how long.

Apr 2, 2025 ∞

Newsletter March 2025: Snowboarding

This month I have done nothing related to programming in my spare time. Partly because it is snowboard season.

Me standing on a snowboard in the mountains of
Åre.

I’m most interested in continuing work on my own code hosting platform that will host my projects. We’ll see if I have the time and motivation next month. Or if something completely different catches my interest.

Mar 19, 2025 ∞

Newsletter February 2025: A New Code Hosting Platform?

When I came across XXIIVV in last month, I immediately got interested in the idea that in order to be able to run our software many, many years from now, we need to target a small virtual machine that we can re-implement in a weekend. Their virtual machine is called Uxn:

Uxn is the virtual machine powering the Hundred Rabbits software.

I spent some time reading about Uxn and trying it out. I also started documenting my research in a blog post. Then I got distracted by other things.

One thing that distracted me was starting working on my own code hosting platform. It will be tailored to my specific needs and my projects. I wrote about some of my needs back in 2017 in A new home for Timeline and I also have new ideas for what I want today.

I made pretty good progress, and I think I can soon start using it for some of my projects.

However, as with most of my hobby projects, I got distracted. This time by a snowboarding vacation. We’ll see what my interest decides to continue working on when I have the time and motivation.

Feb 1, 2025 ∞

Newsletter January 2025: Inspired and Motivated by New Laptop and Reading

New Laptop

I got a new laptop this month. It was almost 10 years since I bought my previous one. I mainly needed a new one to be able to smoothly browse certain websites and for better performance when editing videos.

When installing the latest version of Fedora on it, I took the time to clean up my dotfiles. Since I jumped quite many Fedora versions, my often used tool rlselect had stopped working. I figured out the problem and documented the fix in Replacing Ctrl-R in Bash without TIOCSTI.

That blog post came naturally to me. I was trying to find a solution to a problem. I found other people having the same problem. When I found a solution, I felt the need to share it to contribute to the discussion and hopefully help someone else. I even wrote a custom version of the blog post tailored to an issue on GitHub.

Bootstrapping

I came across Bootstrappable Builds. Bootstrapping is an interesting problem that I’ve mainly come across in my work on RLMeta. They write that

To gain trust in our computing platforms, we need to be able to tell how each part was produced from source.

I started thinking how this would apply to RLMeta. The RLMeta “binary” is a Python file. So it needs Python to bootstrap itself. I’m not sure if that qualifies as a problem according to Bootstrappable.

The “binary” is not really human readable, so it is not feasible to inspect it. On the other hand, the source code says exactly how it is produced, and we can verify that it produces itself.

One way to figure out if this is a problem or not is to see if it is vulnerable to the “Trusting Trust” attack. The article Reflections on Rusting Trust talks about how to make this attack in the Rust compiler. I don’t fully understand it, but it could be interesting to try with RLMeta.

XXIIVV

I came across XXIIVV. There are so many things in there that interest me.

One of those things is the idea that in order to be able to run our software many, many years from now, we need to target a small virtual machine that we can re-implement in a weekend. You can find more on this in devlog and the transcript of the talk Weathering Software Winter.

One thing that cause our software to break is when the things that it depend on change or go away. For that reason, I’m reluctant to pull in third party dependencies when building software. But what if the language our software is written in disappears? That is less likely than third party dependencies changing, but there is still a risk. Especially in the long run.

But if we can not depend on third party software or languages, we have to implement the whole software stack ourselves. That is a lot of work. There is probably a balance where the trade-off of depending on something is worth it. And that balance differs depending on our goals with the software. However, my feeling is that many things that we pull in third party dependencies for, we can quite easily implement ourselves. And get rid of the bloat of the 80% of the third party dependency that we don’t use. In addition to getting rid of bloat, it also increases understandability. We don’t need to figure out how a third party dependency work, we just need to figure out how a much smaller set of our code works.

Another area where preserving software is of interest to me is my website. About half a year ago, I moved to Micro.blog. Some of my posts now only exists there. The platform gives me some things that I like such as ease of posting and interaction with others. But what if Micro.blog goes away? What happens to my words? I think I need to go back to having the source code for my website in a git repo. Then I should be able to compile my website for different targets. One for publishing online. It might be an export to Micro.blog so that I can continue to use some of its features. But it might also be a pdf export. That way I can print the pdf and have my whole website preserved as a physical book in my bookshelf. That will most likely live for much longer than any technology. And of course, compiling my website should depend on as few dependencies as possible. Perhaps even target a small VM as XXIIVV does it?

A Note on Reading

I was able to read and write about the topics above partly because of a realization that I had earlier this month:

Today’s realization is that you can get important things done by consistently working on them for 15 minutes at the start of every day.

By doing it at the start of the day, you ensure that it gets done. And the rest of the day you don’t need to be stressed about not working on your important thing, because you already have.

I want to read more. But it is easier to just scroll through my feeds and read headlines. What I’ve tried now is to bookmark things that look interesting. Then I spend some time in the mornings to carefully read those pieces. It’s been a quite positive experience for me. I’ve also used that trick to get more boring tasks done. It might not work if you are a night person though.

Dec 8, 2024 ∞

Newsletter November 2024: A New Project

Compared to last month, this month I did some programming in my spare time. I had fewer commitments, and my mind started thinking about various programming projects. We also got the first snowfall of the season and I got to enjoy a run in it:

Me running in a snow-covered landscape with the
sun setting in the background.

A New Code Editor

The programming project that I started working on is code editor that is a mix of a text editor and a structured editor. It is all text, but parsers and pretty printers allow you to work with a tree structure and not think too much about syntax. The code is available on GitHub, and here is what it looks like when editing a JSON document:

Screenshot of rledit editing a JSON document with a
selection.

I got the idea for this project when trying out Black. Black automatically formats Python code for you so that you don’t have to think about it. I’ve been interested in structured editors for some time, but my feeling is that they are not user friendly because they limit what you can type. Then I came up with this idea of an editor that constantly parses what you type. If the parse is successful, it pretty prints it for you and provides you with edit operations on the AST. But it is all still just text, so you can type whatever. In the worst case, the parse fails and you have to fix it manually.

So far, it looks quite promising. And most importantly, I’m having fun experimenting. The most likely scenario is that the project will not be a success, but I will learn something and have fun doing so. But you never know. One day, one of these projects just might turn into something that is invaluable.

TDD

This month I also watched a presentation by Kent Beck called TDD: Theme & Variations. For me, it was a nice refresher on the origins of TDD.

One thing that I appreciate with TDD, that I partially had forgotten, is how it reduces anxiety. Kent reminded me of it in the presentation. When all tests are passing and you can’t think of any more tests to write, you are done, and you know that it works. That reduces anxiety a lot.

Nov 3, 2024 ∞

Newsletter October 2024: Primitive Obsession?

Normally I do something related to programming in my spare time every month. I read something that I find interesting and want to share. Or I have some thought related to programming that I want to share. This is the first month since I started these monthly updates in June 2019 that I’ve got nothing of that. I’ve been occupied with other things, and I’ve also done quite a bit of programming at work. Perhaps that has satisfied my interest for programming.

One thing that I’ve done a lot at work this month is wrapping simple data structures in classes. Instead of passing around lists and dictionaries, I’ve created classes holding that data and only serialized it at the edges of the application. Every time I have done this, I wish I had done it sooner. It’s so good. And in nine times out of ten, those classes have attracted some functionality that fits perfectly. They have provided one place to put functionality instead of scattering it throughout the codebase.

What am I talking about? Let me give an example.

Imagine that we talk to a user service that has an API something like this:

service.get_all_users() -> ["user1", "user2"]
service.set_users_in_group("group1", ["user1", "user2"]) -> OK

The API works with users represented as list of strings. Now we want to write a function that applies some domain logic to assign users to groups. It is so easy and tempting to write something like this:

def update_r_users(service)
    r_users = []
    for user in service.get_all_users():
        if "r" in user:
            r_users.append(user)
    service.set_users_in_group("users_with_r_in_name", r_users)

One problem with this code is that it mixes calls to the user service with domain logic, so is is difficult to test domain logic without invoking the service. It’s also more difficult to reason about.

What if we instead did this:

def update_r_users(service)
    service.set_users_in_group(
        "users_with_r_in_name",
        Users.from_service(service.get_all_users()).filter_name("r").serialize()
    )

class Users:

    @classmethod
    def from_service(cls, users):
        return cls(users)

    def __init__(self, users):
        self.users = users

    def filter_name(self, text):
        return Users([
            user
            for user in self.users
            if text in user
        ])

    def serialize(self):
        return self.users

This is what I mean by wrapping simple data structures in classes. Why is this better?

First of all, I think update_r_users now reads a lot better. The filtering logic is no longer mixed with the calls to the service.

This comes at the cost of writing the Users class which is quite long for the relative functionality that it provides. In the beginning, I often find it hard to motivate myself to write these classes. It feels like a lots of boilerplate code for not much benefit. However, I often find that these sort of classes attract functionality, at which point they start to become more useful.

Another thing that they do is encapsulate the data format from the user service. Say that the API of the service changes. There is now more information about users:

service.get_all_users() -> [{"name": "user1", "age": 21}, {"name": "user2", "age": 43}]
service.set_users_in_group("group1", ["user1", "user2"]) -> OK

We can update the User class accordingly:

class Users:

    @classmethod
    def from_service(cls, users):
        return cls(users)

    def __init__(self, users):
        self.users = users

    def filter_name(self, text):
        return Users([
            user
            for user in self.users
            if text in user["name"]
        ])

    def serialize(self):
        return [user["name"] for user in self.users]

The update_r_users stays the same. We control the API of Users. Our application can safely depend on it. And we can change the internals.

I couldn’t find a name for this sort of pattern. My first though was that it was a way to avoid primitive obsession. And it is. But it feels like more than that. Bill suggested that it might be just “good old encapsulation”. Dan said that it depends on the context and that it could be an anti-corruption layer in DDD terms. What would you call it?

Anyway, I’ve been doing this sort of thing a lot this month, and I thought it was worth sharing.

Oct 14, 2024 ∞

Newsletter September 2024: Bash Redirects and Reading

This month I read How to Use a Zettelkasten to Write Stories Packed with Emotion. It inspired me to write something using that technique. The topic that came to mind was Bash redirects. It resulted in the blog post Bash Redirects Explained.

I also started reading How Children Learn by John Holt. He says that children are naturally interested in exploring the world. I started thinking about what environments kill exploration. I though that fear of doing something wrong will discourage exploration. Then I made a parallel to a common practice in software development: pull requests. I thought that pull requests discourage experiments because changes can only propagate after approval, and the idea behind pull requests is to only approve “good” changes. First of all, the learning opportunities of mistakes are gone. Second of all, you might loose interest in experimenting because you are afraid of making a mistake. You build a culture of discouraging making mistakes. I though that a better approach is to focus on making the cost of mistakes small so that we can do lots of them and learn from them.

I also read Debugging Teams by Ben Collins-Sussman and Brian Fitzpatrick. They talk about the myth of the genius programmer and that all great things are built by teams. I believe that is true, but sometimes I have a hard time to acknowledge it because I also enjoy working by myself.

Sep 1, 2024 ∞

Newsletter August 2024: Smart Notes and Blogging

In August I had my last week of summer vacation. I read a book that made me want to improve my note taking tool, I did some programming on hobby projects, and I migrated my blog to a new platform. All while doing many runs.

Collage of activities in August.

A System for Writing

I read A System for Writing by Bob Doto. This review is what made me buy the book:

When I first tried to get my head around Zettelkasten, I consulted Sönke Ahrens’ extremely useful book How to Take Smart Notes. This was—and still is—seen as the book to read on the subject. I predict A System for Writing will replace it.

I liked Sönke’s book, and I was curious to learn more about this topic from a source that seemed credible.

I got some new insights about the Zettelkasten method. In particular the idea that it is important to articulate how two ideas relate. I’ve previously mostly connected ideas without context. The connection was obvious when I made it, but most likely less obvious when I come back to it.

I tried to research this a bit more and found the article Backlinking Is Not Very Useful – Often Even Harmful:

A good link context explains what you can expect if you follow the link. But it can also explain the nature of the relationship between both notes.

I wanted to try this out.

Smart Notes

I use my Smart Notes app to take notes using the Zettelkasten method. When I studied the method more, I got some new ideas that I implemented:

I configured different note types to show in different colors. Zettelkasten has a few different kinds of notes, and quickly being able to see, by color, which it is, has been helpful.
I added the ability to change the order of notes. This was needed to be able to create structure notes with links in specific order.
I added the ability to give links a text description. This is what allowed me to experiment with articulating how two ideas relate to each other. I wrote more about the evolution of this feature in Linking (and how it has evolved) in Smart Notes.

One-File Programs

I continued to work on One-File Programs.

I worked on an engine app that can automatically reload other apps when they change. This gives faster feedback, especially when working on graphical applications. It is similar to an approach I’ve written about previously in How to get fast feedback on graphical code?

I also worked on the no scrollbars app. I wanted to see if we can get rid of scrollbars in some GUI elements. The idea is to make items larger around the area of the mouse so that they can more easily get selected. Here is one screenshot:

Screenshot of no scrollbars experiment.

I want do continue work with this repo, but the rest of the time this month, I spent on my blog.

Micro.blog

I found out about Micro.blog and moved my blog to it.

I documented how I did a redirect to my old archived site for pages that I have not yet migrated to the new blog in Poor man’s redirect in a static site.
I started a “photo blog” documenting my runs and how my life improves from them.

Some reflections on Micro.blog:

Micro.blog makes it easy to create a website that feels like a home.

I can share and develop my interests. I can structure content with categories and pages. Others can come have a look. Comment if they wish. And you can see conversations that I’ve had with visitors.
It makes blogging fun. It is easy to publish and you can get comments from the community on your posts.

Aug 1, 2024 ∞