timeline

May 2, 2025 ∞

Newsletter April 2025: projects2

This month I’ve done a lot of programming. I ended up working more on my own code hosting platform. I call it projects2. Why two? Because it’s my second attempt at implementing this idea. Second attempt in recent times at least.

In my 2017 blog post, A new home for Timeline, I wrote

My suggested way forward is therefore to develop a new platform whose core features are registration free discussions and pull requests. In addition, it would need features common to many platforms like hosting of releases and a project web page.

In my previous attempt I focused on registration free discussions. This time, I decided to instead focus on creating the minimal possible software that allowed us to move away from SourceForge for Timeline and also get rid of the Jenkins instance that I run. That way, the infrastructure for running Timeline would not depend on proprietary systems or “complicated” third party software (Jenkins) which is overkill for our needs.

What follows is a demo of the current state of projects2.

Requirements

To use projects2 we need the following:

A machine running Fedora Linux that we have root access to
A domain that resolves to that machine
An SSL certificate for that domain

I use DNSimple to purchase domains and SSL certificates and Linode to provision Fedora servers.

Initial setup

projects2 is implemented as a single Python script, projects2.py, which is used to configure a single Fedora Linux machine to act as a code hosting platform.

We configure our server in a config.ini file. Let’s use this for the demo:

[Global]
InstanceName = projectsdemo
Domain = projectsdemo.rickardlindberg.me
Title = A demo site for projects2.
Description = This site showcases the project2 code hosting platform.

[User:admin]
DisplayName = Rickard
SshKeys = <my public ssh key>
Projects = *

[WildcardCertificate]
pem = <my ssl certificate>
key = <my ssl private key>

Next we run the bootstrap command, which should only be run once on a fresh Fedora install:

$ path/to/projects2.py bootstrap
ssh root login prompt
...
Ensuring user scm...
Ensuring passwordless sudo for scm...
Ensuring folder /home/scm/.ssh...
Ensuring authorized keys...
Ensuring folder /opt/projectsdemo...
Ensuring folder /opt/projectsdemo/web/artifacts...
Ensuring myself...
Ensuring myself api...
Ensuring config.ini...
Ensuring folder /home/scm/.ssh...
Ensuring authorized keys...
Ensuring SSH configured...
Ensuring sshd is restarted...
Ensuring hostname is projectsdemo.rickardlindberg.me...
Ensuring folder /opt/projectsdemo/web...
Ensuring folder /opt/projectsdemo/web/artifacts...
Ensuring folder /opt/projectsdemo/web/scm...
Ensuring folder /opt/projectsdemo/events...
Ensuring pem...
Ensuring key...
Setting up tools...
Setting up CI...
Setting up timezone...
Building /opt/projectsdemo/web/index.html...

We now have our Fedora server configured as a code hosting platform! But it looks a little empty at the moment:

The initial projectsdemo website which shows only
the instance title, descriptions, and empty placeholders for projects and
events.

Adding a project

Let’s add a project to our config.ini and also fix two typos that I made in the title and description:

[Global]
...
Title = A demo site for projects2
Description = This site showcases the projects2 code hosting platform.

[Project:demo]
Scm = git
Description = A demo project.

To apply these changes, we run the update command:

$ path/to/projects2.py update
Ensuring myself...
Ensuring myself api...
Ensuring config.ini...
Ensuring folder /home/scm/.ssh...
Ensuring authorized keys...
Ensuring SSH configured...
Ensuring hostname is projectsdemo.rickardlindberg.me...
Ensuring folder /opt/projectsdemo/web...
Ensuring folder /opt/projectsdemo/web/artifacts...
Ensuring folder /opt/projectsdemo/web/scm...
Ensuring folder /opt/projectsdemo/events...
Ensuring pem...
Ensuring key...
Setting up tools...
Setting up CI...
Setting up timezone...
Configuring project demo...
Ensuring pre-receive hook...
Ensuring post-update hook...
Building /opt/projectsdemo/web/demo.html...
Building /opt/projectsdemo/web/index.html...

And now the demo project appears on the website along with the fixed texts:

The projectsdemo website which now also shows the
projects placeholder filled in with the demo project.

Pushing code to the project

We have an empty git project setup up. Let’s push some code to it:

$ git init demo
$ cd demo
$ git branch -m main
$ vim README.md
$ git add README.md
$ git commit -m 'Add readme.'
$ git remote add origin scm@projectsdemo.rickardlindberg.me:demo.git
$ git push -u origin main
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Writing objects: 100% (3/3), 239 bytes | 239.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Hello from projects2 pre-receive hook!
remote: Ensuring /opt/projectsdemo/web/demo gone...
remote: Building /opt/projectsdemo/web/index.html...
remote: Building /opt/projectsdemo/web/demo.html...
To projectsdemo.rickardlindberg.me:demo.git
 * [new branch]      main -> main
branch 'main' set up to track 'origin/main'.

The website updates to show that we pushed some code to the demo project:

The projectsdemo website which now alos shows the
recent events placeholder filled in with one entry.

We can make more changes as usual and push them:

$ ...make changes...
$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 281 bytes | 281.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
remote: Hello from projects2 pre-receive hook!
remote: Ensuring /opt/projectsdemo/web/demo gone...
remote: Building /opt/projectsdemo/web/index.html...
remote: Building /opt/projectsdemo/web/demo.html...
To projectsdemo.rickardlindberg.me:demo.git
   b81aad4..6f7b10a  main -> main

And the diff will appear in the event log:

The projectsdemo website which now shows two
events for the demo project where the most recent event also includes a diff.

CI

You might have noticed that the push log includes lines like these:

remote: Building /opt/projectsdemo/web/index.html...
remote: Building /opt/projectsdemo/web/demo.html...

When we push code, projects2, intercepts that push to update the website. This mechanism is also used for Continuous Integration (CI).

In our repo, we can add files called Dockerfile*.ci. Those define Docker images in which the CI scripts are run. Let’s add Dockerfile.py312.ci to our demo project:

FROM python:3.12

CMD ["python3.12", "build.py"]

It says that the CI command to run is python3.12 build.py. Here is what build.py looks like:

#!/usr/bin/env python3

import json
import os
import sys

binary_path = "binary"
site_root = "html"

with open(binary_path, "w") as f:
    f.write("the compiled binary")

os.makedirs(site_root)
with open(os.path.join(site_root, "index.html"), "w") as f:
    f.write("hello from demo site")

with open("Dockerfile.py312.ci.files", "w") as f:
    f.write(json.dumps({
        "artifacts": [
            {
                "source": binary_path,
                "destination": "binary",
            },
        ],
        "site": site_root,
    }))

It simulates building a binary which looks like this:

$ cat binary
the compiled binary

And it builds a project website that looks like this:

$ cat html/index.html
hello from demo site

It tells projects2 about these artifacts via the file Dockerfile.py312.ci.files that looks like this:

{
  "artifacts": [
    {
      "source": "binary",
      "destination": "binary"
    }
  ],
  "site": "html"
}

When we push this change, we can see the following addition in the log:

remote: Building Dockerfile.py312.ci...
remote: Running Dockerfile.py312.ci...

The Dockerfile.py312.ci.files is parsed by projects2 and the binary file has been saved as an artifact along with the project website. The link to the binary is shown in the event log:

The projectsdemo website which shows a link to an
artifact in the most recent event.

And we can verify that it is correct like this:

$ curl https://projectsdemo.rickardlindberg.me/artifacts/demo/binary
the compiled binary

The project website is also published at <domain>/demo:

The demo project website which shows the
placeholder text.

This CI workflow is so nice. In my opinion, it is also much better than Jenkins'. It implements real CI. We just push code as we normally do. If the build brakes, the code will not get pushed and we can try again.

Summary

projects2 is now complete enough that I can start using it for my projects. For Timeline, we can replace all infrastructure from SourceForge and my Jenkins instance with projects2. Almost. We still use the mailing list from SourceForge. And maybe we will continue doing that. I’m not sure that registration free discussions and pull requests are as important to me as I thought. Mostly because there are not many contributors to my projects. But I might incorporate some kind of communication mechanism into projects2.

I couldn’t have implemented projects2 say 5 years ago. I wasn’t as good at Agile development and couldn’t have implemented the simplest thing that could possible work. I have many prior projects to thank for that. In particular I did the simplest thing that could possibly work. Here’s what happened. and Agile Game Development with Python and Pygame. I also couldn’t have come up with this solution for CI without prior reading, thinking, and prototyping solutions. I think the takeaway here is that you need to do many projects. From each project you will learn something that you can incorporate into your next project. Most projects will fail, but you will learn something. And eventually you will have a success. I have a feeling that projects2 might be a success. It is successful now in the sense that I actually use it. Time will tell for how long.

Sep 28, 2019 ∞