∞

I needed to submit some heic photos to a service that only accepted jpg. I didn’t know about the heic format, but a little searching gave me a solution:

$ heif-convert
bash: heif-convert: command not found...
Install package 'libheif' to provide command 'heif-convert'? [N/y] y
...
$ find . -iname '*.heic' -exec heif-convert -q 100 {} {}.jpg \;
∞

Today was the first day of snow this season. Not much. I’m looking forward to many more runs on a white trail.

Me running on a trail with a little snow.
∞

I was researching how to run Black (and possibly other formatters) from Vim and found Ergonomic mappings for code formatting in Vim. It was very helpful.

∞

How would you improve this code?

def update_r_users(service)
    r_users = []
    for user in service.get_all_users():
        if "r" in user:
            r_users.append(user)
    service.set_users_in_group("users_with_r_in_name", r_users)

Find out what I did it in my latest newsletter.

∞

Today I learned about the Rison data serialization format. I wrote a function to convert a Python value to Rison format. It was an elegant recursive function with partial support for the format.

∞

I’ve used testing without mocks quite extensively now. I’ve also used it in a work project for more than a year. My experience is that it’s the best testing strategy that I’ve ever used. I’ve never felt more confident that my code works. I refactor code without fear of it breaking. It’s so good.

∞

It’s getting dark. It gives variation to the running.

Me running in the dark.
∞

Various things have kept me from running for a while. Today I had enough. I just had to go for a short run. It was the first run with warmer clothes. The weather was nice. I reclaimed some energy.

Me running.
∞

Pull requests discourage experiments because changes can only propagate after approval. The idea behind PRs is to only approve “good” changes.

First, the learning opportunities of mistakes are gone. Second, you might loose interest in experimenting because you are afraid of making mistakes.

∞

Today I just needed to run. I had not run since I hurt my achilles tendon almost a month ago. I wanted to see if it still hurt. I felt something, but not too much. I think I still need to take it easy with running, but man it felt good moving again.

Me running.
∞

If you want to know how to implement a Bash-like shell, with support for redirects, in only 31 lines of Python, you should check out my latest blog post Bash Redirects Explained.

∞

Do you know the difference between the following Bash commands?

program 2>&1 >/tmp/log.txt
program >/tmp/log.txt 2>&1

If not, you might be interested in my latest blog post Bash Redirects Explained.

∞

Bash Redirects Explained

I thought I knew how Bash redirects worked.

If I wanted to redirect the output of a command to a file, I’d type this:

program > /tmp/log.txt

If I wanted to pipe both stdout and stderr to a text editor for further processing, I’d type this:

program 2>&1 | vim -

I knew that 2>&1 meant redirect stderr to stdout making it appear on stdout as well.

I knew certain patterns for certain situations. But when I encountered situations where I had not learned a pattern, I was lost. For example, I could not explain the difference between

program 2>&1 >/tmp/log.txt

and

program >/tmp/log.txt 2>&1

And I got scared when I saw something like this:

program < input.txt > output.txt 2>&1

Have you also been there? What did you do?

I would search the Internet for a pattern that matched the use case, or just try different alternatives and notice how they behaved.

I did this until one day when I learned a mental model for how Bash redirects work. Now I no longer need to rely on patterns. I can easily parse any situation and use any combination of redirects for my purposes.

The rest of this article explains this mental model.

The Standard Streams

A process has three standard streams attached to it:

  • stdin (0)
  • stdout (1)
  • stderr (2)

Diagram of the three streams of a process.

When we start a program from the terminal, Bash sets up the standard streams as follows:

  • stdin: terminal/keyboard
  • stdout: terminal
  • stderr: terminal

What redirects do is to modify what the standard streams point to before the program starts executing.

  • < means modify stdin.
  • > means modify stdout.
  • 2> means modify stderr.

That is the mental model: redirects modify standard streams before program execution.

Let’s evaluate a few examples using this mental model to see how it works.

Logcat Utility

To be able to show what happens in different examples, we have a utility program, logcat.py, that makes use of all three streams. It reads text from stdin, logs the arguments and the length of the text to stderr, and writes the text to stdout. It looks like this:

#!/usr/bin/env python

import sys

text = sys.stdin.read()

sys.stderr.write(f"Args: {(sys.argv[1:])}\n")
sys.stderr.write(f"Read {len(text)} characters.\n")

sys.stdout.write(text)

Example: No Redirect

Let’s start with an example without redirects to see the operation of logcat.py:

$ ./logcat.py ignored arguments

Before logcat.py starts executing, Bash sets up the standard streams as follows:

  • stdin: terminal/keyboard
  • stdout: terminal
  • stderr: terminal

When execution starts, logcat.py waits for input. If we type hello in the terminal (followed by a return and ctrl+d), the following is printed to the terminal:

Args: ['ignored', 'arguments']
Read 6 characters.
hello

We can see that it read our input from the terminal/keyboard and wrote the log messages along with our input to the terminal as well.

Example: Redirect Stdin

Now let’s modify stdin to instead of the terminal/keyboard be the logcat.py source code:

$ ./logcat.py ignored arguments <logcat.py

This instructs Bash to modify stdin to point to the file logcat.py.

Before logcat.py starts executing, Bash sets up the standard streams as follows:

  • stdin: logcat.py (opened in read mode)
  • stdout: terminal
  • stderr: terminal

When execution starts, the following is printed to the terminal:

Args: ['ignored', 'arguments']
Read 182 characters.
#!/usr/bin/env python

import sys

text = sys.stdin.read()

sys.stderr.write(f"Args: {(sys.argv[1:])}\n")
sys.stderr.write(f"Read {len(text)} characters.\n")

sys.stdout.write(text)

We can see that the redirect operation is stripped from the arguments. Only Bash sees it and does not pass it along to the program. Furthermore we can see that the logcat.py source code is printed to the terminal.

Example: Redirect Stdin and Stdout

Let’s say we’re only interested in the log messages, and want to throw away stdout:

$ ./logcat.py ignored arguments <logcat.py >/dev/null

This instructs Bash to modify stdin to point to the file logcat.py and to modify stdout to point to the file /dev/null.

Before logcat.py starts executing, Bash sets up the standard streams as follows:

  • stdin: logcat.py (opened in read mode)
  • stdout: /dev/null (opened in write mode)
  • stderr: terminal

When execution starts, the following is printed to the terminal:

Args: ['ignored', 'arguments']
Read 182 characters.

We can see that the redirect operations are all stripped from the arguments and the source code has been written to /dev/null and is thus not shown in the terminal.

Extended Mental Model

Let’s extended the mental model to clarify how Bash operates.

When Bash parses a command, it divides it into two parts: the arguments and the redirects. Before it starts executing the program with the arguments, it goes through the redirects, in order, and configures the standard streams before execution.

Example: Redirect All Streams

Let’s see how we can interpret a more complex command using the extended mental model:

$ ./logcat.py <logcat.py is the >out.txt best 2>&1 thing

If we split this into arguments and redirects, we get this:

  • Arguments: ./logcat.py, is, the, best, thing
  • Redirects: <logcat.py, >out.txt, 2>&1

Now, let’s evaluate the redirects in order. The state of the standard streams at start is this:

  • stdin: terminal/keyboard
  • stdout: terminal
  • stderr: terminal

Then we evaluate <logcat.py and get this:

  • stdin: logcat.py (opened in read mode)
  • stdout: terminal
  • stderr: terminal

Then we evaluate >out.txt and get this:

  • stdin: logcat.py (opened in read mode)
  • stdout: out.txt (opened in write mode)
  • stderr: terminal

Then we evaluate 2>&1, which means modify stderr (2>) to be whatever stdout points to (&1), and get this:

  • stdin: logcat.py (opened in read mode)
  • stdout: out.txt (opened in write mode)
  • stderr: out.txt (opened in write mode)

After the standard streams have been set up, execution of ./logcat.py is the best thing starts. Nothing appears on the terminal since all output has been redirected to out.txt:

$ cat out.txt
Args: ['is', 'the', 'best', 'thing']
Read 182 characters.
#!/usr/bin/env python

import sys

text = sys.stdin.read()

sys.stderr.write(f"Args: {(sys.argv[1:])}\n")
sys.stderr.write(f"Read {len(text)} characters.\n")

sys.stdout.write(text)

Mini Shell

I created a mini version of a shell to demonstrate how straight forward it is to implement redirects with POSIX system calls. It works exactly as the extended mental model, and because it is running software, it fills in some more details of the model. I would guess that Bash does something similar even though I haven’t read its source code.

First off, here is a demo that shows how the mini shell can replicate the complex example from above:

$ ./minishell.py
~~?~~> ./logcat.py <logcat.py is the >out.txt best 2>&1 thing
~~0~~> cat out.txt
Args: ['is', 'the', 'best', 'thing']
Read 182 characters.
#!/usr/bin/env python

import sys

text = sys.stdin.read()

sys.stderr.write(f"Args: {(sys.argv[1:])}\n")
sys.stderr.write(f"Read {len(text)} characters.\n")

sys.stdout.write(text)

And here is the implementation in only 31 lines of Python:

#!/usr/bin/env python

import os
import sys

STDIN  = 0
STDOUT = 1
STDERR = 2

statuscode = "?"
while True:
    sys.stdout.write(f"~~{statuscode}~~> ")
    sys.stdout.flush()
    command = input()
    pid = os.fork()
    if pid == 0:
        args = []
        for part in command.split(" "):
            if part.startswith("<"):
                os.dup2(os.open(part[1:], os.O_RDONLY), STDIN)
            elif part.startswith(">"):
                os.dup2(os.open(part[1:], os.O_WRONLY|os.O_CREAT, 0o644), STDOUT)
            elif part == "2>&1":
                os.dup2(STDOUT, STDERR)
            elif part.startswith("2>"):
                os.dup2(os.open(part[2:], os.O_WRONLY|os.O_CREAT, 0o644), STDERR)
            else:
                args.append(part)
        os.execvp(args[0], args)
    else:
        _, statuscode = os.waitpid(pid, 0)

To understand how this works, you need some knowledge of the POSIX system calls fork, waitpid, open, dup2, and execvp. But even if you don’t understand the specifics, I think this codified model can help in understanding how Bash operates. Let’s look at an example.

Example: Duplicated Files

Let’s see if we can explain the difference between the following commands using the mini shell for the model:

$ ./logcat.py <logcat.py >out.txt 2>out.txt
$ ./logcat.py <logcat.py >out.txt 2>&1

At a first glance, it looks like both commands redirect both stdout and stderr to the out.txt file. But if we evaluate it like mini shell does, we see that the first example will open the file twice (two calls to os.open creating two file handles), whereas the second example will open the file only once and then duplicate the file handle for stderr.

When two file handles are created, writes to the two streams will attempt to write to the same location in the file and they will overwrite each other. Furthermore, buffering might alter in which order writes happen, so it is not clear what will actually end up in the file. So to make sure all output is captured in the file, the second example should be used where the file is only opened once.

Conclusion

There is still more to Bash redirects than what I have explained here. But this mental model (along with its extended versions) have helped me reason about Bash redirects. I hope it will do the same for you.

∞

I just implemented a small shell in 29 lines of Python that has support for redirects:

$ ./minishell.py 
~~?~~> echo hello
hello
~~0~~> wc -l minishell.py
29 minishell.py
~~0~~> wc -l <minishell.py >report.txt
~~0~~> cat report.txt
29
∞

Linking (and how it has evolved) in Smart Notes

I created Smart Notes to be a digital note taking application that follows principles from the Zettelkasten method, as explained in Sönke Ahren’s book How to Take Smart Notes.

Two fundamental aspects of the Zettelkasten method are notes and links between them.

In a physical Zettelkasten, notes are written on index cards (or some similar format), given unique, hierarchical ids, and stored sequentially in a slip-box:

Illustration of physical Zettelkasten.

Sönke writes that you should add a new note to the slip-box directly behind the note you are referring to. That creates one kind of link. He also writes that you can add it behind multiple notes by by referring to its unique in other notes.

Here is how I realized this aspect of the Zettelkasten method in Smart Notes:

Screenshot of Smart Notes showing how notes and links are represented.

Some design decisions and my reasoning about them:

  • Notes are visualized as a physical index cards. That restricts the amount of text that you can write on them. I felt like a problem with a digital approach might be that you write too much on your notes because there is unlimited space. I was also interested in mimicking the feeling of working with physical index cards on a table.

  • Links are visualized as strings between notes. You can link however you want. In the above example, the note related to both notes are added behind both the first note and the note that is related to the first note. (The note is shown twice, but it is the same note.)

  • Note ids are not shown in the UI. In a physical Zettelkasten, note ids are important to be able to find a note when following a link. (If a note says “see note X” you need to be able to locate X in the slip-box.) But in Smart Notes the link is right there and shows the note that it links to.

Trains of thought and branches

Another aspect of the Zettelkasten method is trains of though. Notes related to each other can form a train of thought, and reviewing that train of though is useful. In a physical Zettelkasten related notes are stored close to each other, and following a train of though can be done by flipping through the cards in an area.

I was thinking about how this aspect of Zettelkasten could be realized in Smart Notes.

I read more about how notes are stored and numbered in a physical Zettelkasten. There seems to be two cases. Either the note is related to an existing note, in which case it is put right behind it, or the note is not related to any existing notes, in which case it is put at the end.

Furthermore, if a note is related to an existing note, a distinction seems to be made if the new note continues the train of though of the previous note or if it branches off. With the Zettelkasten numbering system, it might be possible to distinguish these different cases. In Introduction to the Zettelkasten Method, Sascha writes:

The very first note is assigned the number 1. If you add a second note that is not related to the first note, it is assigned the number 2. But if you want to continue the first note, or inject something into its content, comment on it, or something along those lines, you branch off. That new note would get assigned the number 1a. If you continue with this new note, you would go on with 1b. If you then want to comment on the note 1a, you would create a note with the address 1a1. So, in short, whenever you continue a train of thought, you increment the last position in the address, be it number or a character from the alphabet. And when you want to expand, intersperse, or comment on a note, you take its address and append a new character.

Smart Note lacks any visible note ids. So I was thinking that I might be missing out on something. What types of hierarchies are created with the numbering scheme in Zettelkasten? Can I already replicate those hierarchies in Smart Notes?

Folgezettel

There seems to be a debate going on whether the numbering scheme in Zettelkasten, sometimes called “folgezettel” is useful or not.

In No, Luhmann was not about Folgezettel, Sascha argues that the kind of links created with folgezettel are weak. We know that there is a relationship between two notes, but we don’t know what that relationship is. Sascha further acknowledges that the folgezettel technique is probably more practically useful in a physical Zettelkasten, but in a digital world, creating annotated links is a better solution. (Creating annotated links is certainly possible in a physical Zettelkasten, but browsing those links would be tedious because the note in the link might be in a totally different place in the slip-box.)

In Folgezettel is More than Mechanism, Bob argues that the folgezettel technique is useful even in a digital world. One point is that it allows you to get a bird’s eye view of the topics and threads that are developing in your Zettelkasten. Another point is that it forces you to think about note relationships when you add notes to the slip-box.

I’m starting to think that the mechanism of folgezettel is just about creating hierarchies of notes. That’s certainly possible already in Smart Notes. Perhaps the numbering scheme is not essential to the Zettelkasten method? Perhaps I should explore annotated links more instead?

Sascha’s point about annotated links and their importance got me thinking how to apply it to Smart Notes. (In Backlinking Is Not Very Useful – Often Even Harmful, Sascha talks more about the importance of adding context to links.)

Here is one thought I had about it early on:

An unclear link between two notes can be clarified by inserting a note in between that explains the reason for the link. A -> B can become A -> explanation -> B.

Many years later, I tried this out. It looked like this:

Screenshot of Smart Notes showing the use of a link note.

I found that the “explanation note” took up much space and made it more difficult to navigate the note network. That’s when I decided that Smart Notes needed the ability to annotate links instead. Here is how it turned out:

Screenshot of Smart Notes showing link annotations.

I’ve only experimented with this a little, but I find that navigation is easier. My hope is that short “linking phrases” (inspired by concept mapping) will be enough to annotate links.

Closing thoughts

With annotated links added, I think Smart Notes is pretty flexible. First of all you can create arbitrary tree-like structures that can mimic the structure you get from folgezettel without having to deal with a complicated numbering scheme. That structure can give you a bird’s eye view that Bob talks about. If you don’t want to emphasize a tree structure, you can use annotated links to link arbitrarily and make the context of the connections clear.

I try to make Smart Notes a digital note taking application that follows principles from the Zettelkasten method. Sometimes it is easy to focus on the specifics of the method and try to replicate it as closely as possible. But I think it is more important to experiment and see what works. Smart Notes has been my way of exploring Zettelkasten, and I will continue to tweak it as I learn, both from reading and researching, and from practicing. Most importantly, I will try to use my notes for creative output and learning and try to notice what features in Smart Notes maximize that.

∞

Did another bike workout at the gym. Similar heart rate and duration as my regular runs, but felt like more effort because I’m not used to cycling. I’d rather be running, but this is a good alternative. Listened to Hej (resten av) internet! Avsnitt 2: Nyhetsbrev (eller brev kort och gott).

∞

Today I bought a gym membership. I spent another 40 minutes on the bike. I felt good after the workout, but I didn’t get any ideas or inspiration like I usually do when I’m running. Hopefully I can incorporate some runs soon, but I will for sure continue biking to help build an aerobic base.