∞

Replacing Ctrl-R in Bash without TIOCSTI

I have previously written about how I use rlselect as a replacement for Ctrl+R in Bash.

It works by creating a key binding in Bash for Ctrl+R that invokes rlselect instead of the default Bash interactive history search command. rlselect looks something like this:

Screenshot of rlselect showing two entries, hello and world, with hello selected.

If you press tab, the current selection is inserted at the prompt. If you press enter, the current selection is executed. This is the same behavior as the default Ctrl+R.

The mechanism for this stopped working in recent Linux kernel versions. I figured out how to solve it and in this blog post I explain how.

Old Mechanism

When rlselect is invoked from Ctrl+R, it is invoked with the --tab and --action flags. The first flag allows the tab key to be used to select a line and the second makes rlselect print the action taken on the first line before to the selection.

Here is an example where enter is pressed when “hello” is selected:

$ (echo hello; echo world) | rlselect --tab --action
enter
hello

Here is an example where tab is pressed when “world” is selected:

$ (echo hello; echo world) | rlselect --tab --action
tab
world

Here is an example where Ctrl+G is pressed:

$ (echo hello; echo world) | rlselect --tab --action
ctrl-g

Ctrl+G aborts, so no selection is printed on the second line.

To feed this output to the prompt, TIOCSTI is used. It simulates that you type characters in the terminal. The full script that Ctrl+R invokes looks like this:

set -e

result=$(tac ~/.bash_history | rlselect --tab --action -- "$@")

python - "$result" << EOF
import fcntl
import sys
import termios

action, selection = sys.argv[1].split("\n", 1)

if action != "tab":
    selection += "\n"

for ch in selection:
    fcntl.ioctl(sys.stdout.fileno(), termios.TIOCSTI, ch)
EOF

The last part is where TIOCSTI is used to simulate that you press the keys of the selection. Unless tab is pressed, it appends a newline to the selection to simulate that Enter is pressed.

The Bash configuration looks like this:

if [[ $- =~ .*i.* ]]; then bind '"\C-r": "\C-a rlselect-history \C-j"'; fi

Here is how it works:

  • Ctrl+R is bound to a series of keystrokes.
  • First Ctrl+A is simulated which takes the cursor to the beginning of the line.
  • Then <space>rlselect-history<space> is typed.
  • Then Ctrl+J is simulated which means accept the current line. Or execute it. The initial space entered in the previous step ensures that the rlselect-history command does not end up in the history. The moving of the cursor to the beginning of the line ensures that anything typed at the prompt is passed as an argument to rlselect-history.

(This configuration also makes the text rlselect-history ... appear in the terminal. The new mechanism makes that go away.)

This mechanism stopped working in recent Linux kernel versions because TIOCSTI can not be used like this. There is apparently security issues with TIOCSTI and it is now only allowed as root.

New Mechanism

The new Bash configuration for Ctrl+R behavior that I came up with looks like this:

rlselect-history() {
    local action
    local selection
    {
        read action
        read selection
    } < <(tac ~/.bash_history | rlselect --tab --action -- "${READLINE_LINE}")
    if [ "$action" = "tab" ]; then
        READLINE_LINE="${selection}"
        READLINE_POINT=${#READLINE_LINE}
        bind '"\C-x2":' # Bind Ctrl+x+2 to do nothing
    elif [ "$action" = "enter" ]; then
        READLINE_LINE="${selection}"
        READLINE_POINT=${#READLINE_LINE}
        bind '"\C-x2": accept-line' # Bind Ctrl+x+2 to accept line
    else
        bind '"\C-x2":' # Bind Ctrl+x+2 to do nothing
    fi
}

if [[ $- =~ .*i.* ]]; then
    # Bind history command to Ctrl+x+1 followed by Ctrl+x+2:
    bind '"\C-r": "\C-x1\C-x2"'
    # Bind Ctrl+x+1 to execute rlselect-history which does two things:
    # 1. Sets READLINE_*
    # 2. Binds Ctrl+x+2 to either accept line or do nothing.
    bind -x '"\C-x1": rlselect-history'
fi

Let’s break this down.

  • Ctrl+R is bound to a series of keystrokes.

  • First Ctrl+X+1 is simulated.

  • Then Ctrl+X+2 is simulated.

  • Ctrl+X+1 is bound to execute the command rlselect-history. The -x to bind ensures that the variables READLINE_* can be set. From man bash on set -x:

    Cause shell-command to be executed whenever keyseq is entered. When shell-command is executed, the shell sets the READLINE_LINE variable to the contents of the readline line buffer and the READLINE_POINT and READLINE_MARK variables […] If the executed command changes the value of any of READLINE_LINE, READLINE_POINT, or READLINE_MARK, those new values will be reflected in the editing state.

  • rlselect-history is defined as a Bash function which allows it to reconfigure the key binding for Ctrl+X+2. Depending on if the current selection should be executed or not, it binds Ctrl+X+2 to either accept-line or nothing.

So the new mechanism relies on using two extra key bindings: Ctrl+X+1 and Ctrl+X+2. I chose them because I don’t use them otherwise. But they can be any two key bindings.

The trick to finding this solution for me was understanding Bash key bindings. This answer on StackOverflow writes the following:

With bind, you can bind keys to do one of three things, but no combination of them:

  • Execute a readline command: bind '"key": command'
  • Execute a series of keystrokes: bind '"key":"keystrokes"'
  • Execute a shell command: bind -x '"key": shell-command'

That made me understand that you can not call accept-line from within rlselect-history because it is executed in the context of bind -x, and readline commands can only be executed in the context of bind '"key": command'.

Resources

Here are some resources that talks about the problem with TIOCSTI that helped me:

∞

Today’s realization is that you can get important things done by consistently working on them for 15 minutes at the start of every day.

By doing it at the start of the day, you ensure that it gets done. And the rest of the day you don’t need to be stressed about not working on your important thing, because you already have.

∞

Newsletter December 2024: Advent of Code

December is the month of Advent of Code. I had told myself not to participate this year because I know I get completely consumed by the problems and it has a negative impact on the rest of my life. It worked. Until December 15th. More on that later.

Code Editor Update

Last month I started working on a new code editor. It is a mix of a text editor and a structured editor. It is all text, but parsers and pretty printers allow you to work with a tree structure and not think too much about syntax.

I continued working on it this month. The big achievement was that I added support for another language in addition to JSON. The other language is rlmeta. Here is a screenshot showing the parser opened in the editor:

A screenshot of releditor editing the parser of rlmeta.

This is a big achievement because it ties everything together. You define a parser and a pretty printer for your language. That gives you all editing capabilities. However, you can also write a code generator, and now you have a full blown programming language with editing support “for free”. This potentially provides an environment to quickly experiment with new programming languages.

Conceptually, I’m quite happy with this achievement. However, there are many things to work on before this is “production ready”. First of all, the performance is pretty horrible because of the constant parsing and pretty printing. Second of all, I need to see if a tree based editor can actually become better than a regular text editor.

Advent of Code

I couldn’t help myself but to participate this year as well. The experience was not as stressful as last year. I still got consumed by the problems, but the feeling was mostly positive. I managed to complete all but 3 problems. Right now, the interest to complete them is pretty low. I might take a look at other solutions to see if I can learn something from that.

My approach to solving the problems is that I try to solve them in order, and I don’t look at others' solutions until I have solved both parts. However, I’m out of ideas to try on the last problems, and I think the competition part is over by now. I might learn something for next year if I look at solutions now.

This year I also practiced object oriented design. So my solutions involve many small objects interacting with each other to produce a solution. It was mostly a success I think. One of my favorite solutions is for day 11.

This year I also think that I got the hang of Dijkstra and A*. (I found Introduction to the A* Algorithm from Red Blob Games really helpful.)

You can find all my solutions on GitHub.

∞

Today I ran part of the way to work. It was a cold, beautiful winter morning in Stockholm.

Me running with water and Stockholm City Hall in the background.
∞

Sometimes, I solve programming problems by coding on paper. A few days ago, it looked like this:

A piece of paper with source code written on it with annotations.
∞

I’ve started working on a code editor that is a mix of a text editor and a structured editor. It is all text, but parsers and pretty printers allow you to work with a tree structure and not think too much about syntax. It is a work in progress. Code is here.

Screenshot of rledit editing a JSON document with a selection.
∞

We got some more snow. I like running in the winter. Especially when there is snow and the sun is shining.

Me running in a snow-covered landscape with the sun setting in the background.
∞

I needed to submit some heic photos to a service that only accepted jpg. I didn’t know about the heic format, but a little searching gave me a solution:

$ heif-convert
bash: heif-convert: command not found...
Install package 'libheif' to provide command 'heif-convert'? [N/y] y
...
$ find . -iname '*.heic' -exec heif-convert -q 100 {} {}.jpg \;
∞

Today was the first day of snow this season. Not much. I’m looking forward to many more runs on a white trail.

Me running on a trail with a little snow.
∞

I was researching how to run Black (and possibly other formatters) from Vim and found Ergonomic mappings for code formatting in Vim. It was very helpful.

∞

How would you improve this code?

def update_r_users(service)
    r_users = []
    for user in service.get_all_users():
        if "r" in user:
            r_users.append(user)
    service.set_users_in_group("users_with_r_in_name", r_users)

Find out what I did it in my latest newsletter.

∞

Today I learned about the Rison data serialization format. I wrote a function to convert a Python value to Rison format. It was an elegant recursive function with partial support for the format.

∞

I’ve used testing without mocks quite extensively now. I’ve also used it in a work project for more than a year. My experience is that it’s the best testing strategy that I’ve ever used. I’ve never felt more confident that my code works. I refactor code without fear of it breaking. It’s so good.

∞

It’s getting dark. It gives variation to the running.

Me running in the dark.
∞

Various things have kept me from running for a while. Today I had enough. I just had to go for a short run. It was the first run with warmer clothes. The weather was nice. I reclaimed some energy.

Me running.
∞

Pull requests discourage experiments because changes can only propagate after approval. The idea behind PRs is to only approve “good” changes.

First, the learning opportunities of mistakes are gone. Second, you might loose interest in experimenting because you are afraid of making mistakes.