RLiterate 2

RLiterate is a graphical tool for doing literate programming. Literate programming is a style of programming in which you don't write source code directly. Instead you write a document which has code snippets interspersed with regular text. It shifts the mindset of the programmer from writing code to writing a document in which the program is presented in small pieces.

This book describes RLiterate, its history, and also includes the complete implementation of the second version.

Showcase

TODO: Show an example how literate programming is done in RLiterate.

Concepts

TODO: Explain concepts and features in greater detail.

Why literate programming?

To me, the central idea in literate programming is that you should write programs for other humans. Only secondary for the machine. I find code easier to understand if I can understand it in isolated pieces. One example where it is difficult to isolate a piece without literate programming is test code. Usually the test code is located in one file, and the implementation in another. To fully understand the piece it is useful to both read the tests and the implementation. To do this you have to find this related information in two unrelated files. I wrote about this problem in 2013 in Related things are not kept together. Literate programming allows you to present the test and the implementation in the same place, yet have the different pieces written to different files. The compiler might require that they are in separate files, but with literate programming, you care first about the other human that will read our code, and only second about the compiler.

Another argument for literate programming is to express the "why". Why is this code here? Timothy Daly talks about it in his talk Literate Programming in the Large. He also argues that programmers must change the mindset from wring a program to writing a book. Some more can be read in Example of Literate Programming in HTML.

Some more resources about literate programming:

Implementation

This chapter contains the complete implementation of the second version of RLiterate.

Files

RLiterate is written in a single Python file that looks like this:

  1. rliterate2.py
#!/usr/bin/env python3

from collections import namedtuple, defaultdict, OrderedDict
from operator import add, sub, mul, floordiv
import base64
import contextlib
import cProfile
import io
import json
import math
import os
import pstats
import sys
import textwrap
import time
import uuid

from pygments.token import Token as TokenType
from pygments.token import string_to_tokentype
import pygments.lexers
import pygments.token
import wx

<<globals>>

<<decorators>>

<<functions>>

<<base base classes>>

<<base classes>>

<<classes>>

if __name__ == "__main__":
    main()

There is also a corresponding test file:

  1. test_rliterate2.py
from unittest.mock import Mock, call

import rliterate2

<<test cases>>

The remainder of this chapter will fill in the sections in those files to create the complete implementation.

Main function

  1. rliterate2.py
  2. functions
def usage(script):
    sys.exit(f"usage: {script} <path>")
  1. rliterate2.py
  2. functions
def parse_args():
    args = {
        "path": None,
    }
    script = sys.argv[0]
    rest = sys.argv[1:]
    if len(rest) != 1:
        usage(script)
    args["path"] = rest[0]
    return args
  1. rliterate2.py
  2. functions
def main():
    args = parse_args()
    start_app(
        MainFrame,
        MainFrameProps(
            Document(args["path"]),
            Session(),
            Theme()
        )
    )

GUI

Main frame

  1. rliterate2.py
  2. classes
class MainFrameProps(Props):

    def __init__(self, document, session, theme):
        Props.__init__(self, {
            "title": PropUpdate(
                document, ["path"],
                format_title
            ),
            "toolbar": ToolbarProps(
                theme
            ),
            "toolbar_divider": ToolbarDividerProps(
                theme
            ),
            "main_area": MainAreaProps(
                document,
                session,
                theme
            ),
        })
  1. rliterate2.py
  2. functions
def format_title(path):
    return "{} ({}) - RLiterate 2".format(
        os.path.basename(path),
        os.path.abspath(os.path.dirname(path))
    )
  1. rliterate2.py
  2. classes
class ToolbarDividerProps(Props):

    def __init__(self, theme):
        Props.__init__(self, {
            "background": PropUpdate(
                theme, ["toolbar_divider", "color"]
            ),
            "min_size": PropUpdate(
                theme, ["toolbar_divider", "thickness"],
                lambda thickness: (-1, thickness)
            ),
        })
  1. rliterate2.py
  2. classes
frame MainFrame %layout_rows {
    Toolbar(
        #toolbar
        %align[EXPAND]
    )
    Panel(
        #toolbar_divider
        %align[EXPAND]
    )
    MainArea(
        #main_area
        %align[EXPAND]
        %proportion[1]
    )
}

Toolbar

  1. rliterate2.py
  2. classes
class ToolbarProps(Props):

    def __init__(self, theme):
        Props.__init__(self, {
            "background": PropUpdate(
                theme, ["toolbar", "background"]
            ),
            "margin": PropUpdate(
                theme, ["toolbar", "margin"]
            ),
            "actions": {
                "rotate_theme": theme.rotate,
            },
        })
  1. rliterate2.py
  2. classes
panel Toolbar %layout_columns {
    %space[#margin]
    ToolbarButton(
        icon    = "quit"
        %margin[#margin,TOP|BOTTOM]
    )
    ToolbarButton(
        icon    = "settings"
        @button = #actions.rotate_theme()
        %margin[#margin,TOP|BOTTOM]
    )
    %space[#margin]
}

Main area

  1. rliterate2.py
  2. classes
class MainAreaProps(Props):

    def __init__(self, document, session, theme):
        Props.__init__(self, {
            "toc": TableOfContentsProps(
                document,
                session,
                theme
            ),
            "toc_divider": TocDividerProps(
                theme
            ),
            "workspace": WorkspaceProps(
                document,
                session,
                theme,
            ),
            "actions": {
                "set_toc_width": session.set_toc_width,
            },
        })
  1. rliterate2.py
  2. classes
class TocDividerProps(Props):

    def __init__(self, theme):
        Props.__init__(self, {
            "background": PropUpdate(
                theme, ["toc_divider", "color"]
            ),
            "min_size": PropUpdate(
                theme, ["toc_divider", "thickness"],
                lambda thickness: (thickness, -1)
            ),
        })
  1. rliterate2.py
  2. classes
panel MainArea %layout_columns {
    TableOfContents[toc](
        #toc
        %align[EXPAND]
    )
    Panel(
        #toc_divider
        cursor = "size_horizontal"
        @drag  = self._on_toc_divider_drag(event)
        %align[EXPAND]
    )
    Workspace(
        #workspace
        %align[EXPAND]
        %proportion[1]
    )
}

<<MainArea>>
  1. rliterate2.py
  2. classes
  3. MainArea
def _on_toc_divider_drag(self, event):
    if event.initial:
        toc = self.get_widget("toc")
        self._start_width = toc.get_width()
    else:
        self.prop(["actions", "set_toc_width"])(
            self._start_width + event.dx
        )

Table of contents

  1. rliterate2.py
  2. classes
class TableOfContentsProps(Props):

    def __init__(self, document, session, theme):
        Props.__init__(self, {
            "background": PropUpdate(
                theme, ["toc", "background"]
            ),
            "min_size": PropUpdate(
                session, ["toc", "width"],
                lambda width: (max(50, width), -1)
            ),
            "has_valid_hoisted_page": PropUpdate(
                document,
                session, ["toc", "hoisted_page"],
                is_valid_hoisted_page
            ),
            "row_margin": PropUpdate(
                theme, ["toc", "row_margin"]
            ),
            "scroll_area": TableOfContentsScrollAreaProps(
                document,
                session,
                theme
            ),
            "actions": {
                "set_hoisted_page": session.set_hoisted_page,
            },
        })
  1. rliterate2.py
  2. functions
def is_valid_hoisted_page(document, page_id):
    try:
        page = document.get_page(page_id)
        root_page = document.get_page()
        if page["id"] != root_page["id"]:
            return True
    except PageNotFound:
        pass
    return False
  1. rliterate2.py
  2. classes
panel TableOfContents %layout_rows {
    if (#has_valid_hoisted_page) {
        Button(
            label   = "unhoist"
            @button = #actions.set_hoisted_page(None)
            %margin[add(1 #row_margin),ALL]
            %align[EXPAND]
        )
    }
    TableOfContentsScrollArea(
        #scroll_area
        %align[EXPAND]
        %proportion[1]
    )
}
Scroll area
  1. rliterate2.py
  2. classes
class TableOfContentsScrollAreaProps(Props):

    def __init__(self, document, session, theme):
        Props.__init__(self, {
            "*": PropUpdate(
                document,
                session, ["toc", "collapsed"],
                session, ["toc", "hoisted_page"],
                session, ["toc", "dragged_page"],
                generate_rows_and_drop_points
            ),
            "rows_cache_limit": PropUpdate(
                document,
                lambda document: document.count_pages() - 1
            ),
            "row_extra": TableOfContentsRowExtraProps(
                document,
                session,
                theme
            ),
            "dragged_page": PropUpdate(
                session, ["toc", "dragged_page"]
            ),
            "actions": {
                "can_move_page": document.can_move_page,
                "move_page": document.move_page,
            },
        })
  1. rliterate2.py
  2. classes
class TableOfContentsRowExtraProps(Props):

    def __init__(self, document, session, theme):
        Props.__init__(self, {
            "row_margin": PropUpdate(
                theme, ["toc", "row_margin"]
            ),
            "indent_size": PropUpdate(
                theme, ["toc", "indent_size"]
            ),
            "foreground": PropUpdate(
                theme, ["toc", "foreground"]
            ),
            "hover_background": PropUpdate(
                theme, ["toc", "hover_background"]
            ),
            "divider_thickness": PropUpdate(
                theme, ["toc", "divider_thickness"]
            ),
            "dragdrop_color": PropUpdate(
                theme, ["dragdrop_color"]
            ),
            "dragdrop_invalid_color": PropUpdate(
                theme, ["dragdrop_invalid_color"]
            ),
            "actions": {
                "set_hoisted_page": session.set_hoisted_page,
                "set_dragged_page": session.set_dragged_page,
                "toggle_collapsed": session.toggle_collapsed,
                "open_page": session.open_page,
            },
        })
  1. rliterate2.py
  2. functions
def generate_rows_and_drop_points(
    document,
    collapsed,
    hoisted_page,
    dragged_page
):
    def traverse(page, level=0, dragged=False):
        is_collapsed = page["id"] in collapsed
        num_children = len(page["children"])
        dragged = dragged or page["id"] == dragged_page
        rows.append({
            "id": page["id"],
            "title_fragments": [{"text": page["title"]}],
            "level": level,
            "has_children": num_children > 0,
            "collapsed": is_collapsed,
            "dragged": dragged,
        })
        if is_collapsed:
            target_index = num_children
        else:
            target_index = 0
        drop_points.append(TableOfContentsDropPoint(
            row_index=len(rows)-1,
            target_index=target_index,
            target_page=page["id"],
            level=level+1
        ))
        if not is_collapsed:
            for target_index, child in enumerate(page["children"]):
                traverse(child, level+1, dragged=dragged)
                drop_points.append(TableOfContentsDropPoint(
                    row_index=len(rows)-1,
                    target_index=target_index+1,
                    target_page=page["id"],
                    level=level+1
                ))
    rows = []
    drop_points = []
    try:
        root_page = document.get_page(hoisted_page)
    except PageNotFound:
        root_page = document.get_page(None)
    traverse(root_page)
    return {
        "rows": rows,
        "drop_points": drop_points,
    }
  1. rliterate2.py
  2. classes
TableOfContentsDropPoint = namedtuple("TableOfContentsDropPoint", [
    "row_index",
    "target_index",
    "target_page",
    "level",
])
  1. rliterate2.py
  2. classes
scroll TableOfContentsScrollArea %layout_rows {
    drop_target = "move_page"
    loop (#rows cache_limit=#rows_cache_limit) {
        TableOfContentsRow[rows](
            $
            #row_extra
            __reuse = $id
            __cache = True
            %align[EXPAND]
        )
    }
}

<<TableOfContentsScrollArea>>
  1. rliterate2.py
  2. classes
  3. TableOfContentsScrollArea
_last_drop_row = None

def on_drag_drop_over(self, x, y):
    self._hide()
    drop_point = self._get_drop_point(x, y)
    if drop_point is not None:
        self._last_drop_row = self._get_drop_row(drop_point)
    if self._last_drop_row is not None:
        valid = self.prop(["actions", "can_move_page"])(
            self.prop(["dragged_page"]),
            drop_point.target_page,
            drop_point.target_index
        )
        self._last_drop_row.show_drop_line(
            self._calculate_indent(drop_point.level),
            valid=valid
        )
        return valid
    return False

def on_drag_drop_leave(self):
    self._hide()

def on_drag_drop_data(self, x, y, page_info):
    drop_point = self._get_drop_point(x, y)
    if drop_point is not None:
        self.prop(["actions", "move_page"])(
            self.prop(["dragged_page"]),
            drop_point.target_page,
            drop_point.target_index
        )

def _hide(self):
    if self._last_drop_row is not None:
        self._last_drop_row.hide_drop_line()

def _get_drop_point(self, x, y):
    lines = defaultdict(list)
    for drop_point in self.prop(["drop_points"]):
        lines[
            self._y_distance_to(
                self._get_drop_row(drop_point),
                y
            )
        ].append(drop_point)
    if lines:
        columns = {}
        for drop_point in lines[min(lines.keys())]:
            columns[self._x_distance_to(drop_point, x)] = drop_point
        return columns[min(columns.keys())]

def _get_drop_row(self, drop_point):
    return self.get_widget("rows", drop_point.row_index)

def _y_distance_to(self, row, y):
    span_y_center = row.get_y() + row.get_drop_line_y_offset()
    return int(abs(span_y_center - y))

def _x_distance_to(self, drop_point, x):
    return int(abs(self._calculate_indent(drop_point.level + 1.5) - x))

def _calculate_indent(self, level):
    return (
        (2 * self.prop(["row_extra", "row_margin"])) +
        (level + 1) * self.prop(["row_extra", "indent_size"])
    )
Row
  1. rliterate2.py
  2. classes
panel TableOfContentsRow %layout_rows {
    TableOfContentsTitle[title](
        #
        @click       = #actions.open_page(#id)
        @drag        = self._on_drag(event)
        @right_click = self._on_right_click(event)
        @hover       = self._set_background(event.mouse_inside)
        %align[EXPAND]
    )
    TableOfContentsDropLine[drop_line](
        indent        = 0
        active        = False
        valid         = True
        thickness     = #divider_thickness
        color         = #dragdrop_color
        invalid_color = #dragdrop_invalid_color
        %align[EXPAND]
    )
}

<<TableOfContentsRow>>
  1. rliterate2.py
  2. classes
  3. TableOfContentsRow
def _on_drag(self, event):
    if not event.initial:
        self.prop(["actions", "set_dragged_page"])(
            self.prop(["id"])
        )
        try:
            event.initiate_drag_drop("move_page", {})
        finally:
            self.prop(["actions", "set_dragged_page"])(
                None
            )
  1. rliterate2.py
  2. classes
  3. TableOfContentsRow
def _on_right_click(self, event):
    event.show_context_menu([
        ("Hoist", self.prop(["level"]) > 0, lambda:
            self.prop(["actions", "set_hoisted_page"])(
                self.prop(["id"])
            )
        ),
    ])
  1. rliterate2.py
  2. classes
  3. TableOfContentsRow
def _set_background(self, hover):
    if hover:
        self.get_widget("title").update_props({
            "background": self.prop(["hover_background"]),
        })
    else:
        self.get_widget("title").update_props({
            "background": None,
        })
  1. rliterate2.py
  2. classes
  3. TableOfContentsRow
def get_drop_line_y_offset(self):
    drop_line = self.get_widget("drop_line")
    return drop_line.get_y() + drop_line.get_height() / 2
  1. rliterate2.py
  2. classes
  3. TableOfContentsRow
def show_drop_line(self, indent, valid):
    self.get_widget("drop_line").update_props({
        "active": True,
        "valid": valid,
        "indent": indent
    })
  1. rliterate2.py
  2. classes
  3. TableOfContentsRow
def hide_drop_line(self):
    self.get_widget("drop_line").update_props({
        "active": False,
    })
Title
  1. rliterate2.py
  2. classes
panel TableOfContentsTitle %layout_columns {
    %space[add(#row_margin mul(#level #indent_size))]
    if (#has_children) {
        ExpandCollapse(
            cursor       = "hand"
            size         = #indent_size
            collapsed    = #collapsed
            @click       = #actions.toggle_collapsed(#id)
            @drag        = None
            @right_click = None
            %align[EXPAND]
        )
    } else {
        %space[#indent_size]
    }
    Text(
        fragments  = #title_fragments
        foreground = self._foreground()
        %align[EXPAND]
        %margin[#row_margin,ALL]
    )
}

<<TableOfContentsTitle>>
  1. rliterate2.py
  2. classes
  3. TableOfContentsTitle
def _foreground(self):
    if self.prop(["dragged"]):
        return self.prop(["dragdrop_invalid_color"])
    else:
        return self.prop(["foreground"])
Drop line
  1. rliterate2.py
  2. classes
panel TableOfContentsDropLine %layout_columns {
    %space[#indent]
    Panel(
        min_size   = makeTuple(-1 #thickness)
        background = self._get_color(#active #valid)
        %align[EXPAND]
        %proportion[1]
    )
}

<<TableOfContentsDropLine>>
  1. rliterate2.py
  2. classes
  3. TableOfContentsDropLine
def _get_color(self, active, valid):
    if active:
        if valid:
            return self.prop(["color"])
        else:
            return self.prop(["invalid_color"])
    else:
        return None

Workspace

  1. rliterate2.py
  2. classes
class WorkspaceProps(Props):

    def __init__(self, document, session, theme):
        self._paragraph_cache = Cache(limit=1000)
        Props.__init__(self, {
            "background": PropUpdate(
                theme, ["workspace", "background"]
            ),
            "margin": PropUpdate(
                theme, ["workspace", "margin"]
            ),
            "column_width": PropUpdate(
                session, ["workspace", "page_body_width"],
                theme, ["page", "margin"],
                theme, ["page", "border", "size"],
                lambda body, margin, border: (
                    body + 2*margin + border
                )
            ),
            "columns": PropUpdate(
                document,
                session, ["workspace", "columns"],
                session, ["workspace", "page_body_width"],
                theme, ["page"],
                self.build_columns
            ),
            "page_body_width": PropUpdate(
                session, ["workspace", "page_body_width"]
            ),
            "actions": {
                "set_page_body_width": session.set_page_body_width,
                "edit_title": document.edit_title,
            },
        })

    <<WorkspaceProps>>
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
@profile_sub("build_columns")
def build_columns(self, document, columns, page_body_width, page_theme):
    columns_prop = []
    for column in columns:
        columns_prop.append(
            self.build_column(document, column, page_body_width, page_theme)
        )
    return columns_prop
  1. rliterate2.py
  2. classes
hscroll Workspace %layout_columns {
    %space[#margin]
    loop (#columns) {
        Column(
            min_size         = makeTuple(#column_width -1)
            column           = $
            workspace_margin = #margin
            actions          = #actions
            %align[EXPAND]
        )
        Panel(
            cursor   = "size_horizontal"
            min_size = makeTuple(#margin -1)
            @drag    = self._on_divider_drag(event)
            %align[EXPAND]
        )
    }
}

<<Workspace>>
  1. rliterate2.py
  2. classes
  3. Workspace
def _on_divider_drag(self, event):
    if event.initial:
        self._initial_width = self.prop(["page_body_width"])
    else:
        self.prop(["actions", "set_page_body_width"])(
            max(50, self._initial_width + event.dx)
        )
Column
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_column(self, document, column, page_body_width, page_theme):
    column_prop = []
    for page_id in column:
        try:
            column_prop.append(
                self.build_page(
                    document.get_page(page_id),
                    page_body_width,
                    page_theme
                )
            )
        except PageNotFound:
            pass
    return column_prop
  1. rliterate2.py
  2. classes
vscroll Column %layout_rows {
    %space[#workspace_margin]
    loop (#column) {
        Page(
            page    = $
            actions = #actions
            %align[EXPAND]
        )
        %space[#workspace_margin]
    }
}
Page
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_page(self, page, page_body_width, page_theme):
    return {
        "id": page["id"],
        "title_fragments": self.build_title_fragments(
            page["title"]
        ),
        "paragraphs": self.build_paragraphs(
            page["paragraphs"],
            page_theme
        ),
        "border": page_theme["border"],
        "background": page_theme["background"],
        "title_font": page_theme["title_font"],
        "margin": page_theme["margin"],
        "body_width": page_body_width,
    }
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_title_fragments(self, title):
    return [{"text": title}]
  1. rliterate2.py
  2. classes
panel Page %layout_rows {
    PageTopRow(
        page    = #page
        actions = #actions
        %align[EXPAND]
        %proportion[1]
    )
    PageBottomBorder(
        #page.border
        %align[EXPAND]
    )
}
  1. rliterate2.py
  2. classes
panel PageTopRow %layout_columns {
    PageBody(
        #page
        actions = #actions
        %align[EXPAND]
        %proportion[1]
    )
    PageRightBorder(
        #page.border
        %align[EXPAND]
    )
}
  1. rliterate2.py
  2. classes
panel PageBody %layout_rows {
    Text(
        fragments = #title_fragments
        max_width = #body_width
        font      = #title_font
        %margin[#margin,ALL]
        @click    = #actions.edit_title(#id)
    )
    loop (#paragraphs) {
        $widget(
            $
            body_width = #body_width
            %margin[#margin,LEFT|BOTTOM|RIGHT]
            %align[EXPAND]
        )
    }
}
  1. rliterate2.py
  2. classes
panel PageRightBorder %layout_rows {
    %space[#size]
    Panel(
        min_size   = makeTuple(#size -1)
        background = #color
        %align[EXPAND]
        %proportion[1]
    )
}
  1. rliterate2.py
  2. classes
panel PageBottomBorder %layout_columns {
    %space[#size]
    Panel(
        min_size   = makeTuple(-1 #size)
        background = #color
        %align[EXPAND]
        %proportion[1]
    )
}
Paragraphs
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_paragraphs(self, paragraphs, page_theme):
    return [
        self._paragraph_cache.get_or_update(
            paragraph["id"],
            "paragraph",
            self.build_paragraph,
            paragraph,
            page_theme
        )
        for paragraph in paragraphs
    ]
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_paragraph(self, paragraph, page_theme):
    BUILDERS = {
        "text": self.build_text_paragraph,
        "quote": self.build_quote_paragraph,
        "list": self.build_list_paragraph,
        "code": self.build_code_paragraph,
        "image": self.build_image_paragraph,
    }
    return BUILDERS.get(
        paragraph["type"],
        self.build_unknown_paragraph
    )(paragraph, page_theme)
Text
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
@profile_sub("build_text_paragraph")
def build_text_paragraph(self, paragraph, page_theme):
    return {
        "widget": TextParagraph,
        "text_fragments": paragraph["fragments"],
    }
  1. rliterate2.py
  2. classes
panel TextParagraph %layout_rows {
    Text(
        fragments     = #text_fragments
        max_width     = #body_width
        break_at_word = True
        line_height   = 1.2
        %align[EXPAND]
    )
}
Quote
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
@profile_sub("build_quote_paragraph")
def build_quote_paragraph(self, paragraph, page_theme):
    return {
        "widget": QuoteParagraph,
        "text_fragments": paragraph["fragments"],
    }
  1. rliterate2.py
  2. classes
panel QuoteParagraph %layout_columns {
    TextWithMargin(
        left_margin   = 20
        right_margin  = 0
        fragments     = #text_fragments
        max_width     = sub(#body_width 20)
        %align[EXPAND]
    )
}
List
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
@profile_sub("build_list_paragraph")
def build_list_paragraph(self, paragraph, page_theme):
    return {
        "widget": ListParagraph,
        "rows": self.build_list_item_rows(
            paragraph["children"],
            paragraph["child_type"]
        ),
    }
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_list_item_rows(self, children, child_type, level=0):
    rows = []
    for index, child in enumerate(children):
        rows.append({
            "text_fragments": [
                {"text": self._get_bullet_text(child_type, index)}
            ]+child["fragments"],
            "level": level,
        })
        rows.extend(self.build_list_item_rows(
            child["children"],
            child["child_type"],
            level+1
        ))
    return rows

def _get_bullet_text(self, list_type, index):
    if list_type == "ordered":
        return "{}. ".format(index + 1)
    else:
        return u"\u2022 "
  1. rliterate2.py
  2. classes
panel ListParagraph %layout_rows {
    loop (#rows) {
        TextWithMargin(
            left_margin  = mul($level 20)
            right_margin = 0
            max_width    = sub(#body_width mul($level 20))
            fragments    = $text_fragments
            %align[EXPAND]
        )
    }
}
Code
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
@profile_sub("build_code_paragraph")
def build_code_paragraph(self, paragraph, page_theme):
    return {
        "widget": CodeParagraph,
        "header": self.build_code_paragraph_header(
            paragraph,
            page_theme
        ),
        "body": self.build_code_paragraph_body(
            paragraph,
            page_theme
        ),
    }
  1. rliterate2.py
  2. classes
panel CodeParagraph %layout_rows {
    CodeParagraphHeader(
        #header
        body_width = #body_width
        %align[EXPAND]
    )
    CodeParagraphBody(
        #body
        body_width = #body_width
        %align[EXPAND]
    )
}
Header
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_code_paragraph_header(self, paragraph, page_theme):
    return {
        "background": page_theme["code"]["header_background"],
        "margin": page_theme["code"]["margin"],
        "font": page_theme["code_font"],
        "path_fragments": self.build_code_path_fragments(
            paragraph["filepath"],
            paragraph["chunkpath"]
        ),
    }
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_code_path_fragments(self, filepath, chunkpath):
    fragments = []
    for index, x in enumerate(filepath):
        if index > 0:
            fragments.append({"text": "/"})
        fragments.append({"text": x})
    if filepath and chunkpath:
        fragments.append({"text": " "})
    for index, x in enumerate(chunkpath):
        if index > 0:
            fragments.append({"text": "/"})
        fragments.append({"text": x})
    return fragments
  1. rliterate2.py
  2. classes
panel CodeParagraphHeader %layout_rows {
    Text(
        fragments     = #path_fragments
        max_width     = sub(#body_width mul(2 #margin))
        font          = #font
        break_at_word = False
        %align[EXPAND]
        %margin[#margin,ALL]
    )
}
Body
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_code_paragraph_body(self, paragraph, page_theme):
    return {
        "background": page_theme["code"]["body_background"],
        "margin": page_theme["code"]["margin"],
        "font": page_theme["code_font"],
        "body_fragments": self.apply_token_styles(
            self.build_code_body_fragments(
                paragraph["fragments"],
                self.code_pygments_lexer(
                    paragraph.get("language", ""),
                    paragraph["filepath"][-1] if paragraph["filepath"] else "",
                )
            ),
            page_theme["token_styles"]
        ),
    }
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_code_body_fragments(self, fragments, pygments_lexer):
    code_chunk = CodeChunk()
    for fragment in fragments:
        if fragment["type"] == "code":
            code_chunk.add(fragment["text"])
        elif fragment["type"] == "chunk":
            code_chunk.add(
                "{}<<{}>>\n".format(
                    fragment["prefix"],
                    "/".join(fragment["path"])
                ),
                {"token_type": TokenType.Comment.Preproc}
            )
    return code_chunk.tokenize(pygments_lexer)
  1. rliterate2.py
  2. classes
panel CodeParagraphBody %layout_rows {
    Text(
        fragments     = #body_fragments
        max_width     = sub(#body_width mul(2 #margin))
        font          = #font
        break_at_word = False
        %align[EXPAND]
        %margin[#margin,ALL]
    )
}
Image
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
@profile_sub("build_image_paragraph")
def build_image_paragraph(self, paragraph, page_theme):
    return {
        "widget": ImageParagraph,
        "base64_image": paragraph.get("image_base64", None),
        "text_fragments": paragraph["fragments"],
    }
  1. rliterate2.py
  2. classes
panel ImageParagraph %layout_rows {
    Image(
        base64_image = #base64_image
        width        = #body_width
        %align[CENTER]
    )
    TextWithMargin(
        left_margin  = 10
        right_margin = 10
        fragments    = #text_fragments
        max_width    = sub(#body_width 20)
        %align[CENTER]
    )
}
Unknown paragraph
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_unknown_paragraph(self, paragraph, page_theme):
    return {
        "widget": UnknownParagraph,
        "fragments": [{"text": "Unknown paragraph type '{}'.".format(paragraph["type"])}],
        "font": page_theme["code_font"],
    }
  1. rliterate2.py
  2. classes
panel UnknownParagraph %layout_rows {
    Text(
        fragments     = #fragments
        max_width     = #body_width
        font          = #font
        break_at_word = False
        %align[EXPAND]
    )
}
Utilities
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def code_pygments_lexer(self, language, filename):
    try:
        if language:
            return pygments.lexers.get_lexer_by_name(
                language,
                stripnl=False
            )
        else:
            return pygments.lexers.get_lexer_for_filename(
                filename,
                stripnl=False
            )
    except:
        return pygments.lexers.TextLexer(stripnl=False)
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
@profile_sub("apply_token_styles")
def apply_token_styles(self, fragments, token_styles):
    styles = self.build_style_dict(token_styles)
    def style_fragment(fragment):
        if "token_type" in fragment:
            token_type = fragment["token_type"]
            while token_type not in styles:
                token_type = token_type.parent
            style = styles[token_type]
            new_fragment = dict(fragment)
            new_fragment.update(style)
            return new_fragment
        else:
            return fragment
    return [
        style_fragment(fragment)
        for fragment in fragments
    ]
  1. rliterate2.py
  2. classes
  3. WorkspaceProps
def build_style_dict(self, theme_style):
    styles = {}
    for name, value in theme_style.items():
        styles[string_to_tokentype(name)] = value
    return styles

Reusable widgets

TextWithMargin
  1. rliterate2.py
  2. classes
panel TextWithMargin %layout_columns {
    %space[#left_margin]
    Text(
        fragments     = #fragments
        max_width     = #max_width
        break_at_word = True
        line_height   = 1.2
        %align[EXPAND]
    )
    %space[#right_margin]
}

Document

  1. rliterate2.py
  2. classes
class Document(Immutable):

    ROOT_PAGE_PATH = ["doc", "root_page"]

    def __init__(self, path):
        Immutable.__init__(self, {
            "path": path,
            "doc": load_document_from_file(path),
        })
        self._build_page_index()

    <<Document>>
  1. rliterate2.py
  2. functions
def load_document_from_file(path):
    if os.path.exists(path):
        return load_json_from_file(path)
    else:
        return create_new_document()
  1. rliterate2.py
  2. functions
def create_new_document():
    return {
        "root_page": create_new_page(),
        "variables": {},
    }

Page index

  1. rliterate2.py
  2. classes
  3. Document
def _build_page_index(self):
    def build(page, path, parent, index):
        page_meta = PageMeta(page["id"], path, parent, index)
        page_index[page["id"]] = page_meta
        for index, child in enumerate(page["children"]):
            build(child, path+["children", index], page_meta, index)
    page_index = {}
    build(self.get(self.ROOT_PAGE_PATH), self.ROOT_PAGE_PATH, None, 0)
    self._page_index = page_index
  1. rliterate2.py
  2. classes
  3. Document
def _get_page_meta(self, page_id):
    if page_id not in self._page_index:
        raise PageNotFound()
    return self._page_index[page_id]
  1. rliterate2.py
  2. classes
class PageNotFound(Exception):
    pass
  1. rliterate2.py
  2. classes
class PageMeta(object):

    def __init__(self, id, path, parent, index):
        self.id = id
        self.path = path
        self.parent = parent
        self.index = index

Pages

  1. rliterate2.py
  2. functions
def create_new_page():
    return {
        "id": genid(),
        "title": "New page...",
        "children": [],
        "paragraphs": [],
    }
  1. rliterate2.py
  2. classes
  3. Document
def get_page(self, page_id=None):
    if page_id is None:
        return self.get(self.ROOT_PAGE_PATH)
    else:
        return self.get(self._get_page_meta(page_id).path)
  1. rliterate2.py
  2. classes
  3. Document
def count_pages(self):
    return len(self._page_index)
  1. rliterate2.py
  2. classes
  3. Document
def move_page(self, source_id, target_id, target_index):
    try:
        self._move_page(
            self._get_page_meta(source_id),
            self._get_page_meta(target_id),
            target_index
        )
    except PageNotFound:
        pass
  1. rliterate2.py
  2. classes
  3. Document
def can_move_page(self, source_id, target_id, target_index):
    try:
        return self._can_move_page(
            self._get_page_meta(source_id),
            self._get_page_meta(target_id),
            target_index
        )
    except PageNotFound:
        return False
  1. rliterate2.py
  2. classes
  3. Document
def _move_page(self, source_meta, target_meta, target_index):
    if not self._can_move_page(source_meta, target_meta, target_index):
        return
    source_page = self.get(source_meta.path)
    operation_insert = (
        target_meta.path + ["children"],
        lambda children: (
            children[:target_index] +
            [source_page] +
            children[target_index:]
        )
    )
    operation_remove = (
        source_meta.parent.path + ["children"],
        lambda children: (
            children[:source_meta.index] +
            children[source_meta.index+1:]
        )
    )
    if target_meta.id == source_meta.parent.id:
        insert_first = target_index > source_meta.index
    else:
        insert_first = len(target_meta.path) > len(source_meta.parent.path)
    if insert_first:
        operations = [operation_insert, operation_remove]
    else:
        operations = [operation_remove, operation_insert]
    with self.transaction():
        self.modify_many(operations)
        self._build_page_index()
  1. rliterate2.py
  2. classes
  3. Document
def _can_move_page(self, source_meta, target_meta, target_index):
    page_meta = target_meta
    while page_meta is not None:
        if page_meta.id == source_meta.id:
            return False
        page_meta = page_meta.parent
    if (target_meta.id == source_meta.parent.id and
        target_index in [source_meta.index, source_meta.index+1]):
        return False
    return True
  1. rliterate2.py
  2. classes
  3. Document
def edit_title(self, source_id):
    try:
        page_meta = self._get_page_meta(source_id)
        self.modify(
            page_meta.path + ["title"],
            lambda title: title + "."
        )
    except PageNotFound:
        pass

Id generation

  1. rliterate2.py
  2. functions
def genid():
    return uuid.uuid4().hex

Code paragraph utilities

  1. rliterate2.py
  2. classes
class CodeChunk(object):

    def __init__(self):
        self._fragments = []

    def add(self, text, extra={}):
        part = {"text": text}
        part.update(extra)
        self._fragments.append(part)

    def tokenize(self, pygments_lexer):
        self._apply_token_types(
            pygments_lexer.get_tokens(
                self._get_uncolorized_text()
            )
        )
        return self._fragments

    def _get_uncolorized_text(self):
        return "".join(
            part["text"]
            for part in self._fragments
            if "token_type" not in part
        )

    def _apply_token_types(self, pygments_tokens):
        part_index = 0
        for token_type, text in pygments_tokens:
            while "token_type" in self._fragments[part_index]:
                part_index += 1
            while text:
                if len(self._fragments[part_index]["text"]) > len(text):
                    part = self._fragments[part_index]
                    pre = dict(part)
                    pre["text"] = pre["text"][:len(text)]
                    pre["token_type"] = token_type
                    self._fragments[part_index] = pre
                    part_index += 1
                    post = dict(part)
                    post["text"] = post["text"][len(text):]
                    self._fragments.insert(part_index, post)
                    text = ""
                else:
                    part = self._fragments[part_index]
                    part["token_type"] = token_type
                    part_index += 1
                    text = text[len(part["text"]):]

Theme

  1. rliterate2.py
  2. classes
class Theme(Immutable):

    base00  = "#657b83"
    base1   = "#93a1a1"
    yellow  = "#b58900"
    orange  = "#cb4b16"
    red     = "#dc322f"
    magenta = "#d33682"
    violet  = "#6c71c4"
    blue    = "#268bd2"
    cyan    = "#2aa198"
    green   = "#859900"

    DEFAULT = {
        "toolbar": {
            "margin": 4,
            "background": None,
        },
        "toolbar_divider": {
            "thickness": 1,
            "color": "#aaaaaf",
        },
        "toc": {
            "background": "#ffffff",
            "foreground": "#000000",
            "indent_size": 20,
            "row_margin": 2,
            "divider_thickness": 2,
            "hover_background": "#cccccc",
        },
        "toc_divider": {
            "thickness": 3,
            "color": "#aaaaaf",
        },
        "workspace": {
            "background": "#cccccc",
            "margin": 12,
        },
        "page": {
            "title_font": {
                "size": 16,
            },
            "code_font": {
                "size": 10,
                "family": "Monospace",
            },
            "border": {
                "size": 2,
                "color": "#aaaaaf",
            },
            "background": "#ffffff",
            "margin": 10,
            "code": {
                "margin": 5,
                "header_background": "#eeeeee",
                "body_background": "#f8f8f8",
            },
            "token_styles": {
                "":                    {"color": base00},
                "Keyword":             {"color": green},
                "Keyword.Constant":    {"color": cyan},
                "Keyword.Declaration": {"color": blue},
                "Keyword.Namespace":   {"color": orange},
                "Name.Builtin":        {"color": red},
                "Name.Builtin.Pseudo": {"color": blue},
                "Name.Class":          {"color": blue},
                "Name.Decorator":      {"color": blue},
                "Name.Entity":         {"color": violet},
                "Name.Exception":      {"color": yellow},
                "Name.Function":       {"color": blue},
                "String":              {"color": cyan},
                "Number":              {"color": cyan},
                "Operator.Word":       {"color": green},
                "Comment":             {"color": base1},
                "Comment.Preproc":     {"color": magenta},
            },
        },
        "dragdrop_color": "#ff6400",
        "dragdrop_invalid_color": "#cccccc",
    }

    ALTERNATIVE = {
        "toolbar": {
            "margin": 4,
            "background": "#dcd6c6",
        },
        "toolbar_divider": {
            "thickness": 2,
            "color": "#b0ab9e",
        },
        "toc": {
            "background": "#fdf6e3",
            "foreground": "#657b83",
            "indent_size": 22,
            "row_margin": 3,
            "divider_thickness": 3,
            "hover_background": "#d0cabb",
        },
        "toc_divider": {
            "thickness": 5,
            "color": "#b0ab9e",
        },
        "workspace": {
            "background": "#d0cabb",
            "margin": 18,
        },
        "page": {
            "title_font": {
                "size": 18,
            },
            "code_font": {
                "size": 12,
                "family": "Monospace",
            },
            "border": {
                "size": 3,
                "color": "#b0ab9e",
            },
            "background": "#fdf6e3",
            "margin": 14,
            "code": {
                "margin": 7,
                "header_background": "#eae4d2",
                "body_background": "#f3ecdb",
            },
            "token_styles": {
                "":                    {"color": base00},
                "Keyword":             {"color": green},
                "Keyword.Constant":    {"color": cyan},
                "Keyword.Declaration": {"color": blue},
                "Keyword.Namespace":   {"color": orange},
                "Name.Builtin":        {"color": red},
                "Name.Builtin.Pseudo": {"color": blue},
                "Name.Class":          {"color": blue},
                "Name.Decorator":      {"color": blue},
                "Name.Entity":         {"color": violet},
                "Name.Exception":      {"color": yellow},
                "Name.Function":       {"color": blue},
                "String":              {"color": cyan},
                "Number":              {"color": cyan},
                "Operator.Word":       {"color": green},
                "Comment":             {"color": base1},
                "Comment.Preproc":     {"color": magenta},
           },
        },
        "dragdrop_color": "#dc322f",
        "dragdrop_invalid_color": "#cccccc",
    }

    def __init__(self):
        Immutable.__init__(self, self.DEFAULT)

    def rotate(self):
        if self.get([]) is self.ALTERNATIVE:
            self.replace([], self.DEFAULT)
        else:
            self.replace([], self.ALTERNATIVE)

Session

  1. rliterate2.py
  2. classes
class Session(Immutable):

    def __init__(self):
        Immutable.__init__(self, {
            "toc": {
                "width": 230,
                "collapsed": [],
                "hoisted_page": None,
                "dragged_page": None,
            },
            "workspace": {
                "page_body_width": 300,
                "columns": [
                    [
                        "cf689824aa3641828343eba2b5fbde9f",
                        "ef8200090225487eab4ae35d8910ba8e",
                        "97827e5f0096482a9a4eadf0ce07764f"
                    ],
                    [
                        "e6a157bbac8842a2b8c625bfa9255159",
                        "813ec304685345a19b1688074000d296",
                        "004bc5a29bc94eeb95f4f6a56bd48729",
                        "b987445070e84067ba90e71695763f72"
                    ]
                ],
            },
        })

    def open_page(self, page_id):
        self.replace(["workspace", "columns"], [[page_id]])

    def set_hoisted_page(self, page_id):
        self.replace(["toc", "hoisted_page"], page_id)

    def set_dragged_page(self, page_id):
        self.replace(["toc", "dragged_page"], page_id)

    def set_toc_width(self, width):
        self.replace(["toc", "width"], width)

    def set_page_body_width(self, width):
        self.replace(["workspace", "page_body_width"], width)

    def toggle_collapsed(self, page_id):
        def toggle(collapsed):
            if page_id in collapsed:
                return [x for x in collapsed if x != page_id]
            else:
                return collapsed + [page_id]
        self.modify(["toc", "collapsed"], toggle)

GUI framework

Compiler

  1. rlgui
  2. rlgui.py
#!/usr/bin/env python2

from collections import defaultdict
import sys

<<rlmeta support library>>

<<grammars>>

<<support functions>>

if __name__ == "__main__":
    parser = GuiParser()
    codegenerator = WxCodeGenerator()
    try:
        sys.stdout.write(
            codegenerator.run("ast", parser.run("widget", sys.stdin.read()))
        )
    except _MatchError as e:
        sys.exit(e.describe())
  1. rlgui
  2. rlgui.py
  3. rlmeta support library
# Placeholder to generate RLMeta support library

Support library

Widget mixin
  1. rliterate2.py
  2. base classes
class WidgetMixin(object):

    def __init__(self, parent, handlers, props):
        self._parent = parent
        self._props = {}
        self._builtin_props = {}
        self._event_handlers = {}
        self._setup_gui()
        self.update_event_handlers(handlers)
        self.update_props(props, parent_updated=True)

    def update_event_handlers(self, handlers):
        for name, fn in handlers.items():
            self.register_event_handler(name, fn)

    @profile_sub("register event")
    def register_event_handler(self, name, fn):
        self._event_handlers[name] = profile(f"on_{name}")(profile_sub(f"on_{name}")(fn))

    def call_event_handler(self, name, event, propagate=False):
        if self.has_event_handler(name):
            self._event_handlers[name](event)
        elif self._parent is not None and propagate:
            self._parent.call_event_handler(name, event, True)

    def has_event_handler(self, name):
        return name in self._event_handlers

    def _setup_gui(self):
        pass

    def prop_with_default(self, path, default):
        try:
            return self.prop(path)
        except (KeyError, IndexError):
            return default

    def prop(self, path):
        value = self._props
        for part in path:
            value = value[part]
        return value

    def update_props(self, props, parent_updated=False):
        if self._update_props(props):
            self._update_gui(parent_updated)

    def _update_props(self, props):
        self._changed_props = []
        for p in [lambda: props, self._get_local_props]:
            for key, value in p().items():
                if self._prop_differs(key, value):
                    self._props[key] = value
                    self._changed_props.append(key)
        return len(self._changed_props) > 0

    def prop_changed(self, name):
        return (name in self._changed_props)

    def _get_local_props(self):
        return {}

    def _prop_differs(self, key, value):
        if key not in self._props:
            return True
        prop = self._props[key]
        if prop is value:
            return False
        return prop != value

    def _update_gui(self, parent_updated):
        for name in self._changed_props:
            if name in self._builtin_props:
                self._builtin_props[name](self._props[name])

    @profile_sub("register builtin")
    def _register_builtin(self, name, fn):
        self._builtin_props[name] = profile_sub(f"builtin {name}")(fn)
Events
  1. rliterate2.py
  2. classes
DragEvent = namedtuple("DragEvent", "initial,dx,dy,initiate_drag_drop")
  1. rliterate2.py
  2. classes
SliderEvent = namedtuple("SliderEvent", "value")
  1. rliterate2.py
  2. classes
HoverEvent = namedtuple("HoverEvent", "mouse_inside")
  1. rliterate2.py
  2. classes
ClickEvent = namedtuple("ClickEvent", "show_context_menu")
Props
  1. rliterate2.py
  2. base classes
class Props(Immutable):

    def __init__(self, props, child_props={}):
        self.dependencies = set()
        data = {}
        for name, value in props.items():
            if isinstance(value, Props):
                value.listen(self._create_props_handler(name, value))
                self._set_initial(data, name, value.get())
                self.dependencies.update(value.dependencies)
            elif isinstance(value, PropUpdate):
                self.dependencies.update(
                    value.parse(self._create_prop_update_handler(name, value))
                )
                self._set_initial(data, name, value.eval())