RLiterate

RLiterate is a tool for authoring documents. You can think of it as a programmer's version of a word processor. This book describes RLiterate and also includes the complete implementation of the program presented in small pieces.

Screenshot of RLiterate editing itself.

A tour of RLiterate

This chapter gives an overview of what types of document you can create with RLiterate and presents the main features of the program.

Document model

RLiterate documents have pages organized in a hierarchy. Pages have a title and paragraphs. Paragraphs can be of different types. The different paragraph types is what make RLiterate documents special. Code paragraphs for example enable literate programming by allowing chunks of code to be defined and then be automatically assembled into the final source file. Text paragraphs are used for writing prose. RLiterate documents can also be exported to different formats for display in different mediums.

Main GUI

The main GUI consists of the table of contents and the workspace.

The main GUI window.

The table of contents shows the outline of the document, and from it, pages can be opened in the workspace. Each page is drawn with a border in the workspace.

Literate programming

TODO: Explain how code paragraphs enable literate programming.

Reading tool

RLiterate is not only a tool for authoring documents, but also a tool for reading documents. The following features support that.

Hoisting a page in the table of contents allows you to focus on a subset of the document.

Page 1 hoisted.

Openining a page and all immediate children allows you to read a subset of the document breath first. It's like reading only the first section of each chapter in an entire book.

Demo page with its children opened. Notice that Sub page is not an immediate child and therefore not shown.

Getting started

This chapter gives practical advice for using RLiterate on your computer.

Status

RLiterate is currently a prototype. Experimentation is encouraged, but it may have serious bugs.

Installing

The source code for RLiterate is in a Git repository hosted on Github:

https://github.com/rickardlindberg/rliterate

Assuming you have Git installed, you can clone the repository like this:

git clone https://github.com/rickardlindberg/rliterate.git

Before you can run RLiterate, you need Python with the following libraries installed:

Here is how they can be installed on a Fedora system:

dnf install python wxPython python2-pygments

Once they are installed, you can run RLiterate like this:

python rliterate.py rliterate.rliterate

This will open the RLiterate document. Enter a different filename as last parameter to create a new document.

Diffing *.rliterate files

When *.rliterate files are version controlled, the textual diff is hard to read. This problem can be sovled in Git by defining a textconv command that converts the *.rliterate file to text that is suitable for diffing. The --diff option to RLiterate outputs a file in a diff friendly format.

First associate *.rliterate files with rliterate using a Git attributes file. For example in $HOME/.config/git/attributes:

*.rliterate diff=rliterate

Then define the textconv command in your git config ($HOME/.gitconfig):

[diff "rliterate"]
      textconv=bash -c 'python $RLITERATE_ROOT/rliterate.py "$0" --diff'

Background

This chapter explains why RLiterate exists and how it came about.

The idea

I started to think about what would become RLiterate when I read the paper Active Essays on the Web. In it they talk about embedding code in documents that the reader can interact with. They also mention Literate programming as having a related goal.

At the time I was working on a program that I thought would be nice to write in this way. I wanted to write an article about the program and have the code for the program embedded in the article. I could have used a literate programming tool for this, but the interactive aspect of active essays made me think that a tool would be much more powerful if the document could be edited "live", similar to WYSIWYG editors. Literate programming tools I were aware of worked by editing plain text files with a special syntax for code and documentation blocks, thus lacking the interactive aspect.

The prototype

So I decided to build a prototype to learn what such a program might be like.

First I came up with a document model where pages were organized in a hierarchy. Each page had paragraphs that could be of different types. This idea was stolen from Smallest Federated Wiki. The code paragraph would allow for literate programming, while the text paragraph would be for prose. I also envisioned other paragraph types that would allow for more interaction. Perhaps one paragraph type could be Graphviz code, and when edited, a generated graph would appear instead of the code.

After coming up with a document model, I implement a GUI that would allow editing such documents. This GUI had to be first class as it would be the primary way author (and read) documents.

The first version of the GUI was not first class though. I started to experiment with a workspace for showing pages.

First GUI prototype of workspace.

Then I contintued with a table of contents. At this point it was based on a tree control present in wxPython.

First GUI prototype of table of contents. Workspace pages also have titles.

All the data so far had been static and hard coded. I started to flesh out the document model and implement GUI controls for manipulating the document. Here is the add button on pages.

Add button that creates the factory paragraph.

Drag and drop was implemented fairly early. I wanted good visual feedback. This was part of the first class idea.

Drag and drop with red divider line indicating drop point.

Drag and drop with good visual feedback was hard to achieve with the tree control from wxWidgets, so at this point I rewrote the table of contents widget as a custom widget.

Custom table of contents that allows drag and drop and custom rendering.

I added more operations for adding and deleting pages.

Context menu that allows adding and deleting pages.

Then finally I added a second paragraph type: the code paragraph type that would enable literate programming.

Code paragraphs showing literate programming in action.

At this point I had all the functionality in place for writing documents and embedding code in them. I imported all the code into an RLiterate document (previously it was written as a single Python file) and started extracting pieces and adding prose to explain the program. This was a tremendously rewarding experience. RLiterate was now bootstrapped.

As I continued to improve the RLiterate document, I noticed features I was lacking. I added them to the program and iterated.

Why literate programming?

To me, the most central idea in literate programming is that we should write programs for other humans. Only secondary for the machine. I find code easier to understand if I can understand it in isolated pieces. One example where it is difficult to isolate a piece without literate programming is test code. Usually the test code is located in one file, and the implementation in another. To fully understand the piece it is useful to both read the tests and the implementation. To do this we have to find this related information in two unrelated files. I wrote about this problem in 2013 in Related things are not kept together. Literate programming allows us to present the test and the implementation in the same place, yet have the different pieces written to different files. The compiler might require that they are in separate files, but with literate programming, we care first about the other human that will read our code, and only second about the compiler.

Another argument for literate programming is to express the "why". Why is this code here? Timothy Daly talks about it in his talk Literate Programming in the Large. He also argues that programmers must change the mindset from wring a program to writing a book. Some more can be read in Example of Literate Programming in HTML.

Some more resources about literate programming:

Similar tools

When researching ideas for RLiterate, I stumbled across ProjecturED. It is similar to RLiterate in the sense that it is an editor for more structured documents. Not just text. The most interesting aspect for me was that a variable name exists in one place, but can be rendered in multiple. So a rename is really simple. With RLiterate, you have to do a search and replace. But with ProjecturED you just change the name and it replicates everywhere. This is an attractive feature and is made possible by the different document model.

Implementation

This chapter gives a complete description of the implementation of RLiterate presented in small pieces.

Files

RLiterate is implemented in Python. The implementation consists of three files: one for the program, one for the test code, and a Makefile for various operations.

  1. rliterate.py
# This file is extracted from rliterate.rliterate.
# DO NOT EDIT MANUALLY!

<<imports>>
<<constants>>
<<base base classes>>
<<base classes>>
<<classes>>
<<functions>>

if __name__ == "__main__":
    <<entry point>>
  1. test_rliterate.py
# This file is extracted from rliterate.rliterate.
# DO NOT EDIT MANUALLY!

<<imports>>
<<fixtures>>
<<test cases>>
  1. Makefile
# This file is extracted from rliterate.rliterate.
# DO NOT EDIT MANUALLY!

<<rules>>

The remainder of this chapter will fill in the sections in those files to create the complete implementation.

Document model

An RLiterate document is a JSON object (hereafter called the document dict) stored in a file. The Document class provides a friendly interface for working with RLiterate documents.

The Document class is initialized with a documetn dict (or None to create a new document):

  1. rliterate.py
  2. classes
class Document(Observable):

    def __init__(self, document_dict=None):
        Observable.__init__(self)
        self._load(document_dict)

    <<Document>>

The default constructor does not know anything about files. It just creates an in-memory document. The from_file constructor reads a document from file and also writes it back to file when it changes:

  1. rliterate.py
  2. classes
  3. Document
@classmethod
def from_file(cls, path):
    if os.path.exists(path):
        document = cls(load_json_from_file(path))
    else:
        document = cls()
    document.listen(lambda:
        write_json_to_file(
            path,
            document.read_only_document_dict
        )
    )
    return document

Loading is the same in both cases and involves

  1. Creating a new empty document dict if none was passed
  2. Convert the document dict to the latest format
  3. Wrap it
  4. Put it in a history object for undo/redo
  1. rliterate.py
  2. classes
  3. Document
def _load(self, document_dict):
    self._history = History(
        DocumentDictWrapper(
            self._convert_to_latest(
                self._empty_page()
                if document_dict is None else
                document_dict
            )
        ),
        size=UNDO_BUFFER_SIZE
    )
  1. rliterate.py
  2. constants
UNDO_BUFFER_SIZE = 10

The current document dict can be accessed via the read_only_document_dict property. But it should be considered read only. For undo/redo to work properly, all edits must go through modify.

  1. rliterate.py
  2. classes
  3. Document
@property
def read_only_document_dict(self):
    return self._history.value

Modifying

There are only three ways in which a document should be modified:

  1. Via the modify method
  2. Via undo
  3. Via redo
  1. rliterate.py
  2. classes
  3. Document
@contextlib.contextmanager
def modify(self, name):
    with self.notify():
        with self._history.new_value(name) as value:
            yield value

def get_undo_operation(self):
    def undo():
        with self.notify():
            self._history.back()
    if self._history.can_back():
        return (self._history.back_name(), undo)

def get_redo_operation(self):
    def redo():
        with self.notify():
            self._history.forward()
    if self._history.can_forward():
        return (self._history.forward_name(), redo)

Document dict wrapper

The document dict wrapper is a helper for working with a document dict. It mainly maintains caches.

A document dict has the following structure:

{
  "root_page": {
    "id": "106af6f8665c45e8ab751993a6abc876",
    "title": "Root",
    "paragraphs": [
      {
        "id": "8f0a9f84821540e89d7c9ca93ed0fbe7",
        "type": "text",
        "fragments": [
          {"text": "This ", "type": "text" },
          {"text": "is",    "type": "strong" },
          {"text": "cool",  "type": "emphasis" }
        ]
      },
      ...
    ],
    "children": [ ... ]
  },
  "variables": { ... }
}

The root object represents a document. It has a root page and variables. The root page has a unique id, a title, paragraphs, and child pages. Paragraphs also have unique ids and different attributes depending on type. (The text paragraph has a list of text fragments with different styles.)

  1. rliterate.py
  2. classes
class DocumentDictWrapper(dict):

    def __init__(self, document_dict):
        dict.__init__(self, document_dict)
        self._pages = {}
        self._parent_pages = {}
        self._paragraphs = {}
        self._cache_page(self["root_page"])

    def _cache_page(self, page, parent_page=None):
        self._pages[page["id"]] = page
        self._parent_pages[page["id"]] = parent_page
        for paragraph in page["paragraphs"]:
            self._paragraphs[paragraph["id"]] = paragraph
        for child in page["children"]:
            self._cache_page(child, page)

    def add_page_dict(self, page_dict, parent_id=None):
        page_dict = copy.deepcopy(page_dict)
        parent_page = self._pages[parent_id]
        parent_page["children"].append(page_dict)
        self._pages[page_dict["id"]] = page_dict
        self._parent_pages[page_dict["id"]] = parent_page

    def get_page_dict(self, page_id=None):
        if page_id is None:
            page_id = self["root_page"]["id"]
        return self._pages.get(page_id, None)

    def get_parent_page_dict(self, page_id):
        return self._parent_pages.get(page_id, None)

    def delete_page_dict(self, page_id):
        if page_id == self["root_page"]["id"]:
            return
        page = self._pages[page_id]
        parent_page = self._parent_pages[page_id]
        index = index_with_id(parent_page["children"], page_id)
        parent_page["children"].pop(index)
        self._pages.pop(page_id)
        self._parent_pages.pop(page_id)
        for child in reversed(page["children"]):
            parent_page["children"].insert(index, child)
            self._parent_pages[child["id"]] = parent_page

    def update_page_dict(self, page_id, data):
        self._pages[page_id].update(copy.deepcopy(data))

    def move_page_dict(self, page_id, parent_page_id, before_page_id):
        if page_id == before_page_id:
            return
        parent = self._pages[parent_page_id]
        while parent is not None:
            if parent["id"] == page_id:
                return
            parent = self._parent_pages[parent["id"]]
        parent = self._parent_pages[page_id]
        page = parent["children"].pop(index_with_id(parent["children"], page_id))
        new_parent = self._pages[parent_page_id]
        self._parent_pages[page_id] = new_parent
        if before_page_id is None:
            new_parent["children"].append(page)
        else:
            new_parent["children"].insert(
                index_with_id(new_parent["children"], before_page_id),
                page
            )

    def paragraph_dict_iterator(self):
        return self._paragraphs.values()

    def add_paragraph_dict(self, paragraph_dict, page_id, before_id):
        paragraph_dict = copy.deepcopy(paragraph_dict)
        paragraphs = self._pages[page_id]["paragraphs"]
        if before_id is None:
            paragraphs.append(paragraph_dict)
        else:
            paragraphs.insert(index_with_id(paragraphs, before_id), paragraph_dict)
        self._paragraphs[paragraph_dict["id"]] = paragraph_dict

    def delete_paragraph_dict(self, page_id, paragraph_id):
        paragraphs = self._pages[page_id]["paragraphs"]
        paragraphs.pop(index_with_id(paragraphs, paragraph_id))
        return self._paragraphs.pop(paragraph_id)

    def move_paragraph_dict(self, page_id, paragraph_id, target_page, before_paragraph):
        if (page_id == target_page and
            paragraph_id == before_paragraph):
            return
        self.add_paragraph_dict(
            self.delete_paragraph_dict(page_id, paragraph_id),
            target_page,
            before_paragraph
        )

    def update_paragraph_dict(self, paragraph_id, data):
        self._paragraphs[paragraph_id].update(copy.deepcopy(data))
  1. rliterate.py
  2. functions
def index_with_id(items, item_id):
    for index, item in enumerate(items):
        if item["id"] == item_id:
            return index

Page

  1. rliterate.py
  2. classes
  3. Document
def add_page(self, title="New page", parent_id=None):
    with self.modify("Add page") as document_dict:
        document_dict.add_page_dict(self._empty_page(), parent_id=parent_id)

def _empty_page(self):
    return {
        "id": genid(),
        "title": "New page...",
        "children": [],
        "paragraphs": [],
    }

def iter_pages(self):
    def iter_pages(page):
        yield page
        for child in page.children:
            for sub_page in iter_pages(child):
                yield sub_page
    return iter_pages(self.get_page())
  1. rliterate.py
  2. classes
  3. Document
def get_page(self, page_id=None):
    page_dict = self.read_only_document_dict.get_page_dict(page_id)
    if page_dict is None:
        return None
    return Page(self, page_dict)

def get_parent_page(self, page_id):
    page_dict = self.read_only_document_dict.get_parent_page_dict(page_id)
    if page_dict is None:
        return None
    return Page(self, page_dict)
  1. rliterate.py
  2. classes
class Page(object):

    def __init__(self, document, page_dict):
        self._document = document
        self._page_dict = page_dict

    @property
    def parent(self):
        return self._document.get_parent_page(self.id)

    @property
    def full_title(self):
        return " / ".join(page.title for page in self.chain)

    @property
    def chain(self):
        result = []
        page = self
        while page is not None:
            result.insert(0, page)
            page = page.parent
        return result

    def iter_code_fragments(self):
        for paragraph in self.paragraphs:
            for fragment in paragraph.iter_code_fragments():
                yield fragment

    def iter_text_fragments(self):
        for paragraph in self.paragraphs:
            for fragment in paragraph.iter_text_fragments():
                yield fragment

    @property
    def id(self):
        return self._page_dict["id"]

    @property
    def title(self):
        return self._page_dict["title"]

    def set_title(self, title):
        with self._document.modify("Change title") as document_dict:
            document_dict.update_page_dict(self.id, {"title": title})

    @property
    def paragraphs(self):
        return [
            Paragraph.create(
                self._document,
                self,
                paragraph_dict,
                next_paragraph_dict["id"] if next_paragraph_dict is not None else None
            )
            for paragraph_dict, next_paragraph_dict
            in zip(
                self._page_dict["paragraphs"],
                self._page_dict["paragraphs"][1:]+[None]
            )
        ]

    @property
    def children(self):
        return [
            Page(self._document, child_dict)
            for child_dict
            in self._page_dict["children"]
        ]

    def delete(self):
        with self._document.modify("Delete page") as document_dict:
            document_dict.delete_page_dict(self.id)

    def move(self, parent_page_id, before_page_id):
        with self._document.modify("Move page") as document_dict:
            document_dict.move_page_dict(self.id, parent_page_id, before_page_id)

Paragraph

  1. rliterate.py
  2. classes
  3. Document
def add_paragraph(self, page_id, before_id=None, paragraph_dict={"type": "factory"}):
    with self.modify("Add paragraph") as document_dict:
        document_dict.add_paragraph_dict(
            dict(paragraph_dict, id=genid()),
            page_id,
            before_id=before_id
        )

def get_paragraph(self, page_id, paragraph_id):
    for paragraph in self.get_page(page_id).paragraphs:
        if paragraph.id == paragraph_id:
            return paragraph
  1. rliterate.py
  2. classes
class Paragraph(object):

    @staticmethod
    def create(document, page, paragraph_dict, next_id):
        return {
            "text": TextParagraph,
            "quote": QuoteParagraph,
            "list": ListParagraph,
            "code": CodeParagraph,
            "image": ImageParagraph,
            "expanded_code": ExpandedCodeParagraph,
        }.get(paragraph_dict["type"], Paragraph)(document, page, paragraph_dict, next_id)

    def __init__(self, document, page, paragraph_dict, next_id):
        self._document = document
        self._page = page
        self._paragraph_dict = paragraph_dict
        self._next_id = next_id

    @property
    def id(self):
        return self._paragraph_dict["id"]

    @property
    def next_id(self):
        return self._next_id

    @property
    def type(self):
        return self._paragraph_dict["type"]

    @contextlib.contextmanager
    def multi_update(self):
        with self._document.modify("Edit paragraph"):
            yield

    def update(self, data):
        with self._document.modify("Edit paragraph") as document_dict:
            document_dict.update_paragraph_dict(self.id, data)

    def delete(self):
        with self._document.modify("Delete paragraph") as document_dict:
            document_dict.delete_paragraph_dict(self._page.id, self.id)

    def move(self, target_page, before_paragraph):
        with self._document.modify("Move paragraph") as document_dict:
            document_dict.move_paragraph_dict(self._page.id, self.id, target_page, before_paragraph)

    def duplicate(self):
        with self._document.modify("Duplicate paragraph") as document_dict:
            document_dict.add_paragraph_dict(
                dict(copy.deepcopy(self._paragraph_dict), id=genid()),
                page_id=self._page.id,
                before_id=self.next_id
            )

    @property
    def filename(self):
        return "paragraph.txt"

    def iter_code_fragments(self):
        return iter([])

    def iter_text_fragments(self):
        return iter([])
Text
  1. rliterate.py
  2. classes
class TextParagraph(Paragraph):

    @property
    def fragments(self):
        return TextFragment.create_list(self._document, self._paragraph_dict["fragments"])

    @property
    def tokens(self):
        return [x.token for x in self.fragments]

    def get_text_index(self, index):
        return self._text_version.get_selection(index)[0]

    @property
    def text_version(self):
        return self._text_version.text

    @property
    def _text_version(self):
        text_version = TextVersion()
        for fragment in self.fragments:
            fragment.fill_text_version(text_version)
        return text_version

    @text_version.setter
    def text_version(self, value):
        self.update({"fragments": text_to_fragments(value)})

    def iter_text_fragments(self):
        return iter(self.fragments)
  1. rliterate.py
  2. functions
def fragments_to_text(fragments):
    text_version = TextVersion()
    for fragment in fragments:
        fragment.fill_text_version(text_version)
    return text_version.text


def text_to_fragments(text):
    return TextParser().parse(text)
  1. rliterate.py
  2. classes
class TextParser(object):

    SPACE_RE = re.compile(r"\s+")
    PATTERNS = [
        (
            re.compile(r"\*\*(.+?)\*\*", flags=re.DOTALL),
            lambda parser, match: {
                "type": "strong",
                "text": match.group(1),
            }
        ),
        (
            re.compile(r"\*(.+?)\*", flags=re.DOTALL),
            lambda parser, match: {
                "type": "emphasis",
                "text": match.group(1),
            }
        ),
        (
            re.compile(r"``(.+?)``", flags=re.DOTALL),
            lambda parser, match: {
                "type": "variable",
                "id": match.group(1),
            }
        ),
        (
            re.compile(r"`(.+?)`", flags=re.DOTALL),
            lambda parser, match: {
                "type": "code",
                "text": match.group(1),
            }
        ),
        (
            re.compile(r"\[\[(.+?)(:(.+?))?\]\]", flags=re.DOTALL),
            lambda parser, match: {
                "type": "reference",
                "text": match.group(3),
                "page_id": match.group(1),
            }
        ),
        (
            re.compile(r"\[(.*?)\]\((.+?)\)", flags=re.DOTALL),
            lambda parser, match: {
                "type": "link",
                "text": match.group(1),
                "url": match.group(2),
            }
        ),
    ]

    def parse(self, text):
        text = self._normalise_space(text)
        fragments = []
        partial = ""
        while text:
            result = self._get_special_fragment(text)
            if result is None:
                partial += text[0]
                text = text[1:]
            else:
                match, fragment = result
                if partial:
                    fragments.append({"type": "text", "text": partial})
                    partial = ""
                fragments.append(fragment)
                text = text[match.end(0):]
        if partial:
            fragments.append({"type": "text", "text": partial})
        return fragments

    def _normalise_space(self, text):
        return self.SPACE_RE.sub(" ", text).strip()

    def _get_special_fragment(self, text):
        for pattern, fn in self.PATTERNS:
            match = pattern.match(text)
            if match:
                return match, fn(self, match)
Quote
  1. rliterate.py
  2. classes
class QuoteParagraph(TextParagraph):
    pass
List
  1. rliterate.py
  2. classes
class ListParagraph(Paragraph):

    @property
    def child_type(self):
        return self._paragraph_dict["child_type"]

    @property
    def children(self):
        return [ListItem(self._document, x) for x in self._paragraph_dict["children"]]

    def get_text_index(self, list_and_fragment_index):
        return self._text_version.get_selection(list_and_fragment_index)[0]

    @property
    def text_version(self):
        return self._text_version.text

    @property
    def _text_version(self):
        def list_item_to_text(text_version, child_type, item, indent=0, index=0):
            text_version.add("    "*indent)
            if child_type == "ordered":
                text_version.add("{}. ".format(index+1))
            else:
                text_version.add("* ")
            for fragment in item.fragments:
                fragment.fill_text_version(text_version)
            text_version.add("\n")
            for index, child in enumerate(item.children):
                with text_version.index(index):
                    list_item_to_text(text_version, item.child_type, child, index=index, indent=indent+1)
        text_version = TextVersion()
        for index, child in enumerate(self.children):
            with text_version.index(index):
                list_item_to_text(text_version, self.child_type, child, index=index)
        return text_version

    @text_version.setter
    def text_version(self, value):
        child_type, children = LegacyListParser(value).parse_items()
        self.update({
            "child_type": child_type,
            "children": children
        })

    def iter_text_fragments(self):
        for item in self.children:
            for fragment in item.iter_text_fragments():
                yield fragment
  1. rliterate.py
  2. classes
class ListItem(object):

    def __init__(self, document, item_dict):
        self._document = document
        self._item_dict = item_dict

    @property
    def fragments(self):
        return TextFragment.create_list(self._document, self._item_dict["fragments"])

    @property
    def child_type(self):
        return self._item_dict["child_type"]

    @property
    def children(self):
        return [ListItem(self._document, x) for x in self._item_dict["children"]]

    @property
    def tokens(self):
        return [x.token for x in self.fragments]

    def iter_text_fragments(self):
        for fragment in self.fragments:
            yield fragment
        for child in self.children:
            for fragment in child.iter_text_fragments():
                yield fragment
Code
  1. rliterate.py
  2. classes
class CodeParagraph(Paragraph):

    <<CodeParagraph>>
Path
  1. rliterate.py
  2. classes
  3. CodeParagraph
@property
def path(self):
    return Path(
        [x for x in self._paragraph_dict["filepath"] if x],
        [x for x in self._paragraph_dict["chunkpath"] if x]
    )

@path.setter
def path(self, path):
    self.update({
        "filepath": copy.deepcopy(path.filepath),
        "chunkpath": copy.deepcopy(path.chunkpath),
    })
  1. rliterate.py
  2. classes
  3. CodeParagraph
@property
def filename(self):
    return self.path.filename
  1. rliterate.py
  2. classes
class Path(object):

    @classmethod
    def from_text_version(cls, text):
        try:
            filepath_text, chunkpath_text = text.split(" // ", 1)
        except:
            filepath_text = text
            chunkpath_text = ""
        return cls(
            filepath_text.split("/") if filepath_text else [],
            chunkpath_text.split("/") if chunkpath_text else [],
        )

    @property
    def text_version(self):
        if self.has_both():
            sep = " // "
        else:
            sep = ""
        return "{}{}{}".format(
            "/".join(self.filepath),
            sep,
            "/".join(self.chunkpath)
        )

    @property
    def text_start(self):
        return self.text_end - len(self.last)

    @property
    def text_end(self):
        return len(self.text_version)

    def extend_chunk(self, chunk):
        return Path(
            copy.deepcopy(self.filepath),
            copy.deepcopy(self.chunkpath)+copy.deepcopy(chunk)
        )

    @property
    def filename(self):
        return self.filepath[-1] if self.filepath else ""

    @property
    def last(self):
        if len(self.chunkpath) > 0:
            return self.chunkpath[-1]
        elif len(self.filepath) > 0:
            return self.filepath[-1]
        else:
            return ""

    @property
    def is_empty(self):
        return self.length == 0

    @property
    def length(self):
        return len(self.chunkpath) + len(self.filepath)

    def __init__(self, filepath, chunkpath):
        self.filepath = filepath
        self.chunkpath = chunkpath

    def is_prefix(self, other):
        if len(self.chunkpath) > 0:
            return self.filepath == other.filepath and self.chunkpath == other.chunkpath[:len(self.chunkpath)]
        else:
            return self.filepath == other.filepath[:len(self.filepath)]

    def has_both(self):
        return len(self.filepath) > 0 and len(self.chunkpath) > 0

    @property
    def filepaths(self):
        for index in range(len(self.filepath)):
            yield (
                self.filepath[index],
                Path(self.filepath[:index+1], [])
            )

    @property
    def chunkpaths(self):
        for index in range(len(self.chunkpath)):
            yield (
                self.chunkpath[index],
                Path(self.filepath[:], self.chunkpath[:index+1])
            )
  1. rliterate.py
  2. classes
  3. Document
def rename_path(self, path, name):
    with self.modify("Rename path") as document_dict:
        for p in document_dict.paragraph_dict_iterator():
            if p["type"] == "code":
                filelen = len(p["filepath"])
                chunklen = len(p["chunkpath"])
                if path.is_prefix(Path(p["filepath"], p["chunkpath"])):
                    if path.length > filelen:
                        p["chunkpath"][path.length-1-filelen] = name
                    else:
                        p["filepath"][path.length-1] = name
                else:
                    for f in p["fragments"]:
                        if f["type"] == "chunk":
                            if path.is_prefix(Path(p["filepath"], p["chunkpath"]+f["path"])):
                                f["path"][path.length-1-filelen-chunklen] = name
Fragments
  1. rliterate.py
  2. classes
  3. CodeParagraph
@property
def fragments(self):
    return CodeFragment.create_list(
        self._document,
        self,
        self._paragraph_dict["fragments"]
    )

def iter_code_fragments(self):
    return iter(self.fragments)
  1. rliterate.py
  2. classes
class CodeFragment(object):

    @staticmethod
    def create_list(document, code_paragraph, code_fragment_dicts):
        return [
            CodeFragment.create(document, code_paragraph, code_fragment_dict)
            for code_fragment_dict
            in code_fragment_dicts
        ]

    @staticmethod
    def create(document, code_paragraph, code_fragment_dict):
        return {
            "variable": VariableCodeFragment,
            "chunk": ChunkCodeFragment,
            "code": CodeCodeFragment,
            "tabstop": TabstopCodeFragment,
        }.get(code_fragment_dict["type"], CodeFragment)(document, code_paragraph, code_fragment_dict)

    def __init__(self, document, code_paragraph, code_fragment_dict):
        self._document = document
        self._code_paragraph = code_paragraph
        self._code_fragment_dict = code_fragment_dict

    @property
    def type(self):
        return self._code_fragment_dict["type"]

They can be converted to text for editing:

  1. rliterate.py
  2. classes
  3. CodeParagraph
@property
def text_version(self):
    self._variable_map = {}
    text_version = TextVersion()
    for fragment in self.fragments:
        fragment.fill_text_version(text_version)
    return text_version.text

def add_variable_map(self, name, id_):
    entry = name
    index = 1
    while entry in self._variable_map and self._variable_map[entry] != id_:
        entry = "{}{}".format(name, index)
        index += 1
    self._variable_map[entry] = id_
    return entry

And converted back again from text:

  1. rliterate.py
  2. classes
  3. CodeParagraph
@text_version.setter
def text_version(self, value):
    with self.multi_update():
        self.update({
            "fragments": self._parse(value)
        })

def _parse(self, value):
    self._parsed_fragments = []
    self._parse_buffer = ""
    for line in value.splitlines():
        match = re.match(self._chunk_fragment_re(), line)
        if match:
            self._parse_clear()
            body_match = re.match(r"^(.*?)(, blank_lines_before=(\d+))?$", match.group(2))
            if body_match.group(2):
                blank_lines_before = int(body_match.group(3))
            else:
                blank_lines_before = 0
            self._parsed_fragments.append({
                "type": "chunk",
                "path": body_match.group(1).split("/"),
                "prefix": match.group(1),
                "blank_lines_before": blank_lines_before,
            })
        else:
            while line:
                variable_match = re.match(self._variable_fragment_re(), line)
                tabstop_match = re.match(self._tabstop_fragment_re(), line)
                if variable_match:
                    self._parse_clear()
                    self._parsed_fragments.append({
                        "type": "variable",
                        "id": self._get_variable_id(variable_match.group(1))
                    })
                    line = line[len(variable_match.group(0)):]
                elif tabstop_match:
                    self._parse_clear()
                    self._parsed_fragments.append({
                        "type": "tabstop",
                        "index": int(tabstop_match.group(1))
                    })