tiny.syn

Overview · API · Code relationships · Verification · Audit

Overview

The package marks up source code for display one line at a time: for each line it records which byte ranges are keywords, strings, comments or other roles. Colors, themes and terminal formatting stay with the caller. The package reads the caller's text in place and keeps no copy of it.

A program that shows code to a person, in a terminal or an editor pane, wants each token marked by its role, so the reader can tell a string from a comment at a glance. Such a program meets many languages, forty here, and often knows a language only by a file name or a short name such as py or bash. Text reaches the program a line at a time, and a comment or string opened on one line can run on into the lines after it.

A line cannot be marked correctly without knowing whether an earlier line left a comment or string open, so that fact has to travel from one line to the next. That fact belongs to one language: a comment left open in C-like code means nothing to a following line of Python, and carrying it across would mark the Python wrongly. A program with a fixed memory budget wants the room for one line's marks set aside in advance, but lines have no fixed length. Dropping the marks of a line that does not fit cannot drop that line's effect on what stays open, because the line can itself open or close a comment.

The highlighting of bat and syntect, and the language modes of CodeMirror, are the sources of three ideas the package keeps: a syntax state carried from line to line, processing text one line at a time, and marks that name a token's role and leave its look to the renderer.

One call scans one line with the scanner for its family of languages and records each marked range (span) as a start byte, an end byte and a role. The scanners are hand-written byte scanners, one per family of languages, and they build no syntax tree. The caller keeps a small value between calls (continuation state), State. State records the open comment or string (its mode, State.Mode) together with the language that opened it, and a line in any other language starts fresh. The caller creates one buffer for spans (span storage), SpanStorage, sized by the longest line whose spans it wants kept, and no scan allocates after that. A longer line (discarded line) is still scanned in full, so what it leaves open stays exact, but its spans are dropped and the call reports the line as discarded. The two properties about carried state and discarded lines are proved for a model of the scanner in verification/syn, and property tests check the Zig scanners against that model.

Definitions

Actions

Public operations.

• name: A caller that shows or stores a language uses this short name, and fromName reads it back.
• highlightLine: Marks up one line of a document and updates state with what the line leaves open.
• fromPath: A caller that opens a file picks the language from its path.
• fromName: A caller turns a short name, such as zig in the README example or bash, into a tag.
• highlight: Marks up one line from a fresh continuation state.

Types and contracts

Public types and contracts.

• Language: A caller passes one tag to highlight or highlightLine to choose the scanner for a line.
• Mode: One kind of comment or string that is still open at the end of a line.
• State: The continuation state: what the last scanned line left open.
• Kind: The role of the text one span covers.
• Span: One marked byte range of a line and its role.
• SpanCapacity: The sizes that follow from one Limits value, computed without allocating.
• SpanDeriveError: The one way sizing can fail.
• SpanLimits: The one limit the caller chooses for span storage.
• SpanMaterialization: What happened to the spans of one line.
• SpanStatus: A caller reads it from Storage.status to check sizes and to count dropped lines, as the README example and the tests do.
• SpanStorage: A caller creates one before scanning and passes it to every highlight and highlightLine call.

Namespaces

Public namespaces.

• state: What one line of a document leaves open for the next: nothing, or an unfinished comment or string together with the language that opened it.
• scan: The namespace holds the calls that mark up text, one for a line on its own and one for a line inside a longer document.
• span: The namespace holds what a scan produces: the marked byte ranges of a line, the buffer that receives them, and the arithmetic that sizes that buffer.
• language: Forty languages the package can mark up, and mappings from a file path or a short name to one of them.

Code relationships

Direct static dependencies extracted from parsed source by semantic graph analysis.

Uses: tiny.hypothesis, tiny.machine
Used by: None

Verification

No verification records are cataloged for this module in this build.

Audit