lib/syn/src/root.zig

daab053ee43316e1809a84551d573ddd1e5bf3d2

 1 //! The package marks up source code for display one line at a time: for each line it records which
 2 //! byte ranges are keywords, strings, comments or other roles. Colors, themes and terminal
 3 //! formatting stay with the caller. The package reads the caller's text in place and keeps no copy
 4 //! of it.
 5 //!
 6 //! A program that shows code to a person, in a terminal or an editor pane, wants each token marked
 7 //! by its role, so the reader can tell a string from a comment at a glance. Such a program meets
 8 //! many languages, forty here, and often knows a language only by a file name or a short name such
 9 //! as `py` or `bash`. Text reaches the program a line at a time, and a comment or string opened on
10 //! one line can run on into the lines after it.
11 //!
12 //! A line cannot be marked correctly without knowing whether an earlier line left a comment or
13 //! string open, so that fact has to travel from one line to the next. That fact belongs to one
14 //! language: a comment left open in C-like code means nothing to a following line of Python, and
15 //! carrying it across would mark the Python wrongly. A program with a fixed memory budget wants the
16 //! room for one line's marks set aside in advance, but lines have no fixed length. Dropping the
17 //! marks of a line that does not fit cannot drop that line's effect on what stays open, because the
18 //! line can itself open or close a comment.
19 //!
20 //! The highlighting of [bat](https://github.com/sharkdp/bat) and
21 //! [syntect](https://github.com/trishume/syntect), and the language modes of
22 //! [CodeMirror](https://codemirror.net/), are the sources of three ideas the package keeps: a
23 //! syntax state carried from line to line, processing text one line at a time, and marks that name
24 //! a token's role and leave its look to the renderer.
25 //!
26 //! One call scans one line with the scanner for its family of languages and records each marked
27 //! range (*span*) as a start byte, an end byte and a role. The scanners are hand-written byte
28 //! scanners, one per family of languages, and they build no syntax tree. The caller keeps a small
29 //! value between calls (*continuation state*), `State`. `State` records the open comment or string
30 //! (its mode, `State.Mode`) together with the language that opened it, and a line in any other
31 //! language starts fresh. The caller creates one buffer for spans (*span storage*), `SpanStorage`,
32 //! sized by the longest line whose spans it wants kept, and no scan allocates after that. A longer
33 //! line (*discarded line*) is still scanned in full, so what it leaves open stays exact, but its
34 //! spans are dropped and the call reports the line as discarded. The two properties about carried
35 //! state and discarded lines are proved for a model of the scanner in `verification/syn`, and
36 //! property tests check the Zig scanners against that model.
37 
38 pub const language = @import("language.zig");
39 pub const span = @import("span/root.zig");
40 pub const scan = @import("scan/root.zig");
41 pub const state = @import("state.zig");
42 
43 pub const Language = language.Language;
44 pub const Kind = span.Kind;
45 pub const Span = span.Span;
46 pub const SpanLimits = span.Limits;
47 pub const SpanCapacity = span.Capacity;
48 pub const SpanDeriveError = span.DeriveError;
49 pub const SpanMaterialization = span.Materialization;
50 pub const SpanStatus = span.Status;
51 pub const SpanStorage = span.Storage;
52 pub const State = state.State;
53 pub const Mode = state.Mode;
54 
55 pub const name = language.name;
56 pub const fromName = language.fromName;
57 pub const fromPath = language.fromPath;
58 pub const highlight = scan.highlight;
59 pub const highlightLine = scan.highlightLine;