How Syntax Highlighting Actually Works: Grammars, Regex, and Tree-sitter

You're editing a file and the colors suddenly go wrong halfway down — strings turn into comments, keywords turn plain, and nothing fixes itself until you delete a stray quote three lines up. Or you open a file in a less common language and get no color at all. Both are symptoms of the same thing: how syntax highlighting actually works under the hood.

Two engines do this work across the editors you use every day. One is a regex-matched grammar — the TextMate model, used by VS Code and others. The other is a real parser that builds a parse tree — the Tree-sitter model, used by GitHub, Neovim, and increasingly VS Code too. Both solve the same problem, tagging each piece of your text so a theme can color it, but they get there in genuinely different ways.

Quick answer

How syntax highlighting works comes down to pattern matching, just at two different levels. A TextMate grammar runs an ordered list of regex patterns against your text line by line and tags whatever matches with a scope name like keyword.control or string.quoted.double. Tree-sitter instead parses the whole file into a real syntax tree, then runs a query over that tree to tag specific nodes with highlight names like keyword or function. Either way, a theme maps those tags to colors. The real difference is whether the editor is matching text or understanding structure — and that's exactly why highlighting sometimes breaks.

How it actually works

TextMate grammars: regex, line by line

A TextMate grammar is a file — plist or JSON — holding an ordered set of regex patterns written in the Oniguruma dialect, similar to PCRE or Perl regex rather than JavaScript's native engine. Each pattern carries a scope name. The grammar runs one line at a time: for each line, the tokenizer tries its patterns in order, and the first match wins that span of text. Multi-line constructs — a block comment, a triple-quoted string — use paired begin and end patterns that carry state from one line to the next.

This is also where a fenced code block comes in. Naming the language on a code fence is the signal that tells the grammar loader which pattern set to use.

Why highlighting desyncs mid-file

A TextMate grammar carries state line by line, so one bad line can throw off every line after it. An unterminated string, a stray quote inside a comment, a construct the grammar's author never anticipated — any of these can leave the tokenizer stuck in the wrong begin/end state. Regex-only highlighting reflects "a very crude understanding of code, obtained by searching for simple, regular expression patterns," and that's exactly why a single typo can recolor the rest of a file until the next matching quote or brace resets the state.

Tree-sitter: a real parse tree

Tree-sitter takes a different path. It builds an actual syntax tree for your file instead of scanning line by line, and it parses incrementally — keeping the tree current as you edit without ever re-parsing the whole file from scratch. That's what keeps highlighting fast and accurate on a large file as you type.

Highlighting runs through a "highlights" query over that tree. The query captures specific nodes and assigns them highlight names — `keyword`, `function`, `type`, `string` — which a theme maps to colors. It's a structural match, not a text match: the parser already knows which quoted text is a string, so there's no state to desync. Tree-sitter's scope names even track TextMate's conventions (source.js and friends), which is why themes mostly carry over when an editor switches engines underneath you.

When this matters (and when you don't need to think about it)

Most of the time, none of this matters. Highlighting just works. It's worth knowing about when:

• A file highlights correctly up to a point, then goes wrong for the rest of it. Classic regex-grammar symptom — look for an unterminated string, comment, or escape character above the break.
• A language shows no highlighting at all. Both grammar types are hand-written, language by language. If nobody's finished a grammar — or a highlight query, for Tree-sitter — yet, the editor falls back to plain text.
• The same file looks subtly different across two editors. Tree-sitter's default highlight captures are intentionally coarser than a TextMate grammar's nested scopes, so a migrated editor may need an extra translation layer to match the old look.

If you just want your notes and code to read clearly, you don't need to pick an engine. You need an editor whose defaults you trust.

How Carets fits in

Carets is a fast, native notes and code editor for iPhone, iPad, and Mac, and syntax highlighting is one of the things it handles for you by default — no grammar to pick, no theme to debug. Open a scratch file, a config snippet, or a README, and the highlighting is just there.

That matters because Carets keeps everything — Markdown, plain text, and code — in one place, organized into files, projects, and tags, instead of scattered across apps. If you're already keeping snippets in plain Markdown files, Carets adds syntax highlighting and fast native performance on iOS and Mac, without asking you to understand grammars or parse trees to get there. Your files stay plain text, so they're portable no matter what you use next.

Carets is available on iPhone, iPad, and Mac — find it on the App Store and see your code and config highlighted the moment you open a file.

Frequently Asked Questions

What's the difference between a TextMate grammar and a Tree-sitter grammar?

A TextMate grammar is an ordered list of Oniguruma regex patterns matched line by line, each pattern tagged with a scope name. A Tree-sitter grammar builds an actual parse tree for the whole file, then a separate highlight query captures nodes in that tree and assigns them highlight names. One matches text; the other understands structure.

Why does syntax highlighting sometimes break halfway through a file?

Regex-based grammars track state line by line with begin/end pairs for things like strings and block comments. An unterminated string, a stray quote, or a construct the grammar's author didn't anticipate can leave the tokenizer in the wrong state — so every line after that point gets tagged with the wrong scope until something resets it.

Why is a language not supported for syntax highlighting?

Both TextMate and Tree-sitter grammars are hand-written, maintained code — someone has to encode that language's syntax rules and keep them current. If no one's written, or finished, a grammar for a language, or a highlight query for it in Tree-sitter's case, an editor falls back to plain text or a close approximation.

Does syntax highlighting slow down large files?

It can, with older, naive regex scanning — reapplying every pattern to every line on each keystroke gets expensive fast. Tree-sitter's incremental parsing avoids most of that cost by reusing the existing parse tree and only re-parsing the part of the file that changed.

Conclusion

Syntax highlighting is either a regex match against your text or a query against a real parse tree. Knowing which one is running explains both the breakage and the gaps. As more editors adopt real parsers, "my language isn't highlighted" should get rarer. It won't disappear, though — someone still has to sit down and write the grammar.