Parsing With Markdoc.parse

The problem: before you can transform anything, you need a tree

Lesson 1 treated Markdoc.parse() as a black box, string in, Ast out. This lesson opens that box: parsing is actually two layers, a flat token stream underneath, and a nested Ast tree built on top of it. Understanding both matters once you start writing custom tags and node overrides later in this course, because a tag's transform function receives Ast nodes, not tokens.

Tokenizer: the flat layer

Markdoc.Tokenizer (borrowed from markdown-it's tokenizer) turns a source string into a flat list of open/close/inline tokens, it doesn't know about nesting, a heading_open token and its matching heading_close token are just two entries in a flat array, the nesting is implied by their order, not by actual tree structure.

Ast: the nested layer Markdoc.parse builds for you

Markdoc.parse(source) runs the tokenizer internally, then builds an actual tree on top: a heading Ast node with a text node as its descendant, instead of two separate open/close tokens. This is the shape every later stage (transform, validate) actually consumes, Ast nodes have .type, .attributes, and .children, and can be walked recursively, which is exactly what this lesson's describeNode() does.

The code, piece by piece

const tokenizer = new Markdoc.Tokenizer();
const tokens = tokenizer.tokenize(source);

Calling the tokenizer directly is not something you'd normally do in real Markdoc usage, Markdoc.parse() already does this internally, it's shown here purely so the flat token layer is visible at least once.

const ast = Markdoc.parse(source);

This is the call every other lesson in this course actually uses. It tokenizes internally and returns the nested Ast.

function describeNode(node, depth = 0) {
  console.log(`${indent}${node.type}...`);
  for (const child of node.children) {
    describeNode(child, depth + 1);
  }
}

A plain recursive walk. node.attributes carries per-node data: level on a heading node, content on a text node, marker on a strong node (which literal characters, ** vs __, made it bold).