pub fn tokenize(source: &str) -> Vec<(SyntaxKind, &str)>Expand description
Tokenise source into a lossless sequence of (SyntaxKind, text) pairs.
Every byte of the input is represented exactly once, so
tokens.iter().map(|(_, t)| *t).collect::<String>() == source always holds.