This class takes a set of highlighting rules, and creates a tokenizer out of them. For more information, see the wiki on extending highlighters.

Constructs a new tokenizer based on the given rules and flags.

rules	Object	Required. The highlighting rules
flag	String	Required. Any additional regular expression flags to pass (like "i" for case insensitive)

Returns an object containing two properties: tokens, which contains all the tokens; and state, the current state.

Tokenizer [edit]