lang.Tokenizer

Creates tokenizer objects which incrementally parse JavaScript code and returns the language tokens.

constructor

instance methods

• finished()
  Returns wether or not the code was completely parsed.
  return value:
  true if code was parsed, false if there is code left to parse.
  


• next()
  Returns the next token.
  return value:
  An Array containing token infomation [type, code, exception];
  exception is only available for the ERR token.
  The type is a constant from the lang.tokens collection.