Tokenization & Lexical Analysis

Loading (deserializing) structured input data into computer memory as an implicit chain of tokens in order to prepare subsequent processing, syntactical/semantical analysis, conversion, parsing, translation or execution.

http://www.nicovideo.jp/watch/sm39061426