Tokenizer/lexer. Examples:

>>> tokenize('1 2 3')
[1, 2, 3, '\n']
>>> tokenize('fd 100')
['fd', 100, '\n']
>>> tokenize('pr "hello\nfd 100\n')
['pr', '"', 'hello', '\n', 'fd', 100, '\n']
>>> tokenize('while [:a>2] [make :a :a+1]')
['while', '[', ':', 'a', '>', 2, ']', '[', 'make', ':', 'a', ':', 'a', '+', 1, ']', '\n']
>>> tokenize('>>>= <= <><> == =>=<')
['>', '>', '>=', '<=', '<>', '<>', '=', '=', '=>', '=<', '\n']
>>> tokenize('apple? !apple .apple apple._me apple10 10apple')
['apple?', '!apple', '.apple', 'apple._me', 'apple10', 10, 'apple', '\n']

Note that every file fed in is expected to end with a 'n' (even if the file doesn't actually). We get common.EOF from the tokenizer when it is done.