This struct was just a container for two pointers and just made things
painful. This structure has been removed and the next pointer has been
moved to the Token struct.
Also, add a null check to the calling context after NewLexer().
Whitespace tokens now contain all equal contiguous whitespace
characters. The token list will no longer contain, eg, three
TT_WHITESPACE tokens for three spaces and will instead have one
TT_WHITESPACE token that has a length of three.
- Parse code blocks that are wrapped sets of three backticks.
- Added a TT_TRIPLEBACKTICK token.
- Added a length field to the Token struct.
- Added an error node.
- Cleaned up some file read code.
- Fixed the header raw text (don't reuse the same buffer for each node).
Fixed parsing the tokens so it actually worked. Only headers are
currently implemented, but header nodes are now properly parsed and
everything else is properly ignored.
- Get the input filename from the command line
- Added in a bunch of checks to avoid segfaults
- Added some more debug info in places
Zorchenhimer