Currently, unexpected tokens in the parser are shown as the text
found, plus the internal token name, including the notorious
"unexpected '::' (T_PAAMAYIM_NEKUDOTAYIM)".
This commit replaces that with a more user-friendly format, with
two main types of token:
* Tokens which always represent the same text are shown like
'unexpected token "::"' and 'expected "::"'
* Tokens which have variable text are given a user-friendly
name, and show like 'unexpected identifier "foo"', and
'expected identifer'.
A few tokens have special cases:
* unexpected token """ -> unexpected double-quote mark
* unexpected quoted string "'foo'" -> unexpected single-quoted
string "foo"
* unexpected quoted string ""foo"" -> unexpected double-quoted
string "foo"
* unexpected illegal character "_" -> unexpected character 0xNN
(where _ is almost certainly a control character, and NN is the
hexadecimal value of the byte)
The \ token has a special case in the implementation just to stop
bison making a mess of escaping it and it coming out as \\
Aside from a few very specific syntax errors for which detailed exceptions are
thrown, generally PHP just emits the default error messages generated by bison on syntax
error. These messages are very uninformative; they just say "Unexpected ... at line ...".
This is most problematic with constructs which can span an arbitrary number of lines, such
as blocks of code delimited by { }, 'if' conditions delimited by ( ), and so on. If a closing
delimiter is missed, the block will run for the entire remainder of the source file (which
could be thousands of lines), and then at the end, a parse error will be thrown with the
dreaded words: "Unexpected end of file".
Therefore, track the positions of opening and closing delimiters and ensure that they match
up correctly. If any mismatch or missing delimiter is detected, immediately throw a parse
error which points the user to the offending line. This is best done in the *lexer* and not
in the parser.
Thanks to Nikita Popov and George Peter Banyard for suggesting improvements.
Fixes bug #79368.
Closes GH-5364.
Primarily to avoid getting fatal errors from token_get_all().
Implemented using a magic E_ERROR token, which the lexer emits to
force a parser failure.
