[svn:perl6-synopsis] r14542 - doc/trunk/design/syn

Author: larry
Date: Sat May 17 14:37:37 2008
New Revision: 14542

Modified:
   doc/trunk/design/syn/S05.pod

Log:
Clarifications to how tied longest tokens are handled under LTM


Modified: doc/trunk/design/syn/S05.pod
==============================================================================
--- doc/trunk/design/syn/S05.pod	(original)
+++ doc/trunk/design/syn/S05.pod	Sat May 17 14:37:37 2008
@@ -14,9 +14,9 @@
    Maintainer: Patrick Michaud <pmichaud@pobox.com> and
                Larry Wall <larry@wall.org>
    Date: 24 Jun 2002
-   Last Modified: 7 May 2008
+   Last Modified: 18 May 2008
    Number: 5
-   Version: 78
+   Version: 79
 
 This document summarizes Apocalypse 5, which is about the new regex
 syntax.  We now try to call them I<regex> rather than "regular
@@ -2094,8 +2094,14 @@
 expressions).  A logical alternation using C<|> then takes two or
 more of these lists and dispatches to the alternative that matches
 the longest token prefix.  This may or may not be the alternative
-that comes first lexically.  (However, in the case of a tie between
-alternatives, the textually earlier alternative does take precedence.)
+that comes first lexically.
+
+However, if two alternatives match at the same length, the tie is
+broken by one of two methods.  If the alternatives are in different
+grammars, standard MRO (method resolution order) determines which
+one to try first.  If the alternatives are in the same grammar, the
+textually earlier alternative takes precedence.  (If a grammar's rules
+are defined in more than one file, the results are undefined.)
 
 This longest token prefix corresponds roughly to the notion of "token"
 in other parsing systems that use a lexer, but in the case of Perl
@@ -2150,6 +2156,11 @@
 Greedy quantifiers and character classes do not terminate a token pattern.
 Zero-width assertions such as word boundaries are also okay.
 
+Because such assertions can be part of the token, the lexer engine must
+be able to recover from the failure of such an assertion and backtrack
+to the next best token candidate, which might be the same length or shorter,
+but can never be longer than the current candidate.
+
 For a pattern that starts with a positive lookahead assertion,
 the assertion is assumed to be more specific than the subsequent
 pattern, so the lookahead's pattern is treated as the longest token;

• [svn:perl6-synopsis] r14542 - doc/trunk/design/syn by larry