perl.perl6.language.regex

perl.perl6.language.regex http://www.nntp.perl.org/group/perl.perl6.language.regex/ ... Copyright 1998-2008 perl.org Fri, 09 May 2008 12:01:43 +0000 ask@perl.org Help required urgently !!!!!! by Manoj.Menon "use matchpairs" not working. Currently having perl 5.0 in my system. Please suggest. Manoj http://www.nntp.perl.org/group/perl.perl6.language.regex/2005/04/msg594.html Thu, 21 Apr 2005 09:54:29 +0000 Re: Exposing regexp engine & compiled regexp's by Branden Quoted from http://www.perl.com/pub/2000/09/ilya.html, an interview with Dr. Ilya Zakharevich: > > Q: Could you describe in more detail what additional text- > handling primitives you would like to see included with Perl? > What string munging operations are absent that really ought to > be included in Perl's core? > > A: The problem: Perl's text-handling abilities do not scale > well. This has two faces, both invisible as far as you confine > yourselves to simple tasks only. The first face is not that > Perl lacks some "operations;" it is not that some "words" are > missing, whole "word classes" are not present. Imagine > expressive power of a language without adjectives. > > In Perl text-handling equals string-handling. But there is > more in a text than the sequence of characters. You see a text > of a program - you can see boundaries of blocks, etc.; you see > an English text, you can see word boundaries and sentence > boundaries, etc. With the exception of the word boundaries, > all these "distinctive features" become very hard to recognize > by a "local inspection of a sequence of characters near an > offset" - unless you agree to use a heuristic which works only > time to time. But a lot of problems require recognition of the > relative position of a substring w.r.t. these "distinctive > features". > > Remember those "abstract algorithms" books and lessons? You > can solve the problems "straightforwardly," or you can do it " > smartly." Typically, "straightforward" algorithms are easy to > code, but they do not scale well. Smart algorithms start by an > appropriate preprocessing step. You organize your data first. > The particular ways to do this may be quite different: you > sort the data, or keep an "index" of some kind "into your data," > you hash things appropriately, your balance some trees, and > so on. The algorithms use the initial data together with such > an "index." > > Perl provides a few primitives to work with strings, which are > quite enough to code any "straightforward" algorithm. What > about "smart" ones? You need preprocessing. Typically, digging > out the info is easy with Perl, but how would you store what > you dug? The information should be kept "off band," for example, > in an array or hash of offsets into the string. > > Now modify the string a little bit, say, perform some s()() > substitutions, or cut-and-paste with substr(). What happens > with your "off band" information? It went out of sync. You > need to update your annotating structures. Do not even think > about doing s()()g, since you do not have enough info about > the changes after the fact. You need to do your s()() one-by- > one - but while s()()g is quite optimized, a series of s()() > is not - and you get stuck again into the land of badly > scaling algorithms. > > (Strictly speaking, for this particular example s()()eg could > save you - as well as code-embedded-into-a-regular-expression, > but this was only a simple illustration of why off-band data > is not appropriate for many algorithms. Please be lenient with > this example!) > > Even if no modification is done, using off-band data is very > awkward: how to check what are the attributes of the character > at offset 2001 when there are many different attributes, each > marking a large subset of the string? > > That was the problem, and the solution supported by many text- > processing systems is to have "in-band annotations", which is > recognized by the editing primitives, and easily queryable. > Perl allows exactly one item of in-band data for strings: pos > (), which is respected by regular expressions. But it is not > preserved by string-editing operations, or even by $s1 = $s2! > > "In-band" data comes in several "kinds". A particular "kind" > describes: > > - how it behaves with respect to insertion or deletion of > characters nearby; > - can the "same" markup appear "several times"; > - can the markup "nest" (like nested comments in some languages > ); and > - is there an internal structure of the markup (as in a loop, > which may be > > [[LABEL DELIM0] KEYWORD [DELIM1 VAR1 SEP VAR2 ... DELIM2] > [DELIM4 EXPR DELIM4] [DELIM5 BODY DELIM6]] > > - with some parts possibly missing, so the internal structure > is a tree). > > Different answers lead to a zoo of intuitively different kinds > of markup, each kind useful for some categories of problems. > You can mark "gaps between" characters, or you can mark > characters themselves. The markup may "name" a position ("the > first __END__ in a Perl program"), or cover a subset of the > string ("show in red", "is a link to this URL", or "inside > comment"). Since the kind of the markup defines what happens > when the string is modified, the system can support self- > consistency of the markup "automatically" (in exceptionally > complicated cases one may need to register a callback or two). > > The second face of problem is not with the expressive power of > Perl, but with the implementation. Perl has a very rigid rule: > a string must be stored in a consecutive sequence of bytes. > Remove a character in the middle of the string, and all the > chars after it (or before it) should be moved. As I said, s()()g > has some optimizations which allow doing such movements "in > one pass", but what if your problem cannot be reduced to one > pass of s()()g? Then each of the tiny modification you do one- > at-a-time may require a huge relocation - or maybe even > copying of the whole string. This is why a lot of algorithms > for text manipulation require a "split buffer" implementation, > when several chunks of the string may be stored (transparently!) > at unrelated addresses. > > Such "split-buffer" strings may look incredibly hard to > implement, as in "all the innards of Perl should be changed", > but it is not. Just store "split strings" similarly to tie()d > data. The FETCH (actually, the low-level MAGIC-read method) > would "glue" all the chunks into one - and would remove the > MAGIC - before the actual read is performed; and now no part > of Perl requires any change. Now four or five primitives for > text-handling may be changed to recognize the associated tie()d > structures - and act without gluing chunks together. We may > even do it in arbitrarily small steps, one opcode at a time. > > Another important performance improvement needed for many > algorithms would be the copy-on-write, when several variables > may refer to the same buffer in memory, or different parts of > the same buffer - with suitable semantic what to do when one > of these variables is modified. (In fact the core of this is > already implemented in one of my patches!) Together with other > benefits, this would solve the performance problems of $& and > friends, as well as would make m/foo/; $& = 'bar'; equivalent > to s/foo/bar/. Having copy-on-write substrings may be slightly > more patch-intensive than copy-on-write strings, though. The > complication: currently the buffers are required to be 0- > terminated (so that they may be used with the system APIs). It > is hard to make 'b' as in substr('abc',1,1) refer to the same > buffer (containing "abc\0") as 'abc'. The solution may be to > remove this requirement, and have two low-level string access > API, SvPV() and SvPVz(), so that SvPVz() may perform the > actual copying (as in copy-on-write) and the appending of \0 - > but only when needed! > > Without these - or similar - changes Perl would not scale well > as a language for efficient text-processing. What is more, I > believe that the changes above can remove most of the > significant bottlenecks for the problems we have in text- > processing of today. At least I know a lot of problems which > would have feasible solutions given these changes. > > And I need not repeat that a handful of small extensions to > the expressive power of the regular expression engine could > radically extend the domain of its applicability. ;-) > That's exactly the kind of thing we can do by exposing regexp engine's guts. Tags in the string we can implement by using lists, just as lisp would do it. If we have SVs that actually independ from the implementation, we can create a ``tagged-string'', that's seen as a string by the script but internally implemented as a list. And, if we have access to the regexp engine's guts, we can implement matches and substitutions against those magic tagged strings. The other thing about copy-on-write would be piece of cake, using the same magic SVs for implementing strings that are substrings of other strings. Branden. http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg593.html Wed, 10 Jan 2001 12:53:28 +0000 Re: Exposing regexp engine & compiled regexp's by Damian Conway > As Rick pointed out, there's no problem with overloading =~ for an > object, in the same way it's done with `eq', and one object's > function could return either an object or a closure (a sub > reference), so that a module could even hide the details of whether > it's using the object interface with the overloaded =~ or the new > behaviour of =~ with a sub lvalue. Good point. I think at this stage we're violently agreeing with each other. ;-) Damian http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg592.html Tue, 09 Jan 2001 19:03:26 +0000 Re: Exposing regexp engine & compiled regexp's by Jarkko Hietaniemi On Tue, Jan 09, 2001 at 12:41:30AM -0500, James Mastros wrote: > On Mon, Jan 08, 2001 at 05:02:17PM -0600, Jarkko Hietaniemi wrote: > > Wouldn't an incremental on-demand engine be much > > more flexible and optimizable (e.g. finding 'the fast path' smells > > like input-driven LRU to me)? > > Umm, I'm not certian that I'm completly following here. It seems that in > the vast majority of all cases, you'd need to compile (or at the very least, > parse) the entire regex. Also, you can get /vast/ efficency gains by > compiling a regex, so you can check the easy things first. What I have in mind is not *that* different from the current way of things, the regex will get compiled, but not before it's first used, and not necessarily all at once. No, I don't have any formal theory, any actual code, and much less benchmarks to prove this, I'm just waving my hands to keep warm. -- $jhi++; # http://www.iki.fi/jhi/ # There is this special biologist word we use for 'stable'. # It is 'dead'. -- Jack Cohen http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg591.html Tue, 09 Jan 2001 06:21:53 +0000 Re: Exposing regexp engine & compiled regexp's by Filipe Brandenburger Damian Conway wrote: >I'm well-known as a non-delving-into-the-guts type of guy. I don't have I totally aggree with you that delving into the guts is the last thing we, the people that use perl as a tool, want to do! The fact is that, the least we know about the internals, the better it is. But for this to be possible, we need modules that provide the functionality we need without making us deal with the rawness of perl guts. And that's why I defend the exposing of the regexp subsystem interface (as all other subsystems), so that one can write (and we can use!!!) modules that can match regexps against file streams, we can have partial matches, approximate, etc... Having the guts interface opens up the door for everything that is doable with a regexp engine. As to the efficiency problem, I said it before and I'm saying it again: my opinion is to put both approaches (overloaded =~ in object module and having =~ work for a sub). One of them is very flexible, and possibly can do things the other can not, and the other is very efficient for the general case, the one that will be used 90% of the time. As Rick pointed out, there's no problem with overloading =~ for an object, in the same way it's done with `eq', and one object's function could return either an object or a closure (a sub reference), so that a module could even hide the details of whether it's using the object interface with the overloaded =~ or the new behaviour of =~ with a sub lvalue. Branden. _________________________________________________________ Oi! Você quer um iG-mail gratuito? Então clique aqui: http://www.ig.com.br/paginas/assineigmail.html http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg590.html Tue, 09 Jan 2001 06:11:19 +0000 Re: Exposing regexp engine & compiled regexp's by James Mastros On Mon, Jan 08, 2001 at 05:02:17PM -0600, Jarkko Hietaniemi wrote: > Wouldn't an incremental on-demand engine be much > more flexible and optimizable (e.g. finding 'the fast path' smells > like input-driven LRU to me)? Umm, I'm not certian that I'm completly following here. It seems that in the vast majority of all cases, you'd need to compile (or at the very least, parse) the entire regex. Also, you can get /vast/ efficency gains by compiling a regex, so you can check the easy things first. -=- James Mastros -- midendian: She never sleeps. mousetrout: But I do. I just regret it after I wake up. AIM: theorbtwo homepage: http://www.rtweb.net/theorb/ ICBM: 40:04:15.100 N, 76:18:53.165 W http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg589.html Mon, 08 Jan 2001 21:42:08 +0000 Re: Exposing regexp engine & compiled regexp's by Damian Conway > The only thing I remark is that I believe all of Perl should be the > most exposed possible, so that unseen levels of introspection > can be achieved. In that philosophy I wrote my idea about > exposing the engine's guts. I'm well-known as a non-delving-into-the-guts type of guy. I don't have a problem with your proposal, since (as Rick points out) it can easily be viewed as a superset of mine, and made no harder to implement (assuming we eventually see method calls draw close to subroutine calls in execution speed). > BTW, I didn't see any comments about my second thought, the one > of inspecting compiled regexps. Did you like it? That also goes in > the direction of exposing all the internals to the module writers... I didn't feel qualified or entitled to comment on it since I *never* operate at the level of the perl guts; not even in modules -- like Switch -- that may appear to do so. Damian http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg588.html Mon, 08 Jan 2001 19:24:46 +0000 Re: Exposing regexp engine & compiled regexp's by branden Damian Conway wrote: > # As Branden proposes: > > package From_STDIN; > > sub new { bless $_[1], $_[0] } > > sub MORE_DATA { $_[0]->getn($_[1]) } > sub ON_FAIL { $_[0]->pushback($_[1]) } > > use overload "=~" => 1; > > package main; > > From_STDIN->new($fh) =~ /pat/; > > >Hmmmm. Potentially more flexible, but also much more ponderous. Sorry I didn't include code the first time, but actually my idea is about much more flexibility than having MORE_DATA and ON_FAIL methods in an object with overloaded ``=~'' Actually, I think the whole interface of the regex engine should be exposed to Perl, so that someone could write an OO package with ``virtual methods'' MORE_DATA and ON_FAIL and manage the guts of the engine so that it behaves like expected. Something like: package RegexBase; use overload '=~' => \&match; sub match { # here is the brains of the class # something involving: # - the guts of the interface of the Regex Engine # - the MORE_DATA method when data is needed # - the ON_FAIL method when a match is failed } package From_STDIN; @ISA = qw(RegexBase); sub MORE_DATA { ... } sub ON_FAIL { ... } What I'm trying to say is that this is the most flexible we can do. If one wants to check success or failure, he can, if he wants to see if there was a state of Failed/Short/Exact/Long/LongFailed (words from your message), he can too. If he wants to inspect in which state the NFA stopped, he also can. Virtually anything that involves Regexp's can be built from there up. I think your `sub' idea, althought a bit confuse and having more than one significate to the same sub, is the most common case, and I think it should be implemented yes. The only thing I remark is that I believe all of Perl should be the most exposed possible, so that unseen levels of introspection can be achieved. In that philosophy I wrote my idea about exposing the engine's guts. I know it's heavy to do things like I say. 1st: that's the price of the flexibility it gives (althought the two approaches can be safely implemented, they are complementary, not conflictant) and 2nd: those are thought to be used by module writers, for problems that can't be solved now in a good way, and where all the complexity it introduces would still be a big win, compared with the way it could be handled in perl5 (namely, reading the whole file in memory or breaking a regexp that would match more than one line, without mentioning the spaghetti flow control turns in!!!) BTW, I didn't see any comments about my second thought, the one of inspecting compiled regexps. Did you like it? That also goes in the direction of exposing all the internals to the module writers... Branden. ---------------------------------------------------------------- Get your free email from AltaVista at http://altavista.iname.com http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg587.html Mon, 08 Jan 2001 19:16:32 +0000 Re: Exposing regexp engine & compiled regexp's by Rick Delaney Damian Conway wrote: > > Branden wrote: > > > Then, what you proposed in RFC 93 through > > > > sub { ... } =~ m/.../; > > > > could be handled by > > > > my $mymatch = MyClassForMatchingFromFileHandles->new($myhandle); > > $mymatch =~ m/.../; > > This is an interesting alternative. The main problem is that matching > against a blessed object already has a useful meaning in Perl: stringify > the object (calling its overloaded stringification operator if possible) > and match against the resulting string. That may not be a problem. In the absence of overloading of =~, the object can be stringified (like now) but if =~ is overloaded then that could take precedence. Or, just make =~ behave the same as eq and use fallback to get the current stringifying behaviour. The biggest problem is making =~ overloadable in the first place. This has a lot of potential but needn't replace your RFC: use overload "=~" => sub { my ($obj, $pat) = @_; my $coderef = $obj->{attr}; $coderef =~ /$pat/; }; > My other problem with this approach is that it's relatively heavy. Let's > take the example in the RFC and implement it both ways: > > # As the RFC proposes: > > sub from_STDIN { > $_[1] ? $fh->pushback($_[0]) : $fh->getn($_[0]) > } > > \&from_STDIN =~ /pat/; > > # As Branden proposes: > > package From_STDIN; > > sub new { bless $_[1], $_[0] } > > sub MORE_DATA { $_[0]->getn($_[1]) } > sub ON_FAIL { $_[0]->pushback($_[1]) } > > use overload "=~" => 1; > > package main; > > From_STDIN->new($fh) =~ /pat/; > > Hmmmm. Potentially more flexible, but also much more ponderous. I wouldn't say *much* more. There is slightly more to do for a module author (in object setup) but for the module user it is the difference between: my $scalar = From_STDIN->new($fh); $scalar =~ /pat/; and my $scalar = From_STDIN_closure($fh); $scalar =~ /pat/; The overload technique can be made considerably less ponderous by creating one class that behaves as your proposal. Then those that like it could just use RFC93 sub { $_[1] ? $fh->pushback($_[0]) : $fh->getn($_[0]) }; my $scalar = RFC93->new; $scalar =~ /pat/; -- Rick Delaney rick.delaney@home.com http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg586.html Mon, 08 Jan 2001 19:11:05 +0000 Re: Exposing regexp engine & compiled regexp's by Damian Conway Branden wrote: > I read your RFC 93. It mentions using a sub to read from the > string. I just think it uses the sub in two conflicting ways, one > for requesting more data from the stream and other for telling > there was a match. It's really using the sub as a interface to whatever source of data it's trying to match. > I thought, too, that requesting it to return > _exactly_ the number of characters that was requested goes against > most unix syscalls convention (like read...), where it's requested > to read at most that number of characters. Err. The RFC never says the request is for exactly a certain number of characters; just that the subroutine will be told how many characters are *known* to be needed in order for the regex to continue matching. The RFC specifically mentions the possibility of returning fewer than requested characters. > What I think is that it could be handled by a OO module. Suppose > there's how to hook into the regexp engine guts, getting responses > as the ones you mentioned above. One could write a OO module, with > methods for reading more data, checking end of data, and > acknowledging a failed or succeeded match. Then, it could overload > the =~ operator, making the regexp engine call the module's methods > instead of its own's. > > Then, what you proposed in RFC 93 through > > sub { ... } =~ m/.../; > > could be handled by > > my $mymatch = MyClassForMatchingFromFileHandles->new($myhandle); > $mymatch =~ m/.../; This is an interesting alternative. The main problem is that matching against a blessed object already has a useful meaning in Perl: stringify the object (calling its overloaded stringification operator if possible) and match against the resulting string. My other problem with this approach is that it's relatively heavy. Let's take the example in the RFC and implement it both ways: # As the RFC proposes: sub from_STDIN { $_[1] ? $fh->pushback($_[0]) : $fh->getn($_[0]) } \&from_STDIN =~ /pat/; # As Branden proposes: package From_STDIN; sub new { bless $_[1], $_[0] } sub MORE_DATA { $_[0]->getn($_[1]) } sub ON_FAIL { $_[0]->pushback($_[1]) } use overload "=~" => 1; package main; From_STDIN->new($fh) =~ /pat/; Hmmmm. Potentially more flexible, but also much more ponderous. > BTW, if you have a C++-based regexp engine with a clean design, > couldn't we use it as a base to a new regexp engine that supports > current (or new) perl's regexp syntax and features and has its guts > exposed? It was very basic, and *very* slow. It was also DFA-based and hence unable to implement full Perl regex semantics. Furthermore, the regex engine is (and should be) one of the most heavily optimized parts of Perl: probably not the place for clean, modular design. :-) However, for what it's worth, I have no objection to making the code available for everyone's amusement. Bear in mind that this was written by a much early version of me (about 0.27), way back in the last millenium, before C++ was standardized and before there was an STL. Surprisingly, it still compiles and runs under g++ 2.8.1. Grab it from: http://www.csse.monash.edu.au/~damian/Perl6/Regex/Regex.tar.gz Damian http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg585.html Mon, 08 Jan 2001 16:01:30 +0000 Re: Exposing regexp engine & compiled regexp's by Jarkko Hietaniemi On Tue, Jan 09, 2001 at 09:53:10AM +1100, Damian Conway wrote: > > > I once brutalized Henry Spencer's engine into telling me when I was > > on my way to a match. This was for a UI: I wanted to be able to say > > that the input should only match this RE, and if they typed something > > that broke the match, I could beep and disallow the character. > > > > I was just a greenhorn then, so it coredumped. But it's another use > > for access to internal engine states and failure reasons. > > Good point. Perhaps, on failure, a regex could set $@ to indicate what the > problem was: > > while (gui_getchar($nextchar)) { > $input.$nextchar =~ m/$pattern/; > if ($@ =~ /fail/) { beep } > else { $input .= $nextchar } > } Another related idea that has been circling in my brain lately is that do we really need to have separate compilation and execution steps in our regex engine? Wouldn't an incremental on-demand engine be much more flexible and optimizable (e.g. finding 'the fast path' smells like input-driven LRU to me)? (Also much worse hell to implement, but let's worry about that later :-) -- $jhi++; # http://www.iki.fi/jhi/ # There is this special biologist word we use for 'stable'. # It is 'dead'. -- Jack Cohen http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg584.html Mon, 08 Jan 2001 15:02:40 +0000 Re: Exposing regexp engine & compiled regexp's by Damian Conway > I once brutalized Henry Spencer's engine into telling me when I was > on my way to a match. This was for a UI: I wanted to be able to say > that the input should only match this RE, and if they typed something > that broke the match, I could beep and disallow the character. > > I was just a greenhorn then, so it coredumped. But it's another use > for access to internal engine states and failure reasons. Good point. Perhaps, on failure, a regex could set $@ to indicate what the problem was: while (gui_getchar($nextchar)) { $input.$nextchar =~ m/$pattern/; if ($@ =~ /fail/) { beep } else { $input .= $nextchar } } Damian http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg583.html Mon, 08 Jan 2001 14:53:15 +0000 Re: Exposing regexp engine & compiled regexp's by Filipe Brandenburger Damian wrote: > > I once wrote a C++-based regex engine (much simpler than Perl's!) > just like this. > > Knowing why a regex failed *is* invaluable when matching regexes > against file streams, but there are more possibilities than you > mentioned: > > "Failed" Did not match because of illegal transition > > "Short" Did not match: did not reach acceptor state > > "Exact" Matched and finished in an acceptor state > > "Long" Passed through an acceptor state, continued to > match, but did not finish in an acceptor state > > "LongFailed" Passed through an acceptor state, continued to > match, but then found an illegal transition > > Ultimately, I decided that what was needed wasn't insight into the cause > of failure, but rather the chance to provide more data to "feed" the > engine so it doesn't have to fail "Short" or "Long". That's why I > proposed RFC 93 (http://dev.perl.org/rfc/93.html) instead of a mechanism > such as you have suggested. > > Damian > Good points, Damian. I read your RFC 93. It mentions using a sub to read from the string. I just think it uses the sub in two conflicting ways, one for requesting more data from the stream and other for telling there was a match. I thought, too, that requesting it to return _exactly_ the number of characters that was requested goes against most unix syscalls convention (like read...), where it's requested to read at most that number of characters. What I think is that it could be handled by a OO module. Suppose there's how to hook into the regexp engine guts, getting responses as the ones you mentioned above. One could write a OO module, with methods for reading more data, checking end of data, and acknowledging a failed or succeeded match. Then, it could overload the =~ operator, making the regexp engine call the module's methods instead of its own's. Then, what you proposed in RFC 93 through sub { ... } =~ m/.../; could be handled by my $mymatch = MyClassForMatchingFromFileHandles->new($myhandle); $mymatch =~ m/.../; What I mean is, by exposing the guts of the regexp engine, we could implement all that's wanted in RFC 93, with a cleaner interface, and even do more, because we can hook up every call to the regexp engine! BTW, if you have a C++-based regexp engine with a clean design, couldn't we use it as a base to a new regexp engine that supports current (or new) perl's regexp syntax and features and has its guts exposed? Branden. _________________________________________________________ Oi! Você quer um iG-mail gratuito? Então clique aqui: http://www.ig.com.br/paginas/assineigmail.html http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg582.html Mon, 08 Jan 2001 04:17:15 +0000 Re: Exposing regexp engine & compiled regexp's by Nathan Torkington Damian Conway writes: > I once wrote a C++-based regex engine (much simpler than Perl's!) > just like this. I once brutalized Henry Spencer's engine into telling me when I was on my way to a match. This was for a UI: I wanted to be able to say that the input should only match this RE, and if they typed something that broke the match, I could beep and disallow the character. I was just a greenhorn then, so it coredumped. But it's another use for access to internal engine states and failure reasons. Nat http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg581.html Sat, 06 Jan 2001 13:19:09 +0000 Re: Exposing regexp engine & compiled regexp's by Damian Conway > 1. I think it should be possible to have ``incomplete matches''. > > Regexp's are interpreted by a NFA, that is a state machine. > I think it would be nice if, when I try to match a regexp against > a string, and the string ends before the regexp matches, it > would be possible to find out in which state the NFA has stopped, > and it would also be possible to start another match from > that state on. I once wrote a C++-based regex engine (much simpler than Perl's!) just like this. Knowing why a regex failed *is* invaluable when matching regexes against file streams, but there are more possibilities than you mentioned: "Failed" Did not match because of illegal transition "Short" Did not match: did not reach acceptor state "Exact" Matched and finished in an acceptor state "Long" Passed through an acceptor state, continued to match, but did not finish in an acceptor state "LongFailed" Passed through an acceptor state, continued to match, but then found an illegal transition Ultimately, I decided that what was needed wasn't insight into the cause of failure, but rather the chance to provide more data to "feed" the engine so it doesn't have to fail "Short" or "Long". That's why I proposed RFC 93 (http://dev.perl.org/rfc/93.html) instead of a mechanism such as you have suggested. Damian http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg580.html Sat, 06 Jan 2001 13:08:53 +0000 Exposing regexp engine & compiled regexp's by Filipe Brandenburger Hello. I have some ideas (actually a wishlist) for the regular expression subsystem (that's what it'll be, right?). I would appreciate if perl6's regexp engine exposes the maximum possible interfaces, and I'll expose why I think this would be nice. 1. I think it should be possible to have ``incomplete matches''. Regexp's are interpreted by a NFA, that is a state machine. I think it would be nice if, when I try to match a regexp against a string, and the string ends before the regexp matches, it would be possible to find out in which state the NFA has stopped, and it would also be possible to start another match from that state on. Suppose I'm reading from a file on a per-line basis. Suppose that I want to strip off XML-style comments from the file. What I need is to match the // regular expression and substitute it for, say, the empty string. The problem is if the file contains multi-line comments, where the ``'' is in another posterior line, this method will fail. If, when I try the match, I see that I have an incomplete match, I could take the state of the regexp engine, and start matching with it in the next line, and so on, until the engine ends the match. Other possibility is to tokenize a file `a la lex'. For example, if I want to parse a C source code file, I break it in identifiers, constants, reserved-words, comments, etc. If I have the whole file in a string, I can break it using the /\G.../gc regular expressions, but reading a whole file in memory can be a problem sometimes. If we have incomplete matches, we can read a block of data from the file, try to match it against the regexp's and, if necessary, read more blocks of data from the file until a match is done. Now I had another idea with the exposing of states of the NFA. It would be possible to join several /\G.../gc regular expressions into one, using |, and, after the match, use the state the NFA stopped to determine which regexp was matched. Other possibility is to implement the equivalent of awk's RS. awk has a RS variable similar to perl5's $/, but awk's RS can handle regular expressions instead of $/'s strings. With NFA's states exposed it would be possible to implement the same behaviour of awk's RS. The possibilities are many. I think it wouldn't be difficult to expose NFA's states as I imagine it. Afterall, probably the C code inside of perl5 has them exposed. I have yet worked with XS and the fact that the regular expression engine isn't exposed in the XS interface let's me a little disappointed. There were times when I wanted to do things I say here and I realised that even with XS I would have to dig for perl's regexp engine or even use an external regexp engine to do the job. 2. I think compiled regexp's should be analysable by perl code. Regexp's in their compiled states are (a little simplification here) a tree. I think it would be nice if there were ways of traversing this tree, to find properties of the regular expressions. This would help several things. For example, consider a module that executes searches in a database table, matching people's names against regular expressions. Suppose now that the database column where the names are stored is indexed by prefix and suffix. If it were possible to traverse regular expressions, the module would be able to optimize searches like /^John/ and /Smith$/, using the indexes to reduce the number of records that the regexp should match. Of course it would have to search through all the records for a middle name, like /F\./, but the possibility to optimize some cases by inspecting the compiled regexp is a big win, at least for me. Traversing a compiled regexp also leads to implementing custom regexp engines, for example, a engine that calculates the difference or distance between the string and what would match the regexp. As with the first case, the possibilities are many. I actually don't know perl5's regexp engine and compiler, but I believe it doesn't escape from my model (NFA with states & a tree). I agree with the fact that code talks louder than speech, but I would like to know if there's something going in this regexp area and if there really is interest in this. Anyway, if there's something going here, I would like very much to join the team, if I can contribute a bit to it. If there's nothing going on and the list considers my idea good, I will try to `kind of' implement it as a perl5 extension (oh, boy! that'll be hard!!!) If I do it, I hope I get help from you in doing it! Thank's. Branden http://www.nntp.perl.org/group/perl.perl6.language.regex/2001/01/msg579.html Fri, 05 Jan 2001 14:15:41 +0000 An Apology by Deven T. Corzine Everyone: I'm sorry about the recent commotion over minor aspects of regex design and implementation. I stumbled into being the most active participant in an argument nobody wanted, myself included. I realized too late (and with the help of a friend) that I had been arguing for a _declarative_ viewpoint, against a prevailing iterative viewpoint. It turns out to be much more difficult to convey the declarative viewpoint clearly and precisely. I still consider the declarative viewpoint to be perfectly valid, but I now realize that it's merely a different (and minority) perspective. I made the initial mistake of assuming everyone saw the same thing I did, and the further mistake of believing that the declarative viewpoint was preferable. I realize now that which is preferable depends on your frame of reference, and it's obvious now that the iterative viewpoint is the prevailing one. (That may be the result of indoctrination as much as natural inclination, but the fact remains.) What seems intuitive from _each_ viewpoint also seems counterintuitive from the other one, a good formula for dissension! While I never assumed that the behavior would necessarily be changed (since historical precedent and/or implementation issues might be compelling), now I believe that it should remain as it is, because it's harder to understand the way I suggested, and changing the behavior could prove disruptive to the programmer's confidence in his/her ability to predict the proper match. (Also, the C-style comment matching example convinced me that there _can_ be true utility in the current behavior, which I hadn't considered...) Many people were quick to jump to conclusions and misinterpret my questions about the design as ignorance. I guess I got a little defensive because of that, and responded to more messages than was wise. I was surprised by the reaction I received, since with Perl's motto (TMTOWTDI), I rather expected divergent viewpoints to be not only accepted, but welcomed! Maybe that was a naive expectation. Or maybe I just did a miserable job trying to explain the idea I was trying to convey. I wish I had thought of the declarative/iterative dichotomy earlier; my friend was the one who managed to label them for me. That brought the controversy clearly into focus for me, and made me realize that the perspectives were different, with neither inherently better than the other. That's when I realized that I had a personal preference for the declarative viewpoint that I didn't notice as such, and decided it was a mistake to argue this to begin with. I suppose I'm now tarred as the villain here, since I started the thread and (because most replies were directed at me) ended up the most active participant in this argument that got blown entirely out of proportion. I hope people can look beyond that, and not prejudge me from one incident. Only time will tell... Sorry for all the trouble. If anyone wants to discuss this issue further, I'll only do it in private at this point; I don't want to further annoy anyone on this list with the topic, even if anyone DOES want to continue! Deven P.S. For anyone curious about the "sleep debt" thing I mentioned before, I'll recommend reading "Sleep Thieves" by Stanley Coren. It's interesting and presents some good evidence to suggest that Americans are particularly sleep-deprived in general, that it's a dangerous and less-productive state, and that we probably ought to be getting 9-10 hours of sleep, not the 7-8 hours that society deems "normal"... (I too often get 6 hours or less!) P.P.S. With my luck, it will probably turn out to have been a mistake to send this message also. Since I can't win anyway, I might as well send it. http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg578.html Mon, 18 Dec 2000 08:59:30 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Bart Lateur On Fri, 15 Dec 2000 13:42:44 -0700, Kevin Walker wrote: >Deven seems to be advocating thinking about regular expressions >without worrying too much about the implementation, even at a fairly >abstract level. Here's a counter example: /aaaabbbbccccdddddbbbbcdddd/ Shouldn't a non-greedy matcher /b.*?d/, according to the OP's rules, match "bcd", the second matching string? That is shorter than the first match. Oh, you want the first match. Well: same thing. It is similar in nature to that question in comp.lang.perl.misc the other day: why doesn't greediness force /a|ab/ to preferably match "ab"? Because it doesn't. Try from left to right, first match wins. A dead simple rule. If Perl's greediness/nongreediness doesn't work for you, rely on something else, to get what you want. -- Bart. http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg577.html Sat, 16 Dec 2000 17:58:30 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by brian d foy On Fri, 15 Dec 2000, Simon Cozens wrote: > On Fri, Dec 15, 2000 at 11:39:08AM -0800, Randal L. Schwartz wrote: > > Tell me how you can do that without breaking much existing code. > > Pssst, Randal, this is Perl 6, not p5p. well, we do have to translate 95% of that code to Perl 6 without a hitch. i imagine that breaking regex behaviour in fovor of something new would make that quite difficult. perhaps that is not what Randal was talking about, but it is something to consider. -- brian d foy <brian@smithrenaud.com> Director of Technology, Smith Renaud, Inc. 875 Avenue of the Americas, 2510, New York, NY 10001 V: (212) 239-8985 http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg576.html Sat, 16 Dec 2000 11:23:05 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by brian d foy On Fri, 15 Dec 2000, Deven T. Corzine wrote: > If we want the first interesting match, and we're preferring early matches > and short matches, I believe that "bccccd" is more interesting. then write a regex that describes that pattern. the pattern is one b followed by some stuff that is not a d up to one d you complain because the "." regex special character does not do what you want. it matches any character except a newline. however, for the case you provide, you don't want any character except a newline. so, your use of "." is your problem, along with your refusal to realize that what you are dealing with is a set of rules, and if you grok the rules, you can do what you want. it has nothing to do with opinion or intuition. neither of those have any place in a completely (well, almost ;) described, human-designed system. this isn't nuclear physics after all. your trouble is not with greediness or shortest matches, as i said before. you just don't understand what you are doing and refuse to beleive otherwise. -- brian d foy <brian@smithrenaud.com> Director of Technology, Smith Renaud, Inc. 875 Avenue of the Americas, 2510, New York, NY 10001 V: (212) 239-8985 http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg575.html Sat, 16 Dec 2000 11:18:44 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by brian d foy On Fri, 15 Dec 2000, Deven T. Corzine wrote: > On 15 Dec 2000, Randal L. Schwartz wrote: > > > >>>>> "Deven" == Deven T Corzine <deven@ties.org> writes: > > > > Deven> As for special-case rules, I believe that my proposed modification would > > Deven> REMOVE a special-case semantic rule, at the cost of added complexity at the > > Deven> implementation level. (The cost decision of whether that added complexity > > Deven> is worthwhile is a separate consideration.) > > No, it would break a much higher overriding rule of "left most match > > wins". > > Can you give a concrete, real-life example of code that my proposed change > would actually break, not a contrived hypothetical case design to break? well, there is the trivial case that it breaks m/b.*d/; if i wrote that, i expect, since it is documented this way, to find the first b and everything up to and including the first d after that. but here's one from some code i wrote last week. i want to find all the groups of things like "212-555-1212". this particular regex adequately describes the un-normalized data that i had to munge. @matches = m/(\d.*?)\s+/g; the right way (the documented way), gives me the entire phone number. yours give me "2". you should be quiet now, or write your own language. -- brian d foy <brian@smithrenaud.com> Director of Technology, Smith Renaud, Inc. 875 Avenue of the Americas, 2510, New York, NY 10001 V: (212) 239-8985 http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg574.html Sat, 16 Dec 2000 11:04:32 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Tom Christiansen >Nice summary, but I'm not buying what you're selling in the elaboration. Then you lose, because I am not allowed to disagree with you anymore. And everyone else has already written you off. And the answer to "what breaks if mimimal matching is overall but maximal matching is local"--or even, "if we change it all"-- is a zillion programs, including just about any progressive match: while (/.*?(\w+)=(\S+)/g) { push @{ $h{$1} }, $2; } I can't wait for that to match the rightmost one and then fail. Bah. >>/dev/null --tom http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg573.html Fri, 15 Dec 2000 15:27:27 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Deven T. Corzine [I delayed responding to this message because it was the longest.] On Thu, 14 Dec 2000, Tom Christiansen wrote: > >No question that's how it's been implemented. But WHY would anyone want > >such behavior? When is it beneficial? > > It is beneficial because this is how it's always been, because it > is faster, because it is more expressive, because it is more powerful, > because it is more intuitive, and because it is more perlian. Nice summary, but I'm not buying what you're selling in the elaboration. > In elaboration: > > 0) All NFAs before POSIX acted this way. It is historically > consistent and perfectly expected. First, all those older regular expression systems were inherently greedy matching algorithms. There is absolutely no conflict between earliest match and greedy matching. The conflict only arises when non-greedy matching is introduced into the equation. Second, by referring to NFA's, you're already dropping down to a lower level, and as I said in a prior message, you've lost the gestalt by then. Once you drop to that level (and stop referencing the semantics of the whole), my point is moot, because it no longer makes sense at that level. Finally, historical precedent is a always a justification not to change anything. Of course, that argued against the wonderful regex extensions that Perl 5 added, since they went against the way "it's always been"... > 1) It is obviously faster to come to an answer earlier on in the > execution than it would be to come to an answer later. It's > like an expression whose evaluation short-circuits. Also, when > the matching sematics permit back tracking and back references, > the combinatoric possibilities can easily explode into virtual > unsolvability as the 2**N algorithm loses its race to the heat > death of the universe. Yes, if Perl did overall-longest or > overall-shorted, this would produce a more predictable time; > however, as we see with DFAs and POSIX NFAs, this prediction > plays out as guaranteed *WORST-CASE* time. It is not acceptable > to make everyone pay the worst-case time. Never penalize > the whole world for the needs or desires or the few. I don't think it should be implemented unless the cost is small or applied only to those situations where it's wanted despite the cost. However, I don't believe it would NECESSARILY be slower to execute -- yes, there's more complexity. The price for that complexity must be paid, but it may be payable in memory by making a larger (but equally fast) NFA for the regular expression. Or maybe it can't. I don't know, because I have not investigated it. It's premature to simply assume it MUST be slower. > 2) Consider the simple case, /A|B/. In your overall longest/shortest, > guaranteed worst-case time, both submatch A and submatch B must > be calculated, and then the lengths of their matches both be compared. > Perl, fortunately, does not do that. Rather, the first one in that > sequence wins. That means that under the current scheme, the > patterns /A|B/ and /B|A/ have different semantics. Under your > worst-case scheme, they do not. Because /A|B/ and /B|A/ mean > something different, more expressivity is provided. This is the > same scenario, albeit expressed slightly differently, as your > situation. The issues manifest in both are equivalent. Then that is another semantic anomaly, because the alternation is supposed to mean that one or the other pattern matches. Logically, "or" should be communitive. If they're not quite, that's another disconnect between the high-level model and the implementation. Maybe /A|B/ and /B|A/ _do_ mean different things to the engine, but they do NOT mean different things in the high-level semantics -- except when you force the high-level semantics to change by fiat, to match the implementation. > 3) This leads to increased power. It's like the difference between > a short-circuiting "or" and one that blindly plods ahead trying > to figure something out even when all is for naught. Compare A&&B > with A&B, for example. If A is 0, then B need not be computed, > yet in the second version, one runs subexpression B nevertheless. > If according to the rules of one particular system, patX and > patY mean different things, whereas in a second system, they are > completely interchangeable, then the first system can express > nuances that the second one cannot. When you have more nuances, > more expressivity, then you have more power, because you can say > things you could not otherwise say. Why do C and its derivatives > such as Perl have short-circuiting Boolean operators? Because > in older languages, such as Fortran and Pascal, where you did > not have them, one quickly found that this was cumbersome and > annoying. It's a question of speed vs. correctness. Correctness is important, but occasionally a little incorrectness is worthwhile for increased speed. Short-circuiting in C is a good example. It's *almost* correct -- the right boolean answer to the final expression WILL be returned, as will the correct matching string for /A|B/ or /B|A/. But it's not quite perfect. In C, side-effects of executing the short-circuited expressions (such as function calls) won't take effect. This is a worthwhile tradeoff, since it can save a LOT of execution time, and the programmer can work around it when it matters. But that semantic anomaly in C is well documented and accepted, so people get by with it. If the cost of "fixing" this non-greedy/leftmost semantic anomaly in Perl's regular expression engine is prohibitive, it's certainly acceptable to say it's a worthwhile tradeoff and document why it's the way it is. But to deny the validity of the alternative approach is a bit more extreme. The current behavior is "correct enough". It's not important to change it. But, if we can fix it and make it completely correct, it's at least worth considering -- but maybe not (in the end) worth actually implementing. > 4) It is more intuitive to the reader and the writer to minimize > strange action at a distance. It's more to remember; or, perhaps > better phrased, more to forget. That's why we don't like > variables set in one place magically affecting innocent code > elsewhere. Maybe it's more applicable here to say that that's > why having mixed precedences and associativities confuses people. > If in an expression like A->B->C->D, you had to know a prior when > evaluating A that D was going to be coming up, it would require > greater look-ahead, more mental storage. Even if a computer could > do it, people would find it harder. That's why we don't write > > &{&{$fnctbl{expr}}(arg1)}(arg2) > > when we can simply write > > $fnctbl{expr}->(arg1)->(arg2) > > It is not intuitive to people to have to do too much look-ahead, > or too much storage. Having distance items interact with one > another is confusing, and we've already got that situation with > backreferences, as in /(\w+)(\w+)\s+\2(\w+)/, which depending on > how you start weighting those +'s into +?'s, can really move > matters around. Let's not exacerbate the counterintuitiveness. Ah, but the current behavior *is* strange action at a distance. Instead of looking at the pattern and the part of the string you expect it to match, you have to consider at-a-distance factors such as the global priority of leftmost matching as a strictly-overriding factor, or whether or not some greedy subexpression earlier in the regular expression will eat the characters or not. I believe the current behavior is counterintuitive, which is exactly why I suggested changing it. > 5) It is more Perlian because of the principle that things that look > different should actually *be* different. /A|B/ and /B|A/ look > quite different. Thus, they should likewise *be* different. Really? So, the semantics of "if (!(expr)) {...}" should be different from "unless (expr) {...}" just because they look different? An optimizer would have a hell of a time getting ANYTHING done if it weren't for the semantic equivalence of different code that can accomplish the same thing. If the high-level semantics are logically equivalent, constraining those semantics for the sake of the implementation is unnecessarily complex. > >I didn't need the long-winded explanation, and I don't need help with > >understanding how that regexp matches what it does. I understand it > >perfectly well already. I'm no neophyte with regular expressions, even if > >Perl 5 does offer some regexp features I've never bothered to exploit... > > All NFAs prior to POSIX behaved in the fashion that Perl's continue > to behave in. I am surprised that over the long course of your > experiences with regexes, that you never noticed this fundamental > principle before. Not all NFAs are used for regexps. Moreover, it's already too low a level; once you start viewing the regexp as a series of instructions for how to move through the states of an NFA, you've already turned it into a program and lost the semantics I'm trying to preserve from the original pattern. And yes, I've used regular expressions heavily for MANY years now. It's not relevant to this discussion, except to say that I'm not ignorant of the way regular expressions work. > >My point is that the current behavior, while reasonable, isn't quite right. > > You're wrong. Don't call it "not right". It's perfectly correct > and consistent. It follows directly from historical behavior of > these things, and quite simply, it's in the rules. It's preferable > for the many reasons I outlined for you above. Historical precedent does have bearing on whether to implement it, but not on what the ideal semantics should be. The codified rules are related to the implementation, and again shouldn't have any bearing on the idealized semantics. As for being preferable, I found your arguments unconvincing; on the contrary, I consider the current behavior counterintuitive, which is the only incentive I see for changing it. > You just want something different. That does not make this wrong. > If I want ice cream and you want hamburgers, this does not make me > wrong. However, for you to tell me I really want hamburgers when I want the regular expressions to be intuitive and consistent across ALL levels, not just at the procedural NFA implementation level. What's wrong with that desire? > >design flaw in Perl 5's regular expressions. I was hoping we could have a > >rational debate about it, rather than propagating the same design flaw into > >Perl 6 through inertia or for legacy reasons. > > It is not a design "flaw". See above. My mistake in terminology. It's too loaded to call it a "flaw", which is why I've taken to calling it an "anomaly", which is more value-neutral. > >> You should probably read the whole chapter. > > >That was uncalled for! Just because I disagree with a particular detail of > >the design doesn't mean that I didn't _understand_ it... > > It was hardly uncalled for. You did not prove that you understood > it. You have not yet done so, as you did not cover any of the > ground which I did above. Perhaps if I had said "Read the source > code", that might have been uncalled for. But I did not say that, > and your retort was gratuitously combative. Combative? I don't believe so. I simply objected to the assumption that disagreeing with the design constitutes ignorance of it. I could prove that I understand regexps, but what would be the point? > You see, it just so happens that there's much more in that chapter, > including the referenced Little Engine section, that when read as > a whole, might well shed better understanding than mere extracts. > You should also check out _Mastering Regular Expressions_ says about > that. I won't quote for you. I'm tired of typing when you could > be reading. I've read the 2nd edition of the camel book, but I haven't seen the new material in the 3rd edition yet. (I've postponed buying it, to leave it as a potential Xmas present. I'll buy it immediately after Xmas if I don't get it as a present.) I glanced over the section in the bookstore. Looks like a good overview of the specific algorithm implemented, but it's not really relevent to the question of idealized semantics. > What is completely uncalled for, though, is to call something a bug > just because you either don't understand it or because you don't > like it. Lamentably, such behavior is remarkably frequent. You > are hardly the first. I apologize for seeming to call it a bug, and for calling it a "flaw" -- those sound too much like value judgements, and encourage people to entrench their positions rather than contemplating alternatives. Deven http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg572.html Fri, 15 Dec 2000 15:04:28 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Simon Cozens On Fri, Dec 15, 2000 at 05:20:35PM -0500, Deven T. Corzine wrote: > It's a pattern, not a program. Yes, it's straightforward to treat it as a > step-by-step procedure for matching that pattern, but by doing so, you lose > something of the gestalt of the whole. You may deal in patterns, but computers deal in programs. There's a reason Gestalt.pm hasn't made it yet. -- Putting a square peg into a round hole can be worthwhile if you don't mind a few shavings. -- Larry Wall http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg571.html Fri, 15 Dec 2000 15:00:02 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Tom Christiansen >Take. It. To. Private. Email. Please. I'm going to do better. I'm taking it to /dev/null. It's not worth my wasting my life over. Nobody agrees with this guy, so it doesn't matter. --tom http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg570.html Fri, 15 Dec 2000 14:31:43 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Deven T. Corzine On Fri, 15 Dec 2000, Tom Christiansen wrote: > >At worst, this should take no more than double the amount of time that the > >single pass did, probably less. Hardly a cause to concern ourselves with > >the heat death of the universe. > > Oh really? We have shown that for the kind of global overall > analysis that you are asking for, that in the general case, all > possible paths much be taken. You cannot short-circuit, because > you must first consider all possibilities and then weigh each valid > result against each other valid result. > > Consider something like /.*/ or /.*?/. For a string a length N, > there are > > (N+1) (N+2) > ----------- > 2 > > substrings that that matches. That means that an 80-byte string > has some 3321 possible substrings, all of which must be considered. > > In the short-circuiting version, the Engine need consider but one > single solitary case for each of those. 3321 is not the double of 1. > > Consider now something like /(.*)(.*)/ or /(.*?)(.*?)/ or /(.*)(.*?)/ > or /(.*)(.*?)/. You now have > > > 2 > ( (N+1) (N+2) ) > --------------- > 4 > > cases to consider, or, in the case of an 80-byte string, some > 11,029,041 possible choices. Where does the combinatorial math have any relevance? I'm not suggesting checking every possibility. > And with the current, normal, standard, short-circuiting system, > the Engine has to consider, hm... could it be just one possibility? > And that's just with two wildcards. People are often writing more > than that. You can still short-circuit. I'm not suggesting examining any further into the text to be searched than it already does. For the second pass, you can scan backwards for a rightmost match and short-circuit that, if you want. (But that would leave some small chance of a shorter match in the middle.) > Can you now see why this would be a problem? And how even in the > cases where it didn't actually break old programs (many of which it > would!) that it would cause many many them to apparently hang, > racing for electron death? It would be a problem if you actually had a combinatorial algorithm. At worst, you're re-testing the regexp N times, where N is the length of the matched string from the first pass, NOT the length of the original string to be searched. Oh, and where are these "many" old programs it would break? I'd still love to hear a real-world example that would actually break... Deven http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg569.html Fri, 15 Dec 2000 14:28:12 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Jarkko Hietaniemi Take. It. To. Private. Email. Please. -- $jhi++; # http://www.iki.fi/jhi/ # There is this special biologist word we use for 'stable'. # It is 'dead'. -- Jack Cohen http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg568.html Fri, 15 Dec 2000 14:22:52 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Deven T. Corzine On Fri, 15 Dec 2000, Tom Christiansen wrote: > >That would be a strange regexp, but I never suggested it. I suggested the > >regexp /b.*?d/ and pointed out that I believe "bccccd" is a more intuitive > >match than "bbbbccccd". That was the matching text, not the regexp, sorry > >if I didn't make that clear. > > Fine. What you said is > > first > find a b > then > find any non-newline, repeated 0 to N times > then > find a d > > What part of "first find a b" do you expect a randomizing solution to? > That's very clear. It's a pattern, not a program. Yes, it's straightforward to treat it as a step-by-step procedure for matching that pattern, but by doing so, you lose something of the gestalt of the whole. I'm aware of the mapping from the pattern to the procedure, but I'm retaining in mind the original intent of the pattern as well as the details of how it's matched. The semantics of the step-by-step procedure are very similar to those of the pattern, but not identical. The semantics I want it to mean are "find a match starting with b, ending with d, with as little as necessary in between". The difference between this and the procedural description you give above is what I'm talking about where we're losing something of the gestalt of the whole. While it's similar, it's not quite the same. Deven http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg567.html Fri, 15 Dec 2000 14:20:48 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Tom Christiansen And while I'm at it, consider /(.*)(.*)(.*)/, which we'll call /ABC./ You need to be able to say all of these independently and in conjunction with one another: whether segment A is longest or shortest overall whether segment B is longest or shortest overall whether segment C is longest or shortest overall whether segment AB is longest or shortest overall whether segment BC is longest or shortest overall whether segment ABC is longest or shortest overall Imagine wanting, in /ABC/, A and B to be minimal, C to be maximal, AB to be maximal, BC to be minimal, and ABC to be maximal. Does this not strike fear into your heart? The very notation we'd have to devise should itself be plenty sufficient to give you serious pause--and that's not even considering the heat-death problem of guaranteed worst-case behavior that the word "overall" mandates. Be very afraid. --tom http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg566.html Fri, 15 Dec 2000 13:48:59 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Tom Christiansen >At worst, this should take no more than double the amount of time that the >single pass did, probably less. Hardly a cause to concern ourselves with >the heat death of the universe. Oh really? We have shown that for the kind of global overall analysis that you are asking for, that in the general case, all possible paths much be taken. You cannot short-circuit, because you must first consider all possibilities and then weigh each valid result against each other valid result. Consider something like /.*/ or /.*?/. For a string a length N, there are (N+1) (N+2) ----------- 2 substrings that that matches. That means that an 80-byte string has some 3321 possible substrings, all of which must be considered. In the short-circuiting version, the Engine need consider but one single solitary case for each of those. 3321 is not the double of 1. Consider now something like /(.*)(.*)/ or /(.*?)(.*?)/ or /(.*)(.*?)/ or /(.*)(.*?)/. You now have 2 ( (N+1) (N+2) ) --------------- 4 cases to consider, or, in the case of an 80-byte string, some 11,029,041 possible choices. And with the current, normal, standard, short-circuiting system, the Engine has to consider, hm... could it be just one possibility? And that's just with two wildcards. People are often writing more than that. Can you now see why this would be a problem? And how even in the cases where it didn't actually break old programs (many of which it would!) that it would cause many many them to apparently hang, racing for electron death? --tom http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg565.html Fri, 15 Dec 2000 13:37:56 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Tom Christiansen >That would be a strange regexp, but I never suggested it. I suggested the >regexp /b.*?d/ and pointed out that I believe "bccccd" is a more intuitive >match than "bbbbccccd". That was the matching text, not the regexp, sorry >if I didn't make that clear. Fine. What you said is first find a b then find any non-newline, repeated 0 to N times then find a d What part of "first find a b" do you expect a randomizing solution to? That's very clear. --tom http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg564.html Fri, 15 Dec 2000 13:25:36 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Deven T. Corzine On Fri, 15 Dec 2000, Tom Christiansen wrote: > >You can't explain why "bbbbccccd" matches without making reference to the > >absolute priority of the leftmost rule. "bccccd" would still make sense > >(locally) without reference to that rule. > > Nope. Nope, nope, and nope. > > Th8is /bbbbccccd/ thing, which is completely unrealistic and > non-real-worldly, says: > > find a > b > such that this is immediately followed by > b > such that this is immediately followed by > b > such that this is immediately followed by > b > such that this is immediately followed by > c > such that this is immediately followed by > c > such that this is immediately followed by > c > such that this is immediately followed by > c > such that this is immediately followed by > d > > If you think that people for "find a b" to suddently mean > something stochastic, you know different people than I do. That would be a strange regexp, but I never suggested it. I suggested the regexp /b.*?d/ and pointed out that I believe "bccccd" is a more intuitive match than "bbbbccccd". That was the matching text, not the regexp, sorry if I didn't make that clear. Deven http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg563.html Fri, 15 Dec 2000 13:14:22 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Deven T. Corzine On Fri, 15 Dec 2000, Nathan Torkington wrote: > Tom Christiansen writes: > > >We may have to "agree to disagree". > > > > I shan't be doing that. > > I think you should, or at least agree to take it private and report > back to the list once you both come to a decision. Once you've stated > your position twice, there's not really much point in saying it a > third time. It's a sign that the discussion is turning on itself. Well, I've tried to clarify each restatement further and/or reframe it, rather than repeating exactly the same thing. That said, if it comes to the point where Tom and myself are the only ones interested in hashing out the issue, I'd be happy to take it offline until the two of us either agree or get tired of spinning in circles, then report back. > I'm glad this thread hasn't turned into a flamewar, but the stakes are > so low that I'm sure it could easily become one. I'd like to make > sure that everyone, Deven, Tom, and the rest of the list, relax and > realize that it's just a programming language. This _is_ a minor issue. I'm very much looking forward to Perl 6; since Perl 5 is such a joy to use, I fully expect Perl 6 to be even better. And I'll be using it, even if the semantic anomaly I believe I see is there. Deven http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg562.html Fri, 15 Dec 2000 13:13:07 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Deven T. Corzine On Fri, 15 Dec 2000, Tom Christiansen wrote: > >We may have to "agree to disagree". > > I shan't be doing that. Well, I'm still willing to discuss it, as long as it remains a discussion and doesn't become a flame war. > >I'm understand why people believe in > >the current semantics, but I've seen no indication that anyone else > >understands why I believe in these alternative semantics, or has tried. > >(Disagreeing with my conclusion doesn't preclude understanding where I'm > >coming from, but nobody seems to.) > > You have not addressed the heat death of the universe as I and > others have illustrated. Finding all possible matches is very often > completely infeasible. Please solve the electron decay problem > before continuing. Where does the heat death of the universe come in? I can give you a SIMPLE way to implement it, but I doubt it's the best way: apply the current rules first, then take the matching substring and search within THAT for a match with the priority of the rules inverted -- prefer non-greediness OVER leftmost matching for the second pass. This WILL get the result I suggest, preferring leftmost matching in general while still maximizing the amount non-greediness (stinginess) within those constraints. At worst, this should take no more than double the amount of time that the single pass did, probably less. Hardly a cause to concern ourselves with the heat death of the universe. Note, I do NOT recommend that implementation; it imposes an obvious speed penalty that shouldn't be imposed on people who don't care. It might make sense as an option, however. However, it does bring another possibility to mind. For those who are willing to pay a 100% speed penalty for simplicity, this sort of two-pass mode could be allowed, and allow juggling of both preferences? Maybe it would be useful to allow a "rightmost matching preference" option for people who could use it. (Might be helpful when working with Hebrew?) > >Well, obviously we could. Maybe we shouldn't, but we could do it. Many, > >many existing programs depended on Perl 4's magic behavior with @'s in > >double-quoted strings, yet Perl 5 broke them all with a fatal error during > >the compile phase. People survived. They adapted and moved on. > > Red herring. Counterexample to the assumption that we can't break existing code by changing the semantics. It's been done before, it could happen again. > >Unlike that incompatibility, this one would probably affect few > >programs. > > You're wrong. Incredibly wrong. Really? Do you have a real-world example that it would break, which would demonstrate how common such breakage would be? Deven http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg561.html Fri, 15 Dec 2000 13:09:53 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Nathan Torkington Tom Christiansen writes: > >We may have to "agree to disagree". > > I shan't be doing that. I think you should, or at least agree to take it private and report back to the list once you both come to a decision. Once you've stated your position twice, there's not really much point in saying it a third time. It's a sign that the discussion is turning on itself. I'm glad this thread hasn't turned into a flamewar, but the stakes are so low that I'm sure it could easily become one. I'd like to make sure that everyone, Deven, Tom, and the rest of the list, relax and realize that it's just a programming language. Nat http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg560.html Fri, 15 Dec 2000 13:08:16 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Tom Christiansen >You can't explain why "bbbbccccd" matches without making reference to the >absolute priority of the leftmost rule. "bccccd" would still make sense >(locally) without reference to that rule. Nope. Nope, nope, and nope. Th8is /bbbbccccd/ thing, which is completely unrealistic and non-real-worldly, says: find a b such that this is immediately followed by b such that this is immediately followed by b such that this is immediately followed by b such that this is immediately followed by c such that this is immediately followed by c such that this is immediately followed by c such that this is immediately followed by c such that this is immediately followed by d If you think that people for "find a b" to suddently mean something stochastic, you know different people than I do. --tom http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg559.html Fri, 15 Dec 2000 13:02:05 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Tom Christiansen >On Fri, 15 Dec 2000, Tom Christiansen wrote: >> >As for special-case rules, I believe that my proposed modification would >> >REMOVE a special-case semantic rule, at the cost of added complexity at the >> >implementation level. >> >> What is this alleged "special-case rule" you are talking about? >> There is no such thing. None. When you write /pat/, it means to >> find the first such pattern. There is no special case here. >The special case is "as long as it has the earliest starting position". >There may be many, many possible matches for a regexp in a given string, >especially with an expression as inclusive as ".*". You want to change things from "find a match", which has the obviously deterministic semantics of finding the first match, and alter that to mean "find all possible matches; now, amongst those...". This is much more complicated, at many levels. You have yet to address my long mail to you. You have yet to read MRE. >So, you have to apply some disambiguating rules to identify which matches >are "interesting" enough to be worth paying attention to. There is no ambiguity. Short-circuiting it not ambiguity. Stopping when you have an answer is not ambiguity. You are mistaken. --tom http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg558.html Fri, 15 Dec 2000 13:00:10 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Deven T. Corzine On Fri, 15 Dec 2000, Simon Cozens wrote: > On Fri, Dec 15, 2000 at 11:39:08AM -0800, Randal L. Schwartz wrote: > > Tell me how you can do that without breaking much existing code. > > Pssst, Randal, this is Perl 6, not p5p. That's why I never suggested fixing it in Perl 5 -- the chance of breaking any existing code did NOT seem worth considering for something so minor. But if Perl 6 is likely to break a few things anyway, then this would be an appropriate time to consider such a change. (If it actually makes sense.) Deven http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg557.html Fri, 15 Dec 2000 12:59:38 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Deven T. Corzine On Fri, 15 Dec 2000, Tom Christiansen wrote: > >Actually, I'm not sure -- it's conceivable that the ending point would ALSO > >move inward for a different starting point within the original match. But > >the ending point should NEVER be advanced further -- that's where the > >"leftmost over nongreedy" rule should apply instead... > > Please show us your implementation for a pattern matching engine > that lets the current end-point vary. This is very exciting, > because now you can relax the restriction that lookbehinds > must be constant width. I don't know if it can be implemented. I'm just not assuming out of hand that it CANNOT. Just because we can't see how to implement it doesn't mean there isn't a technique we haven't thought of yet. I won't go so far as to say that "I can't think of one" until such time as I've seriously TRIED to. When I do try, maybe then I'll be convinced that the answer is beyond me, or that it doesn't exist. Until then, I consider it an interesting, yet unresolved question. However, that wasn't the point of the above paragraph. I meant that you might have a match that the current engine would return, which starts at position x and ends at position y. Obviously, a less-greedy match is possible which starts at position x', where x < x' <= y. It might be possible that such a match could also end at position y' where y' < y. I don't have an example in mind, but I can't rule out the possibility. In both cases above, it's the same thing -- I'm reluctant to jump to the conclusion that something doesn't exist just because _I_ don't see it yet. Deven http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg556.html Fri, 15 Dec 2000 12:57:58 +0000 Re: Perl 5's "non-greedy" matching can be TOO greedy! by Tom Christiansen >We may have to "agree to disagree". I shan't be doing that. >I'm understand why people believe in >the current semantics, but I've seen no indication that anyone else >understands why I believe in these alternative semantics, or has tried. >(Disagreeing with my conclusion doesn't preclude understanding where I'm >coming from, but nobody seems to.) You have not addressed the heat death of the universe as I and others have illustrated. Finding all possible matches is very often completely infeasible. Please solve the electron decay problem before continuing. >Well, obviously we could. Maybe we shouldn't, but we could do it. Many, >many existing programs depended on Perl 4's magic behavior with @'s in >double-quoted strings, yet Perl 5 broke them all with a fatal error during >the compile phase. People survived. They adapted and moved on. Red herring. >Unlike >that incompatibility, this one would probably affect few programs. You're wrong. Incredibly wrong. --tom http://www.nntp.perl.org/group/perl.perl6.language.regex/2000/12/msg555.html Fri, 15 Dec 2000 12:56:08 +0000