| Note: these documents may be out of date. Do not use as reference! |
To see what is currently happening visit http://www.perl6.org/
Regex assertions in plain Perl code
Maintainer: Bart Lateur <bart.lateur@skynet.be> Date: 28 Sep 2000 Last Modified: 1 Oct 2000 Mailing List: perl6-language-regex@perl.org Number: 348 Version: 2 Status: Frozen
Likely the most justifiable reason to want to be able execute embedded Perl code while trying to match a regex pattern, is to include some extra tests on the data just matched. The fact that the current implementation of the "experimental" /(?{...})/ construct, doesn't do just that, feels like a design mistake. So the proposal is to drop the current implementation of (?{...}) in favour of code assertions.
Only some final tweaks on the proposed draft for v2, and that's it.
Likely the most justifiable to want to be able to execute Perl code in a regex, is to have additional checks, to see if what you just matched is indeed acceptable. Quite a few prominent Perl people have expressed having been unpleasantly surprised when they found out that the current implementation of /?{...}/ doesn't do that.
Assertions are already familiar to people writing regexes: for example, /\b/, lookahead and lookbehind, and anchors, are all assertions, but only in matching subpatterns, not in code. Adding the option to do similar tests in code, seems like a powerful addition, while maintaining the basic spirit of regexes.
The main problem with the current implementation of (?{...}) is that it "always succeeds". In case of an assertion, it is the outcome of the execution of the embedded code, that decides if the test succeeds (return value is true), meaning it is safe to continue; or fails (return value is false), in which case the regex engine should abort exploring this branch, and immediately backtrack.
RFC 197 proposes a specific syntactic addition to regexes, just to check if a number is in a specific range. This is just one of the many things that could easily achieved using an assertion:
/(\d+)(?{$1<256})/ # proposed syntax, recycling Perl5's syntax
If your string = "1234", the subpattern /(\d+)/ initially matches all digits, stuffs them in $1, and calls the assertion code. This code additionally if what was matched is within range, in this case, less than 256. If this fails, the submatch fails.
This feature might result in matching some unexpected substrings: both "123 and "234" are acceptable matches according to this assertion. In order only to match what you really want to match, you may have to add some anchors, lookahead and/or lookbehind. So it will take some getting used to. This does not make it less valuable.
The current implementation for embedding code in regexes, is not aimed at assertions. Instead, it's only useable for its "effect at a distance", which is simply horrible. Basically, it simply provides a means to pass data around between various parts of the regex. This is a unhealty situation, since it makes regexes that use it, look incredibly obfuscated, and it requires deep knowledge on how a match in a regex proceeds.
In addition, it promotes changing global data structures. Since the code is executed everytime something promising is matched, and not just after a complete match, the code will probably be executed more often than desired.
That is the reason why Perl5 has built-in support to temporarily modify global data structures, so that the effect can automatically be undone when backtracking. This makes the implementation very tricky. Removing this feature, which is entirely unnecessary for the spirit of assertions, will likely reduce the complexity of implementation quite a bit.
Richard Proctor even wants to go further still: in his RFC 274, "Generalised Additions to Regexs", he proposes to add support for executing some embedded Perl code only while backtracking, specifically to undo changes by hand. I think that this would make the situation even worse than it is today.
Even knowing when and why the embedded code is executed, is far from obvious. Take this example:
$_ = "SKIP buzzer";
if(/(?{print "Testing\n";})([a-z])\1(?{print "Got a match: $1\n"})/) {
print "YES\n";
} else {
print "NO\n";
}
This prints:
Testing Testing Testing Got a match: z YES
The fact that the embedded code is called 3 times, not more, surely suprised me. It probably will surprise many people. Apparently, it is only executed once for every lowercase letter, not just for any character.
This inpredictability is yet another reason to discourage incrementally modify global data structures.
The above considerations, which are annoying at least for executing embedded generic code, are not hindrances at all for processing assertions.
In spirit, assertions are intended to have a local effect only, which is only used immediately to direct processing of a match. All it is supposed to do, is optionally veto a branch. It won't have an effect on some other bizarre location in the regex. There isn't a secret variable where the result is stored, to be used elsewhere. You're not supposed to change global data structures.
Therefore, we don't really need this localization. You basically don't need to undo anything. If you insist on modifying global data, you still can; only: you're on your own.
The current implementation of the /(?{code})/ construct makes simple assertion tests using embedded code hard, but not impossible. As MJD has pointed out: "It is not pretty".
(?{COND})
(?(?{not COND})(?!))
(?(?{not do { COND }})(?!))
(?(?{COND})|(?!))
This makes assertion look like a magic incantation, not something trivial, as it should have been.
In the spirit of assertions, local is not necessary. It might just as
well be removed. I think it would benefit ease of iumplementation quite
a bit, as I think it is likely the most complex part of it.
The behaviour for local, in the current embedded code, is rather
unintuitive. In the Perl5 code
/(?{local $c = $c + 1})/
it looks as if the scope for the localized variable should be scoped to the embedded code block. It is not. Instead, the value is maintained, permanently, unless the regex backtracks, in which case a previous value, from the part of the regex just in front of the backtracking, is restored. Are you still with me? I thought not.
This thing becomes largely unnecessary. First of all, its current main purpose, I think, is emulating assertions. Second, there is nothing you can do with it, that you can't do without it. For example, this regex:
/$pre(?(?=$lookahead)($pattern))$post/
behaves exactly like:
/$pre((?!$lookahead)|($pattern))$post/
i.e. if the lookahead succeeds, the pattern needs to be matched, otherwise it is skipped. In the latter, it is: if the negative lookahaed succeeds, nothing needs to be matched, otherwise the pattern must be matched.
The IF/ELSE case is slightly more complex. This:
/$pre(?(?=$lookahead)($if)|($else))$post/
can become:
/$pre((?=$lookahead)($if)|($?!$lookahead)($else))$post/
The other syntax version of conditional regexes, is this rather strange looking:
(?(1)yes:no)
for which the "1" is actually a representation of defined $1.
With the aid of the new assertions, this can become:
(?:(?{defined $1})yes|(?{not defined $1})no)
The outer parentheses only serve for grouping, and are not part of the basic syntax.
I see no reason to hang on to this feature.
I do not wish to express an opinion on whether this one should stay. Its main purpose is to allow matching of nested structures, i.e. a grammar instead of a regex. In that, it is not related to assertions in any way.
Replace the old implementation of (?{...}). Optionally, throw out all the "localizing" code. Do not set any special variables.
The new feature should:
I have been told that people in general do not use the current implementation of (?{...}) in production code, because it tends to dump core. Also, perlre calls this feature "highly experimental, and [it] may be changed or deleted without notice". You had been warned. ;-)
You can assign any value you want to $^R or any other variable, if you wish, and make the assertion succeed by adding a constant true value at the end of the code. This can be done automatically:
(?{ CODE })
(?{$^R = do { CODE }; 1})
This may require an extra "local", if that feature is to be retained.
If conditional regexes are to stay, oddly enough, then for such a regex
where the condition is an assertion in Perl code, the code doesn't need
to change at all. Indeed, this is one case of where (?{...})
currently indeed behaves as an assertion; $^R doesn't get set, either.
In case this feature is deleted, see possible substitutions in the discussion above.
thread on perl-5-porters, started by Jeffrey Friedl: "how to abort a match branch?"
<www.xray.mpe.mpg.de> <www.xray.mpe.mpg.de>
relevant messages from Mark-Jason Dominus:
<www.mail-archive.com> <www.mail-archive.com> <www.mail-archive.com>
It looks like in the first message, MJD mistook the current (?{...}) for an assertion. In a later message, he no longer made this mistake, and introduced the "unpretty" syntax for doing assertions in Perl5.
RFC 274: Generalised Additions to Regexs (see: the suggestion to add even more support for localization and unwinding)
RFC 308: Ban Perl hooks into regexes
perlre man page, sections on /(?{ code })/ and on /(?(condition)yes-pattern|no-pattern)/ and /(?(condition)yes-pattern)/