develooper Front page | perl.perl5.porters | Postings from January 2001

Re: Announce : Tokener reporting patch

Thread Previous | Thread Next
Jarkko Hietaniemi
January 17, 2001 06:51
Re: Announce : Tokener reporting patch
Message ID:
> Why is this humungous version of cat interesting? Because of the way it does

So let's call this B::Sabertooth :-)

> it. The C file creates an interpeter, and passes your input to Perl; Perl
> parses it, and the hacked version of toke.c will put information about the
> tokenisation of that file into an array. mangle.c then exports that array to
> @main::toke, and parses and runs, which looks like this:
>     my $oldline = "x";
>     for (@toke) {
>         my ($type, $line, $flags, $text) = split /:/, $_, 4;
>         if ($line ne $oldline) { print "\n"; $oldline=$line; }
>         print $text;
>     }

Nifty.  Could this be turned into a backend?  Say,
perl -MB::Toke -eblah would spit out as tokens.

$jhi++; #
        # There is this special biologist word we use for 'stable'.
        # It is 'dead'. -- Jack Cohen

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About