develooper Front page | perl.perl5.porters | Postings from January 2001

Announce : Tokener reporting patch

Thread Next
Simon Cozens
January 17, 2001 03:00
Announce : Tokener reporting patch
Message ID: contains three files: a
patch to Perl, a C file and a Perl file. 

If you patch Perl, build it, and then build the C file as an embedding
application using the Perl library you've just created, you'll have a program
which takes a Perl script as standard input, and spits out the same script on
standard output, skipping blank lines and comments.

Why is this humungous version of cat interesting? Because of the way it does
it. The C file creates an interpeter, and passes your input to Perl; Perl
parses it, and the hacked version of toke.c will put information about the
tokenisation of that file into an array. mangle.c then exports that array to
@main::toke, and parses and runs, which looks like this:

    my $oldline = "x";
    for (@toke) {
        my ($type, $line, $flags, $text) = split /:/, $_, 4;
        if ($line ne $oldline) { print "\n"; $oldline=$line; }
        print $text;

Now, why is that cool, and why am I telling you about it? 

Ever wanted a really good syntax highlighting editor for Perl? Or an
indent(1)-style utility? Or a pretty-printer? We know that the only thing that
can parse Perl is Perl - so let's use Perl to do it. $type, in the above code,
tells us what type each token is, (and $flags provides additional information
about it) so we can highlight it in the appropriate way. This makes it really
trivial for us to build syntax-aware tools.

Yes, it keeps spacing, apart from tabs.
Yes, it's aware of BEGIN{} and CHECK{} blocks and it doesn't execute them.
Yes, this means it's probably only 95% instead of 100% accurate. Sorry.
Yes, it's available under GPL/AL. Let me know if you do cool things with it.

Have a nice day.

I saw a beggar leaning on his wooden crutch,
    And he said to me, "You must not ask too much".
And I saw a pretty woman leaning in her darkened door,
    She cried to me, "Hey, why not ask for more?"

Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About