develooper Front page | perl.perl5.porters | Postings from July 2008

Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)

Thread Previous | Thread Next
Tom Christiansen
July 29, 2008 14:20
Re: Creative and *routine* use of so-called "magic" ARGV (was [perl #2783] Security of ARGV using 2-argument open)
Message ID:
In-Reply-To: Message from Abigail <> 
   of "Tue, 29 Jul 2008 09:29:13 +0200." <20080729072913.GM30221@almanda> 

> On Tue, Jul 29, 2008 at 01:08:21AM -0400, Mark Mielke wrote:

>> If I want to write a secure application, I'm not sure I would choose  
>> Perl. If I did use Perl, or any other language, I would expect to have  
>> to put in effort and have a clue.

> I wouldn't expect anyone to write a non-trivial secure application without
> having to put in effort and having a clue. Regardless of the language.

It's amazing what people expect.  If I've heard it once--and I have--then
I've heard it said a hundred times that, "We can't use {grep or ++ or
multiple inheritance or WHATEVER} in our Perl code because the Virtual
Basic programmers we've hired to maintain our code don't understand
{WHATEVER}."  And nobody (but me) even ever laughs, despite how if you'd
just s/Perl/C++/, then I find that everyone *would* laugh.

These aren't competent professionals.  They're programming strumpets just
doing what they're paid to do, and thinking doesn't appear part of the job
description (or of their managers').

> But if you think this is an important issue, wouldn't it make much
> more sense to teach people RIGHT NOW that they shouldn't rely on
> while (<>) automatically open files for them, then to wait a couple
> of years before 5.12 is released, and then a few more years before 
> everyone has upgraded to 5.12?

Because while you can lead a horse to water, but you can't make it drink.
And if you let it drink freely on its own, it may drink itself to death.

Which just shows that you can't win for trying.

I spent years on it--YEARS.  They still just won't learn.  Things like this
have been in the FAQ since BEFORE Camel-1 even; they've been in plenty of
the available documentation since then--a body of writing I note with
extreme dismay and contemptuous disdain, has been remarkably bowdlerized
with Pythonesque Right-Thinking-Only since last I visited it.  

It used to be right there prominently in open in the perl 4.036 manpage:

       The  filename  that  is  passed  to open will have
       leading and trailing whitespace deleted.  In order
       to  open a file with arbitrary weird characters in
       it, it's necessary  to  protect  any  leading  and
       trailing whitespace thusly:

	   $file =~ s#^(\s)#./$1#;
	   open(FOO, "< $file\0");

But that is now hidden away at best.  And it's no longer
even in the FAQ at all, which is where it belongs!

Certainly the 1997 version of perlfaq5 covered it.  It once read:

   How can I open a file with a leading ">" or trailing blanks?

   Normally perl ignores trailing blanks in filenames, and interprets
   certain leading characters (or a trailing "|") to mean something
   special. To avoid this, you might want to use a routine like this. It
   makes incomplete pathnames into explicit relative ones, and tacks a
   trailing null byte on the name to make perl leave it alone:

	sub safe_filename {
	    local $_  = shift;
	    return m#^/#
		    ? "$_\0"
		    : "./$_\0";

	$fn = safe_filename("<<<something really wicked   ");
	open(FH, "> $fn") or "couldn't open $fn: $!";

   You could also use the sysopen function (see sysopen).

What do we get today?  Bowdlerization!

    How can I open a file with a leading ">" or trailing blanks?

    (contributed by Brian McCauley)

    The special two argument form of Perl's open() function ignores
    trailing blanks in filenames and infers the mode from certain leading
    characters (or a trailing "|"). In older versions of Perl this was the
    only version of open() and so it is prevalent in old code and books.

    Unless you have a particular reason to use the two argument form you
    should use the three argument form of open() which does not treat any
    characters in the filename as special.

	    open FILE, "<", "  file  ";  # filename is "   file   "
	    open FILE, ">", ">file";     # filename is ">file"

HELLO?  What happened to the right answer that was there before?

But what should I expect?  The perl faq is now a document that thinks 
this is somehow clear and consistent code, and it's anything but:

    # ...
    local($^I, @ARGV) = ('.orig', glob("*.c"));
    while (<>) {
	    if ($. == 1) {
		    print "This line should appear at the top of each file\n";
	    s/\b(p)earl\b/${1}erl/i;        # Correct typos, preserving case
	    close ARGV if eof;              # Reset $.
    # $^I and @ARGV return to their old values here

It's *terribly* misleading!  Notice that while loop's close curly.  Oh
wait, got it wrong, didn't you?  This is very poor style.  Worse, it
doesn't protect the things that it should protect, and doesn't even do what
it claims to do!  At the very least, it needs to be rewritten as:

    # ...
	local $^I = ".orig";
	local($_, *ARGV, *ARGVOUT);
	@ARGV = glob("*.c");
	while (<>) {  
	    if ($. == 1) {
		print "This line should appear at the top of each file\n";
	    s/\b(p)earl\b/${1}erl/i;        # Correct 1st typo; preserve *only* initial case 
	    print;			    # This will go to ARGVOUT, the temp file
	    close ARGV if eof;              # Reset $.
	}  # and rename files on each new $ARGV; 
	# XXX: pity we didn't check the close, but neither did Perl
    # Here $^I, $_, @ARGV, $ARGV, *ARGV{IO}, and *ARGVOUT{IO}
    # all return to any previous values held before block entrance.

There are entries like that throughout.  Makes me want to 
pull my hair out for wasted effort.

Which brings me to my final complaint of the day.  That code is 
wrong because Perl itself is wrong.  No really, it is.


    % df -h .
    Filesystem     Size    Used   Avail Capacity  Mounted on
    /dev/wd0a      124M    117M   -5.0K   100%    /

    % ls -l it*
    -rw-r--r--  1 tchrist  wheel  0 Jul 29 14:06 it

    % perl -i.orig -pe 'print "this is more stuff\n"' it

    % echo $?

    % ls -l it*
    0 -rw-r--r--  1 tchrist  wheel  0 Jul 29 15:05 it
    0 -rw-r--r--  1 tchrist  wheel  0 Jul 29 14:06 it.orig

To this day, Perl's implicit closing of files doesn't warn you of errors,
let alone exit nonzero.  This makes it do wrong thing and not even tell you
it did them wrong.  This is a *true* problem, because checking for the
success of print() is neither necessary nor sufficient to detect the
success of print().  Yes, you read that correctly.  It's because of
buffering, plus the persistence of the err flag on the file structure.

I've never convinced anybody this is important.  Since *every* 
program should do this for correctness, it has to be in the run-time
system to avoid it ever being forgotten.  Sure, there's stuff
like this you can do:

    END { close(STDOUT) || die "can't close stdout: $!" }

But that sort of thing should happen on all implicitly closed things.  And
it really must.  Even IO::Handle::close doesn't bother.  Perl knows what
handles it's flushing closing during global destruction, at least if you
don't just blindly fflush(0).

Watch again, starting from before the bogus, ill-reported rename attempt:

    % df -h .
    Filesystem     Size    Used   Avail Capacity  Mounted on
    /dev/wd0a      124M    117M   -5.0K   100%    /

    % ls -l it
    -rw-r--r--  1 tchrist  wheel  0 Jul 29 14:06 it

    % perl -e 'print "stuff\n"' >> it; echo $?                      

    % perl -e 'open(my $fh, ">>it") || die "open $!"; print $fh "stuff\n"; print STDOUT "ok now\n"'
    ok now

    % echo $?

    % ls -l it
    -rw-r--r--  1 tchrist  wheel  0 Jul 29 14:06 it

This is all incorrect behavior on Perl's part.  Even cat knows better!

    % echo foo | cat >> it; echo $?
    cat: stdout: No space left on device

| Notice how even my cat is smarter than your perl!? :( |

What's that about, eh?

But I've been saying this for years, just like everything else.
Makes no difference.  Depressing, eh?

BTW, this proves my point about checking for print's status
being a waste of time, neither necessary nor sufficient to 
catch failed prints!

    % perl -e 'open(my $fh, ">>it") || die "open $!"; print $fh "stuff\n" or die "print $!"; print STDOUT "ok now\n"'; echo exit status was $?
    ok now
    exit status was 0

And you can't get the danged thing to detect its folly:

    % perl -WE 'open(my $fh, ">>it") || die "open $!"; say $fh "stuff" or die "print $!"; say "ok now"' ; echo exit status was $?
    ok now
    exit status was 0

I firmly believe that this needs to be a part of *EVERY* Perl program,
which consequently means it should *not* be there, but in the core itself:

    END { close(STDOUT) || die "can't close stdout: $!" }

And I believe this *much* more than some here believe <ARGV> needs to
implicitly map {"< $_\0"} @ARGV (since that breaks existing working code,
whereas mine fixes existing broken code).  Furthermore, I also believe all
*IMPLICIT* closes that fail need to generate a mandatory io warning, but
that of STDOUT should be a fatal.  I've believed all this for a long time;
said it often enough.  Anything else is just plain wrong behavior.  I'm not
quite sure whether ARGVOUT failure should be a warning or a fatal, but it
should be *something* suitably noisy.

And no, Abigail, *please* understand that I don't mean *you* personally in
any way in this.  Why, I'd even bet a beverage you agree with me about this
fh silent-closing problem!  Rather I'm referring to the overall groupthink
and practice here, whereby simple stuff that's wrong just *never* gets
addressed, let alone fixed, but esoteric stuff that's well-documented and
at best a fringe issue gets all the airspace.  Meanwhile, the documentation
continues to--well, you know.  Best be charitable and just say "ramify with
baroque embellishments", but you quite know what that means, I'm sure.

'Nuff of this: time for sunshine.  


    Quotes below are all from Dorothy Parker:

	"You can lead a whore to culture, but you can't make her think."

	"You can't teach an old dogma new tricks."    

	"If you want to know what God thinks of money, just look at 
	 the people he gave it to."   			 

	"All I need is room enough to lay a hat and a few friends."

	"Brevity is the soul of lingerie."   

	"The cure for boredom is curiosity. There is no cure for curiosity."

Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About