perl.perl5.porters

perl.perl5.porters http://www.nntp.perl.org/group/perl.perl5.porters/ ... Copyright 1998-2008 perl.org Sun, 27 Jul 2008 09:19:48 +0000 ask@perl.org Re: [perl #2783] Security of ARGV using 2-argument open -It'safeature by Abigail On Sat, Jul 26, 2008 at 08:59:59AM +0000, Ed Avis wrote: > Mark Mielke mark.mielke.cc> writes: > > >What legitimate scenarios use files with names "<x"? > > Very few. But if such a filename does exist, you'd want the program to handle > it in a sane way. > > Unfortunately the world is not always legitimate. Every program using while > (<>) has to come with a health warning that it is unsafe to use unless you kno > that no ><| characters are contained in the filenames. > > Remember gets() in the standard C library? What's the problem with it? No > legitimate use would ever enter a million characters of text when given a prom > to enter 'yes' or 'no'. And if users are so stupid as to enter too much text > and overflow the fixed buffer, that is a problem with the users and they shoul > be educated to take more care. > > Yet today, gets() is deprecated and nobody uses it - whether writing 'security > code' or not. It is just too dangerous. I think gets() is an excellent example. Guess what? gets() in the current C library is still the same as it was long time ago. The functionality of gets() didn't change. Additional functionality, using a different name was used. Perl went this way as well. 3-arg open already exists. Abigail http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138790.html Sun, 27 Jul 2008 00:46:20 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by jvromans [Quoting Mark Mielke, on July 26 2008, 21:06, in "Re: [perl #2783] Sec"] > This hardly constitutes evidence. It is your guess. That you haven't > been able to point to a real-life scenario of this issue being a problem > suggests to me that your guess is wrong. Sleep well, Mark. -- Johan http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138789.html Sat, 26 Jul 2008 23:00:38 +0000 Re: [perl #57322] perlbug AutoReply: ungetc() to :scalar might cause problems by Goro Fuji Oh, I forgot to attach the files. http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138788.html Sat, 26 Jul 2008 22:37:59 +0000 Re: [perl #2783] Security of ARGV using 2-argument open -It'safeature by Aristotle Pagaltzis * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 04:00]: > The assumption that users are using ">", "<", or "|" as the > first or last would seem to be proven false 1. Absence of evidence is not evidence of absence. 2. *I* never assumed that *users* would do this. What I *did* assume is that malicious attackers might be able to exploit sloppy code to create such files in places where subsequently a script using `while (<>)` can pick them up, presumably running under a different user name, thus allowing escalation of privilege. (Unfortunately `while (<>)` encapsulates so much convenient functionality all at once (opening of multiple files, nice error reporting, a special case for empty @ARGV, etc) that when put together with the 2-arg open hazard, it ends up being an attractive nuisance.) You’ll hopefully understand that I cannot give you any reliable data about how often attackers do this on my systems; if I knew of such cases I wouldn’t be faffing around on p5p. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138787.html Sat, 26 Jul 2008 20:17:06 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke For the record, though - Making <> use 3-argument open with "r" will not break any of my programs that I have ever written, and I don't think anybody who uses my programs would rely on the 2-argument open "feature", as I usually de-emphasize the Perl nature of my programs, and the programs are not primarily used by advanced Perl programmers. :-) The problem I have is the desire to significantly change Perl 5 by new comers. Perl 5 is a foundation at this point. Even back at Perl 5.004, compatibility was important. With Perl 5.10 I expect things to be slowing down, not speeding up. Perl 6 is the major change. Feel free to ensure that Perl 6 does not include a 2-arg open(). The feature under discussion has existed for as long as I remember using Perl. This suggests to me 1992 or earlier. I've known how it has worked, and never experienced a problem with it since then. The idea that in 2008 it would suddenly be a huge issue is a bit unbelievable to me. I totally agree that users should be using 3-arg open wherever possible, and that programs with security concerns should use taint-mode checking, and that taint-mode checking should flag <> as a potential problem. I disagree that existing behaviour needs to be broken to satisfy the above, or that more than the above is necessary. This is just one feature of many I've seen lately that has raised my concern. This particular feature on its own doesn't matter to me. Many similar changes that will eventually ensure that some program I have running on Perl 5.004 or Perl 5.6 today, will stop working in Perl 5.10 tomorrow, concerns me greatly. At present, Perl 5.6 is widely used on our Solaris machines, Perl 5.8 is widely used on our Linux and Windows machines, but Perl 5.004 is still used on some of our older HP-UX and Solaris machines. My programs already have Perl versions checks that adjust behaviour. I am not keen to add more. Compatibility is very important. If RHEL 6.x begins providing Perl 5.10, and a user begins using Perl 5.10 in our application, I don't want to be screwed. I hope this concern can be understood by the newcomers. I believe it is already well understood by the old timers. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138786.html Sat, 26 Jul 2008 19:25:13 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Aristotle Pagaltzis wrote: > * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:10]: > >> Again, what is the worst that can really happen? So the user is >> able to access a resource that they already have permission >> to. So what? There are so many ways to get a user to screw up >> > > Do you regard symlink attacks as a hole in the respective > software? If so, please explain to me how your opinion in that > case is not self-contradictory with the opinion you expressed > above. > I don't understand the question. If it is a reference to a comparison between different types of exploits for code that was written by an amateur or that does not have taint-mode checking enabled, I'm not convinced that this is a serious problem compared to others, or that others need to be fixed. In the other poster's example, I don't agree that gets() should be removed from libc, just because it can be exploited. I'm happy with gcc warnings listing it as a security concern when requested. Perl's taint-mode provides this feature and more. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138785.html Sat, 26 Jul 2008 19:03:34 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Aristotle Pagaltzis wrote: >> Good. This is interesting. Please report back on the number of >> actual files in 1988, 1998, and 2008, that use ">", "<", or >> "|" as the first or last character in the file name. >> > > You’re being onery on purpose, but I’ll humour you: I found none > of those, but a handful of files with spaces at the beginning of > their name. Thanks for playing. > With real metrics, the conclusion would seem to be that 2-arg open or implicit 2-arg open via <> should at least deal with filenames that begin with spaces at the beginning. The assumption that users are using ">", "<", or "|" as the first or last would seem to be proven false, at least in the one case you are able to measure. Given that you have the data, can you also check whether "http:" or "file:" is ever the prefix for a file name? Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138784.html Sat, 26 Jul 2008 18:56:55 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Aristotle Pagaltzis wrote: > * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:20]: > >> Feel free to educate me by pointing to a situation where open() >> handling http:// makes a PHP script exploitable, where blocking >> http:// would have made the PHP script impenetrable. >> > > PHP `require` uses `open`. > Be more specific please. You seem to be referencing some sort of "require $variable;" where $variable is from an external source. None of the PHP code I have does this. If you mean something else, please be more specific. If you mean to suggest that "require $variable;" is common - I would find this surprising. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138783.html Sat, 26 Jul 2008 18:53:09 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It's afeature by Mark Mielke Aristotle Pagaltzis wrote: > * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:10]: > >> This logic does not follow. That the shell provides a feature, >> therefore Perl does not need to provide the feature, is pretty >> backwards thinking in my opinion. You need to provide >> something more substantial than this. >> > Asserting that I have to doesn’t make it so. Please give me a use > case for 2-arg open magic using implicitly from the command line > that cannot be implemented with process substitution, or can only > be implemented with process substitution with significantly more > difficulty. If you can’t, then my assertion is correct. > No. Your assertion misses the point. The shell provides an echo command. Since the shell provides an echo command that maps similar to the perl print function, and since one can always convert perl -e print to echo, therefore print is not required in Perl. Right? I don't think so. That the shell provides a function is NOT evidence that Perl should not. You've already agreed that "-" provides value. Beyond this, it comes to opinion as to whether the function provides value in Perl, or whether it is a waste in Perl. A Windows COMMAND.COM user might like the flexibility provided by Perl implicit 2-arg open, as COMMAND.COM does not provide all of the same functions. There are other variations that are possible, although not necessarily any more real than the problems claimed, such as the ability to ensure that the next 2-arg open is only executed once the previous 2-arg open is read successfully. For example: ( expensive command 1; expensive command 2; ) | perl -ne 'if (/bad/) { die }' Vs: perl -ne 'if (/bad/) { die }' 'expensive command 1 |' 'expensive command 2 |' I don't think the above are legitimate uses, but then, I don't think <> is a real operator to be used by a complex program. >> Specifically, point out a place where the existing behaviour is >> causing a *real* problem. >> > > If the feature is no longer pulling its weight, then the burden > of proof is on your side, not mine. > > So go ahead: what use case does the feature solve? > Incorrect. Compatibility itself is a strong reason to keep a feature. You need to prove that: a) the feature provides substantial harm (either in terms of maintenance effort or support issues), b) by removing or restricting the feature, you will provide a benefit of some sort (reduced maintenance effort or reduced support issues), and c) that the feature is not relied on by other people and existing code will not break I don't think you've succeeded any of these three. I'm not convinced that this is a real problem. I'm not convinced that "fixing" it will reduce the maintenance cost or support issues, and I'm not convinced that the features are no longer in use. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138782.html Sat, 26 Jul 2008 18:51:03 +0000 Re: [perl #2783] Security of ARGV using 2-argument open -It'safeature by Aristotle Pagaltzis * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:20]: > Aristotle Pagaltzis wrote: >> Yes. The filenames in home directories on servers I >> administrate today are brimming with spaces, parens and square >> brackets. And those are from Windows users, who mount their >> home directories on their Windows machine or use SFTP or such >> to shuttle files around. No one (in relative terms) uses Unix >> without a desktop environment anymore either. Now what >> characters that Windows users cannot type in Save As do you >> think are a problem in GNOME or KDE? Question marks? Colons? >> Asterisks? > > Good. This is interesting. Please report back on the number of > actual files in 1988, 1998, and 2008, that use ">", "<", or > "|" as the first or last character in the file name. You’re being onery on purpose, but I’ll humour you: I found none of those, but a handful of files with spaces at the beginning of their name. Thanks for playing. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138781.html Sat, 26 Jul 2008 18:49:11 +0000 Re: [perl #2783] Security of ARGV using 2-argument open -It'safeature by Aristotle Pagaltzis * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:20]: > Feel free to educate me by pointing to a situation where open() > handling http:// makes a PHP script exploitable, where blocking > http:// would have made the PHP script impenetrable. PHP `require` uses `open`. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138780.html Sat, 26 Jul 2008 18:44:26 +0000 Re: [perl #2783] Security of ARGV using 2-argument open -It'safeature by Aristotle Pagaltzis * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:10]: > Again, what is the worst that can really happen? So the user is > able to access a resource that they already have permission > to. So what? There are so many ways to get a user to screw up Do you regard symlink attacks as a hole in the respective software? If so, please explain to me how your opinion in that case is not self-contradictory with the opinion you expressed above. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138779.html Sat, 26 Jul 2008 18:41:33 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It's afeature by Aristotle Pagaltzis * Mark Mielke <mark@mark.mielke.cc> [2008-07-27 03:10]: > Aristotle Pagaltzis wrote: >> Modern shells already have process substitution, which can >> do anything a user could do with magical filenames and more >> besides. *That* is a homogenous approach to argument processing, >> and you are absolutely right: it’s a BIG FAT FEATURE. So what is >> the value of retaining an API that can and does surprise people >> in nasty ways in exchange for no added functionality whatsoever? >> >> The *only* reason to keep that API is backwards compatibility. >> It’s a good reason to be sure, but I would prefer a way forward >> that satisfies that demand without perpertuating the vestigial >> pragmatics of yesteryear for all eternity and a day beyond. >> > > This logic does not follow. That the shell provides a feature, > therefore Perl does not need to provide the feature, is pretty > backwards thinking in my opinion. You need to provide > something more substantial than this. Asserting that I have to doesn’t make it so. Please give me a use case for 2-arg open magic using implicitly from the command line that cannot be implemented with process substitution, or can only be implemented with process substitution with significantly more difficulty. If you can’t, then my assertion is correct. > Specifically, point out a place where the existing behaviour is > causing a *real* problem. If the feature is no longer pulling its weight, then the burden of proof is on your side, not mine. So go ahead: what use case does the feature solve? Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138778.html Sat, 26 Jul 2008 18:36:45 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Aristotle Pagaltzis wrote: > * Mark Mielke <mark@mark.mielke.cc> [2008-07-26 22:30]: > >> More people use rich file system identifiers these days than >> UNIX file system. On Windows, it will happily accept http:// >> vs ftp:// vs file:// vs \\host\share in many of the "open >> file" dialogs. This is a good thing, and the trend should >> continue. PHP already supports this in their open functions >> and this is widely understood to be a major feature. >> > > Yes, it reduces the amount of work necessary to hack sites > written by PHP neophytes so much. All crackers agree, it’s the > coolest feature ever. > No - I don't think so. The crackers likely agree that eval or non-escaped literal substitution are the coolest features ever. I'm not sure why http:// would make the list. Feel free to educate me by pointing to a situation where open() handling http:// makes a PHP script exploitable, where blocking http:// would have made the PHP script impenetrable. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138777.html Sat, 26 Jul 2008 18:19:36 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Aristotle Pagaltzis wrote: > * Mark Mielke <mark@mark.mielke.cc> [2008-07-26 18:00]: > >> Do you have evidence? Perhaps a list of filenames from 1988, >> 1998, and 2008, to compare against? >> > > Yes. The filenames in home directories on servers I administrate > today are brimming with spaces, parens and square brackets. And > those are from Windows users, who mount their home directories on > their Windows machine or use SFTP or such to shuttle files > around. No one (in relative terms) uses Unix without a desktop > environment anymore either. Now what characters that Windows > users cannot type in Save As do you think are a problem in GNOME > or KDE? Question marks? Colons? Asterisks? > Good. This is interesting. Please report back on the number of actual files in 1988, 1998, and 2008, that use ">", "<", or "|" as the first or last character in the file name. Thanks, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138776.html Sat, 26 Jul 2008 18:17:09 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Aristotle Pagaltzis wrote: > * Mark Mielke <mark@mark.mielke.cc> [2008-07-25 23:15]: > >> How many people have been "burned" by this "problem"? Is the >> new class of dimwits exceedingly more dimwitted than the >> previous? Why was this not a problem 10 years ago? >> > > Just because you didn’t hear about it before doesn’t mean no one > else knew of it. I’ve been preaching the use of 3-arg open ever > since it was introduced, and I’ve known of the diamond operator’s > problem for just as long. > I think you misunderstand. I also agreed with 3-arg open ever since it was introduced. What I don't agree with, is that 2-arg open should be "broken" because of a fear that somebody somewhere might be burned. As for <>, it's much to trivial for any serious program with complex requirements, so I find it questionable that the subset is as serious of a concern as some people are making it out to be. eval "" is dangerous as well. Would you suggest blocking it as well? Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138775.html Sat, 26 Jul 2008 18:15:55 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It's afeature by Mark Mielke Aristotle Pagaltzis wrote: > Modern shells already have process substitution, which can > do anything a user could do with magical filenames and more > besides. *That* is a homogenous approach to argument processing, > and you are absolutely right: it’s a BIG FAT FEATURE. So what is > the value of retaining an API that can and does surprise people > in nasty ways in exchange for no added functionality whatsoever? > > The *only* reason to keep that API is backwards compatibility. > It’s a good reason to be sure, but I would prefer a way forward > that satisfies that demand without perpertuating the vestigial > pragmatics of yesteryear for all eternity and a day beyond. > This logic does not follow. That the shell provides a feature, therefore Perl does not need to provide the feature, is pretty backwards thinking in my opinion. You need to provide something more substantial than this. Specifically, point out a place where the existing behaviour is causing a *real* problem. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138774.html Sat, 26 Jul 2008 18:09:00 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Johan Vromans wrote: > Mark Mielke <mark@mark.mielke.cc> writes >> Do you have evidence? Perhaps a list of filenames from 1988, 1998, and >> 2008, to compare against? >> > > Ah, come on. There are many more people in 2008 using some Linux with > Firefox, Thunderbird and OpenOffice than there were in 1998 and 1988. > There are many more people in 2008 using some Linux totally unaware of > command line shells and special characters than there were in 1998 and > 1988. The chances that files with special characters in their names > are created is therefore bigger in 2008 than they were in 1998 and > 1988 This hardly constitutes evidence. It is your guess. That you haven't been able to point to a real-life scenario of this issue being a problem suggests to me that your guess is wrong. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138773.html Sat, 26 Jul 2008 18:06:22 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Andy Armstrong wrote: > On 26 Jul 2008, at 21:29, Mark Mielke wrote: >> More people use rich file system identifiers these days than UNIX >> file system. On Windows, it will happily accept http:// vs ftp:// vs >> file:// vs \\host\share in many of the "open file" dialogs. This is a >> good thing, and the trend should continue. PHP already supports this >> in their open functions and this is widely understood to be a major >> feature. > > s/feature/security hole/ :) Joking aside - productivity and security are always values at odds with each other. The most secure solution is the solution that provides no functions at all. Second best is a locked room with no network access and restricted physical access. The moment you let a user do _anything_ you open the door to exploitation. It is not correct to conclude that flexibility is unacceptable. It depends on the application requirements. I wouldn't hire a minimum wage Perl programmer with no experience to write a security system. It would be ridiculous. I, and the majority of the traditional Perl users, chose Perl because of its flexibility. It's ability to do things that other languages could not, or could not do well, at the time. Yes, flexibility means effort is required to secure the solution. This is normal, no matter what language you choose. Again, what is the worst that can really happen? So the user is able to access a resource that they already have permission to. So what? There are so many ways to get a user to screw up - that <> doesn't make my list. Social engineering is probably the easiest method of having a user screw up. Convince a stupid user to type "rm -fr /" as root as the solution to their problem with a driver or program that doesn't work. I've seen it happen. To put <> into this class is silly to me. Why not put open() in general in this class? Why not remove all the system calls. That way the user won't be able to do ANYTHING dangerous. See where this goes? :-) Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138772.html Sat, 26 Jul 2008 18:05:17 +0000 Re: [perl #2783] Security of ARGV using 2-argument open -It'safeature by Aristotle Pagaltzis * Mark Mielke <mark@mark.mielke.cc> [2008-07-26 22:30]: > More people use rich file system identifiers these days than > UNIX file system. On Windows, it will happily accept http:// > vs ftp:// vs file:// vs \\host\share in many of the "open > file" dialogs. This is a good thing, and the trend should > continue. PHP already supports this in their open functions > and this is widely understood to be a major feature. Yes, it reduces the amount of work necessary to hack sites written by PHP neophytes so much. All crackers agree, it’s the coolest feature ever. > Users who don't read the document will always be surprised. No one, and I mean NO ONE, reads the documentation, *every last line* of the documentation, before using a program. If *you* do, you are either a liar or a lawyer. “Read the documentation or get what you deserve” is utterly wrong-headed snobbism and completely counteproductive. > One could argue that an experienced user might be "surprised" > that "-" was NOT interpretted as STDIN. This argument is > entirely relative to the person's expectations. The special intepretation of `-` is a completely different beast from the special interpretation of `rm -rf ~ |`. There is no reason that getting rid of the latter would have to lose us the former. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138771.html Sat, 26 Jul 2008 16:43:40 +0000 Re: [perl #2783] Security of ARGV using 2-argument open -It'safeature by Aristotle Pagaltzis * Mark Mielke <mark@mark.mielke.cc> [2008-07-26 18:00]: > Johan Vromans wrote: >> [Quoting Mark Mielke, on July 26 2008, 11:44, in "Re: [perl #2783] Sec"] >> >>> You are mistaken to believe this is a new problem - my >>> questions is with the urgency. Why does the behaviour need to >>> change? Why is it a problem today when it was not a problem >>> 10 years ago? >> >> My point was that the problem is bound to get bigger now more >> people are using not-filename-limiting tools > > Do you have evidence? Perhaps a list of filenames from 1988, > 1998, and 2008, to compare against? Yes. The filenames in home directories on servers I administrate today are brimming with spaces, parens and square brackets. And those are from Windows users, who mount their home directories on their Windows machine or use SFTP or such to shuttle files around. No one (in relative terms) uses Unix without a desktop environment anymore either. Now what characters that Windows users cannot type in Save As do you think are a problem in GNOME or KDE? Question marks? Colons? Asterisks? Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138770.html Sat, 26 Jul 2008 16:31:39 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Andy Armstrong On 26 Jul 2008, at 21:29, Mark Mielke wrote: > More people use rich file system identifiers these days than UNIX > file system. On Windows, it will happily accept http:// vs ftp:// vs > file:// vs \\host\share in many of the "open file" dialogs. This is > a good thing, and the trend should continue. PHP already supports > this in their open functions and this is widely understood to be a > major feature. s/feature/security hole/ :) -- Andy Armstrong, Hexten http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138769.html Sat, 26 Jul 2008 16:19:25 +0000 Re: [perl #2783] Security of ARGV using 2-argument open -It'safeature by Aristotle Pagaltzis * Mark Mielke <mark@mark.mielke.cc> [2008-07-25 23:15]: > How many people have been "burned" by this "problem"? Is the > new class of dimwits exceedingly more dimwitted than the > previous? Why was this not a problem 10 years ago? Just because you didn’t hear about it before doesn’t mean no one else knew of it. I’ve been preaching the use of 3-arg open ever since it was introduced, and I’ve known of the diamond operator’s problem for just as long. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138768.html Sat, 26 Jul 2008 16:16:29 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It's afeature by Aristotle Pagaltzis * Tom Christiansen <tchrist@perl.com> [2008-07-20 07:30]: > And the day that I can no longer rely upon the overlying system > to automatically understand that "-" is STDIN for input or > STDOUT for output is the day that I will fork a parallel copy > of Perl that maintains traditional and expected behavior. > However, I trust that will never need to occur, ofr no pumpking > has ever been so blindly cavalier--which can probably be read > as "foolish" if you're of that bent. Go ahead. Being able to fork is what free software is all about. I do agree that treating the filename `-` as special is a very nice feature. It cannot cause problems of the sort that filenames with 2-arg-open metacharacters can, either. But then, there is no reason that this feature would have to be chucked out along with the intepretation of other special characters. > What you don't seem to understand is that taking a homogeneous > approach to argument processing is not a bug, but a feature--a > BIG FEATURE. Perhaps you're too much of a youngster to remember > the days that shells didn't glob command-line arguments for > you, and each program had to parse and process its own command > line string. But I am not. Those were dark days of > unpredictability. It's the wrong way to (not) go about things. I fail to follow. First you say that a program doing its own magical interpretation of metacharacters in filenames (like pipes and angle brackets) is a BIG FEATURE. Then you say that the days when programs did their own magical interpretation of metacharacters in filenames (like question marks and asterisks) were dark and ruled by unpredictability. Which one is it? Modern shells already have process substitution, which can do anything a user could do with magical filenames and more besides. *That* is a homogenous approach to argument processing, and you are absolutely right: it’s a BIG FAT FEATURE. So what is the value of retaining an API that can and does surprise people in nasty ways in exchange for no added functionality whatsoever? The *only* reason to keep that API is backwards compatibility. It’s a good reason to be sure, but I would prefer a way forward that satisfies that demand without perpertuating the vestigial pragmatics of yesteryear for all eternity and a day beyond. Regards, -- Aristotle Pagaltzis // <http://plasmasturm.org/> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138767.html Sat, 26 Jul 2008 16:09:52 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Zefram Mark Mielke wrote: >Ah. You didn't mean /$/ - you meant /...$/. Yes, I meant the $ regexp operator in general, not a pattern consisting only of that operator. > this is ANOTHER case, where I disagree that >anything should be "fixed". Oh, I think /$/ is impossible to fix in perl5. There are a great many programs that run regexps on newline-terminated input lines and use /$/ intending for it to match before the newline. These would seriously break if /$/ changed to match only at end of string. I wasn't arguing for this to change, but using it as an example of another case where a very short (and therefore attractive) operator tries to DWIM by complex magical behaviour and ends up surprising the programmer. I think <> could be changed, however. I'm OK with it continuing to process "-" as stdin; you're right that this is a common Unix convention. But its handling of ">foo" and "rm -rf / |" are certainly not conventional. I think (unlike /$/) that the intentional uses of those features are sufficiently rare that it's worth breaking them to make the operator less surprising for everyone else. > Users who intend to match end-of-string and >only end-of-string should be using \z. Yes, indeed they need to. But most of them don't: they use the shorter and more familiar operator, and end up with a different program from the one they intended to write. And I remember the days when /\z/ didn't exist: for a while I used /(?!.)/s as the only way to be sure. After writing my earlier message, I had a think about what start/end anchor operators should logically exist, depending on how your string is structured into lines, and which ones we actually have in Perl. I came up with these sets, defining the behaviour explicitly in terms of the regexp operators with simplest behaviour: PURPOSE START END sensible pairs: single undelimited line \A \z single line with \n terminator \A (?=\n\z) zero or more \n-terminated lines (?:\A|(?<=\n))(?!\z) (?=\n) one or more \n-separated lines (?:\A|(?<=\n)) (?=\n|\z) actual pairs: /^...$/, /\A...\Z/ \A (?=\n?\z) /^...$/m (?:\A|(?<=\n)(?!\z)) (?=\n|\z) I am mystified as to the circumstances under which one might actually want the behaviour of /$/ (without /m) or /^/m. Certainly they can be correctly used, with a bit of care, but as far as I can see they never completely match the actual semantics of what constitutes a line start or end. > just so that somebody who didn't understand what >they were doing in the first place, will not break in an obscure use case. I think these (<> and some of the regexp things) are unreasonably difficult to understand. /^/m is so difficult to understand that its own implementors have trouble with it. It was documented incorrectly in perlre for years, until I discovered the undocumented /(?!\z)/ bit of its behaviour and pointed it out (bug #27053, resolved by a documentation change in 5.10). I understand these operators. I will jump through the necessary hoops to write the program that I intend, even if the hoop is using a 20-character regexp (as in the table above) instead of the single character that I know from grep. Not to put too fine a point on it, I'm unburned because I know the language inside out and I'm anal about correctness. I'm in a small minority on all three points. >Users who don't read the document will always be surprised. Users who read the documentation for perl programs that use <> (such as our hypothetical wc-in-perl) generally don't get told about the magic meaning of "rm -rf / |" as an argument. They will be surprised and burned by the present behaviour. You have to read quite a long way into the <> documentation (in perlop) before it mentions pipe magic. It starts off saying it "emulate[s] the behavior of sed and awk", which covers "-" and the no-arguments case. It then speaks about filenames, and gives an example using glob() in a way that makes it (the example) vulnerable to the very problems we are discussing. It never mentions the magic interpretation of ">foo" or that it loses trailing spaces. Magic behaviour of "-" is less surprising, less dangerous, and more likely to be documented than any of the others. It also has a well-known workaround, ./*, which is also good for avoiding interpretation as switches. The pipe thing can't be worked around that way: consider the file "./; rm -rf / |". Pipe magic here is very nasty. >In this case, the expectations should be well instilled. I expect generally unpredictable behaviour if I give a filename containing unusual characters to a program written by anyone else. (Not specifically programs written in scripting languages, or programs written by amateurs, but just programs written by anyone who wasn't me. Hell, I once encountered commercial *kernel* code that lost trailing spaces in filenames.) This expectation has been instilled by years of experience, and reinforced by the psychological understanding that most programmers just don't think about the unusual cases. -zefram http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138766.html Sat, 26 Jul 2008 15:18:32 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Johan Vromans Mark Mielke <mark@mark.mielke.cc> writes: > Do you have evidence? Perhaps a list of filenames from 1988, 1998, and > 2008, to compare against? Ah, come on. There are many more people in 2008 using some Linux with Firefox, Thunderbird and OpenOffice than there were in 1998 and 1988. There are many more people in 2008 using some Linux totally unaware of command line shells and special characters than there were in 1998 and 1988. The chances that files with special characters in their names are created is therefore bigger in 2008 than they were in 1998 and 1988. -- Johan http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138765.html Sat, 26 Jul 2008 14:11:48 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Zefram wrote: > Mark Mielke wrote: > >> I don't understand. What is /$/ reasonably supposed to do? >> > > It is very frequently used in an attempt to anchor to the end of the > string. Such as > > die "invalid keyword" unless $keyword =~ /^(foo|bar|baz)$/; > > In this context the programmer usually doesn't intend to accept "foo\n". > Newline is a shell metacharacter, of course, and often significant in > file formats, so there's lots of scope for breakage. > Ah. You didn't mean /$/ - you meant /...$/. Similar to other issues, the real-life problems with accepting "foo" vs "foo\n" are pretty few. I expect the /...$/ cases are more than the 2-argument cases in terms of real-life problems, however, this is ANOTHER case, where I disagree that anything should be "fixed". Users who intend to match end-of-string and only end-of-string should be using \z. It is not the job of Perl to redefine long-standing operators to become less functional and create compatibility problems, just so that somebody who didn't understand what they were doing in the first place, will not break in an obscure use case. >> Please point out a real-life >> program that uses /$/ that isn't mere theory. >> > > I presume you mean a real-life program that misbehaves due to misuse > of /$/. On a quick look through /usr/local/share/perl, I found this > in Carp::Clan: > > :unless ( /^-?\d+(?:\.\d+(?:[eE][+-]\d+)?)?$/ > : ) # Looks numeric > :{ > : s/([\\\'])/\\$1/g; # Escape \ and ' > ... > > This is used when displaying function arguments in a stack trace: it's > trying to show numeric values as unquoted numbers and any other defined > value as a quoted string. So these are how it's meant to work: > > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo('abc') called at -e line 1 > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc\n")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo('abc\x0A') called at -e line 1 > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo(123) called at -e line 1 > $ > > And this one goes wrong: > > $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123\n")' > Carp::Clan::__ANON__(): a at -e line 1 > main::foo(123 > ) called at -e line 1 > $ > > OK, this one's not likely to produce a security hole, but it's just the > first instance I set eyes on. It's a very common antipattern. > > It seems like a pretty harmless one. >> Plain honest-to-Ritchie file access is obsolete. >> > > I respectfully disagree. I find the Unix filesystem is an excellent > abstraction, and I use it all the time. > More people use rich file system identifiers these days than UNIX file system. On Windows, it will happily accept http:// vs ftp:// vs file:// vs \\host\share in many of the "open file" dialogs. This is a good thing, and the trend should continue. PHP already supports this in their open functions and this is widely understood to be a major feature. >> I want to see http:// >> automatically parsed. >> > > It's possible to map URIs into Unix filename space. (I had a go at > this myself for Linux a while back, but didn't develop it very far.) > It's also possible to map Unix filenames into URI space. If you want > access to both HTTP and local files in one context, you can still get this > with pure file operations or with pure URI operations, at your option. > If you insist on mixed parsing, then you're a long way from providing a > filename context, and you have to document the rules in order for users > to be able to make intentional use of the features. > > The rules are pretty well recognized, even by the non-technical. The non-technical would recognize http:// a long time before they would recognize some non-standard overlay of the UNIX file name space with URI. >> We're decades past the requirements you are >> raising, >> > > The requirement to not surprise the user? > Users who don't read the document will always be surprised. In standard UNIX, passing a "-" in has traditionally meant STDIN. One could argue that an experienced user might be "surprised" that "-" was NOT interpretted as STDIN. This argument is entirely relative to the person's expectations. In this case, the expectations should be well instilled. Perl has done what it has done for over a decade, and if somebody is truly surprised today - they should pick up the manual and give it another read. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138764.html Sat, 26 Jul 2008 13:29:49 +0000 [perl #57302] Unexpected conversion from "/usr/local" to ".../.." by Markus Elfring # New Ticket Created by Markus Elfring # Please include the string: [perl #57302] # in the subject line of all future correspondence about this issue. # <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=57302 > This is a bug report generated with the help of perlbug 1.36 running under perl 5.10.0. ----------------------------------------------------------------- [Please enter your report here] I downloaded the source file package "http://www.cpan.org/src/perl-5.10.0.tar.gz". The included script "Configure" asked me some questions for program locations and directories while reasonable default values like "/usr/bin/perl" and "/usr/local" were provided. I chose some options and tried a compilation. I decided to adjust a few options and run the configuration script again. I was surprised that a couple of dots were presented as new default values that I did not change before in the previous run. I noticed that the dots were recorded in the file "config.sh". So I replaced the dots in assignments for variables like "binexp" and "prefix" with understandable text by an editor. I saw that the unexpected dots notation reappeared after each reconfiguration try. Are there any open issues for the build script maintenance? [Please do not change anything below this line] ----------------------------------------------------------------- --- Flags: category=install severity=medium --- Site configuration information for perl 5.10.0: Configured by elfring at Wed Jul 23 10:28:59 CEST 2008. Summary of my perl5 (revision 5 version 10 subversion 0) configuration: Platform: osname=linux, osvers=2.6.25.11-default, archname=x86_64-linux-thread-multi-ld uname='linux sonne 2.6.25.11-default #2 smp preempt mon jul 14 22:26:03 cest 2008 x86_64 x86_64 x86_64 gnulinux ' config_args='' hint=previous, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=define, use64bitall=define, uselongdouble=define usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DSOCKS -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O3', cppflags='-D_REENTRANT -D_GNU_SOURCE -DSOCKS -fno-strict-aliasing -pipe -I/usr/local/include -D_REENTRANT -D_GNU_SOURCE -DSOCKS -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64' ccversion='', gccversion='4.3.1 20080507 (prerelease) [gcc-4_3-branch revision 135036]', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16 ivtype='long', ivsize=8, nvtype='long double', nvsize=16, Off_t='off_t', lseeksize=8 alignbytes=16, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /lib64 /usr/lib64 /usr/local/lib64 libs=-lnsl -ldb -ldl -lm -lcrypt -lutil -lpthread -lc perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc libc=/lib/libc-2.8.so, so=so, useshrplib=true, libperl=libperl.so gnulibc_version='2.8' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/local/lib/perl/5.10.0/x86_64-linux-thread-multi-ld/CORE' cccdlflags='-fPIC', lddlflags='-shared -O3 -L/usr/local/lib' Locally applied patches: --- @INC for perl 5.10.0: /usr/local/lib/perl/5.10.0/x86_64-linux-thread-multi-ld /usr/local/lib/perl/5.10.0 /usr/local/lib/perl/site/5.10.0/x86_64-linux-thread-multi-ld /usr/local/lib/perl/site/5.10.0 . --- Environment for perl 5.10.0: HOME=/home/elfring LANG=de_DE.UTF-8 LANGUAGE (unset) LD_LIBRARY_PATH (unset) LOGDIR (unset) PATH=/opt/kde3/bin:/usr/lib/mozart/bin:/home/elfring/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/cross/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/gnome/bin:/usr/lib/qt3/bin PERL_BADLANG (unset) SHELL=/bin/bash http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138763.html Sat, 26 Jul 2008 10:22:08 +0000 [perl #57288] PerlIO::via::QuotedPrint not flushing by Kevin Ryde # New Ticket Created by Kevin Ryde # Please include the string: [perl #57288] # in the subject line of all future correspondence about this issue. # <URL: http://rt.perl.org/rt3/Ticket/Display.html?id=57288 > When a PerlIO::via::QuotedPrint layer is pushed on an opened file, autoflush or an explicit ->flush() doesn't send output to the file immediately. The program qp.pl below shows the problem. I hoped the OUT->flush would make the "hello" appear in /tmp/foo immediately, but that doesn't happen, not until after the program exits after the sleep. I guess QuotedPrint doesn't supply a FLUSH routine, and the default for PerlIO::via in that case is to do nothing, so a top-level flush doesn't reach the :perlio+:unix full-buffered output layer below. Several other PerlIO::via output modules on cpan behave the same, I guess having used QuotedPrint as a model. I wonder if either - The PerlIO::via docs could advise every output layer to supply a FLUSH function calling $fh->flush on the sublayer. - PerlIO::via could make such a "propagated" flush the default if you don't supply a FLUSH routine. The latter could make life easier for straightforward non-buffering transformer layers. The only question would be whether anyone has omitted a FLUSH func deliberately wanting not to call flush on the sublayer. That'd seem a bit unlikely to me, or only applicable to something that was designed to be at the bottom of the stack anyway (an output capture of some sort say) ... --- Flags: category=library severity=wishlist --- Site configuration information for perl 5.10.0: Configured by Debian Project at Thu May 8 13:32:42 UTC 2008. Summary of my perl5 (revision 5 version 10 subversion 0) configuration: Platform: osname=linux, osvers=2.6.24.4, archname=i486-linux-gnu-thread-multi uname='linux ninsei 2.6.24.4 #1 smp preempt fri apr 18 15:36:09 pdt 2008 i686 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2 -g', cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='', gccversion='4.2.3 (Debian 4.2.3-5)', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /usr/lib64 libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=/lib/libc-2.7.so, so=so, useshrplib=true, libperl=libperl.so.5.10.0 gnulibc_version='2.7' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib' http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138762.html Sat, 26 Jul 2008 10:12:03 +0000 B-Generate-1.12_10 stable & uploaded by Reini Urban Hi Josh, I've uploaded the latest series of B-Generate-1.12_x which is becoming stable now. These are the current maintainers of B::Generate B::Generate ABERGMAN Artur Bergman co-maint JJORE B::Generate CLKAO Chia-liang Kao (高嘉良) co-maint JJORE B::Generate GRUBER Anton Berezin co-maint JJORE B::Generate JCROMIE Jim Cromie co-maint JJORE B::Generate JJORE Joshua ben Jore first-come JJORE B::Generate MSCHWERN Michael G Schwern co-maint JJORE B::Generate MSTROUT Matt S Trout co-maint JJORE B::Generate SWALTERS Scott Walters co-maint JJORE The list of fixes for the latest releases for 5.10 comes from: Anton Berezin Reini Urban Dmitry Karasik (See Changes) The module pm says this: MAINTAINERS This is just a list of people who have submitted patches to the module. To find someone to actually maintain this, please try contacting perl5-porters. Josh Jore, Michael Schwern, Jim Cromie, Scott Walters. This list is incomplete. I'm asking for permissions for co-maint and release a 1.13 then. -- Reini Urban http://phpwiki.org/ http://murbreak.at/ http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138761.html Sat, 26 Jul 2008 09:05:25 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Johan Vromans wrote: > [Quoting Mark Mielke, on July 26 2008, 11:44, in "Re: [perl #2783] Sec"] > >> You are mistaken to believe this is a new problem - my questions is with >> the urgency. Why does the behaviour need to change? Why is it a problem >> today when it was not a problem 10 years ago? >> > > My point was that the problem is bound to get bigger now more people > are using not-filename-limiting tools Do you have evidence? Perhaps a list of filenames from 1988, 1998, and 2008, to compare against? Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138760.html Sat, 26 Jul 2008 08:58:49 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Ed Avis wrote: > Mark Mielke mark.mielke.cc> writes: > > >> What legitimate scenarios use files with names "<x"? >> > > Very few. But if such a filename does exist, you'd want the program to handle > it in a sane way. > > Unfortunately the world is not always legitimate. Every program using while > (<>) has to come with a health warning that it is unsafe to use unless you know > that no ><| characters are contained in the filenames. > No, it doesn't. At least not until you can prove this is a real issue hurting people every day. > Remember gets() in the standard C library? What's the problem with it? No > legitimate use would ever enter a million characters of text when given a prompt > to enter 'yes' or 'no'. And if users are so stupid as to enter too much text > and overflow the fixed buffer, that is a problem with the users and they should > be educated to take more care. > > Yet today, gets() is deprecated and nobody uses it - whether writing 'security > code' or not. It is just too dangerous. > You are exaggerating. If anybody truly allocated a buffer for gets() that was a million characters, it wouldn't be a problem. Allocating a Mbyte for an input buffer "just in case", however, is ridiculous. People didn't allocate a million characters - they allocated 64, or 512, or 1024. The chance that this is hurt is far, far greater. You are correct that the interface is wrong for gets() is wrong, but in your analogy you have ignored the fact that 2-arg open is a FEATURE not a LIMITATION. The situations are not the same. In gets(), there is a limitation which can cause the executing program to see data corruption, arbitrary code execution, or program termination. In Perl <>, there is a feature that users can pass shell pipes instead of file names or "-" instead of STDIN. The worst that can happen is that the user requests a file they have permission to overwrite, be overwritten. This might happen if the user is naive enough to use ">" as the first character in their file name. What cases of this problem exist? What cases prove that the value of the feature is far less than the cost of allowing the feature? Is this theory, or do you have a legitimate complaint? >> I strongly disagree. You are turning an existing feature with existing >> syntax into a non-feature, because you think there MIGHT be a >> theoretical somebody incompetent enough to hang themselves with a bit of >> rope. >> > > First I would like to say that this is best not treated as a moral issue. Those > bad programmers, deserve what they get, we should teach them a lesson, etc. > You are incorrect. Programming languages can do baby sitting duty, or they can provide plenty of rope. There is a need for both in this world. Both types of languages exist today. Perl has traditionally been the "plenty of rope" type, and this is a major reason why Perl adoption was so high for many years. What other scripting languages allowed users to run arbitrary system calls? You are suggesting that this Perl tradition be turned over. Your reason for requesting it is a trivial theoretical case that has probably happened once in the life time of Perl. > If the measure of incompetence is using 'while (<>)' in a program without > realizing that it is dangerous, then I have been incompetent many times and so > have lots of others. Again, have a look at the examples you can find on Google > Code Search. All of these people have been incompetent and written programs > that are unsafe to use on arbitrary lists of files. All the programs either > need to be patched to use a safe form of open, or else need to mention the > gotcha in their documentation. > Your definition of unsafe and mine are different. What is the worst that can happen with these theoretically unsafe programs? Somebody has a file called ">fun". they pass "*" in. Their program overwrites "fun". So what? In order for their to be damage, "fun" would need to exist and contain valuable information. What are the chances that both ">fun" and "fun" exist? The only danger is in setuid usage, where one uses special privileges to overwrite a file they should not have access to. This is precisely what taint-mode is for, and I fully agree that <> should raise issues in taint-mode. I don't agree at all that non-taint-mode should warn or error or have alternate behaviour from <> as it is today. > The bar for incompetence or being a dimwit is set remarkably low IMHO. Surely > you should not need to be one of the expert 10% (or even the top 90%) to read > some files and count the number of lines. Yet I bet that if you get a sample of > perl programmers and ask them to implement a simple 'wc -l' in perl, more than > nine out of ten will write one that breaks if given a filename beginning with >, > and most of those without even thinking there might be a problem. > By "break" you mean - fail to allow the use of ">" in the beginning of a file, which is a pretty minimal case. Again - show examples of this problem in the field. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138759.html Sat, 26 Jul 2008 08:57:26 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by jvromans [Quoting Mark Mielke, on July 26 2008, 11:44, in "Re: [perl #2783] Sec"] > You are mistaken to believe this is a new problem - my questions is with > the urgency. Why does the behaviour need to change? Why is it a problem > today when it was not a problem 10 years ago? My point was that the problem is bound to get bigger now more people are using not-filename-limiting tools. -- Johan http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138758.html Sat, 26 Jul 2008 08:55:31 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Johan Vromans wrote: > Mark Mielke <mark@mark.mielke.cc> writes: > > >> Why was this not a problem 10 years ago? >> > > Long time ago filenames were meant for shell- and command line apps. > Modern *ix applications (e.g., word processors) allow people to freely > supply any filename they want, spaces and whatever included. E.g., > when saving a HTML page, using the <title> would be an acceptabe > default choice for a file name. > > So whether it's wrong, or stupid, or whatever -- the chances of the > problem happening will grow. > My memory tells me that "word processors" and the like have *always* accepted weird characters. In 1990 or so, I still remember the 'ls' output being messed up because one of the filenames had a BACKSPACE in it. Nothing has changed in this regard. Perhaps people use GUIs more - but even this I don't believe. I've used UNIX GUIs on HP-UX and Solaris since the late 1980s. Frame Maker didn't have file name restrictions that I recall? You are mistaken to believe this is a new problem - my questions is with the urgency. Why does the behaviour need to change? Why is it a problem today when it was not a problem 10 years ago? Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138757.html Sat, 26 Jul 2008 08:44:14 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Johan Vromans Mark Mielke <mark@mark.mielke.cc> writes: > Why was this not a problem 10 years ago? Long time ago filenames were meant for shell- and command line apps. Modern *ix applications (e.g., word processors) allow people to freely supply any filename they want, spaces and whatever included. E.g., when saving a HTML page, using the <title> would be an acceptabe default choice for a file name. So whether it's wrong, or stupid, or whatever -- the chances of the problem happening will grow. -- Johan http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138756.html Sat, 26 Jul 2008 07:56:35 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Zefram Mark Mielke wrote: >I don't understand. What is /$/ reasonably supposed to do? It is very frequently used in an attempt to anchor to the end of the string. Such as die "invalid keyword" unless $keyword =~ /^(foo|bar|baz)$/; In this context the programmer usually doesn't intend to accept "foo\n". Newline is a shell metacharacter, of course, and often significant in file formats, so there's lots of scope for breakage. > Please point out a real-life >program that uses /$/ that isn't mere theory. I presume you mean a real-life program that misbehaves due to misuse of /$/. On a quick look through /usr/local/share/perl, I found this in Carp::Clan: :unless ( /^-?\d+(?:\.\d+(?:[eE][+-]\d+)?)?$/ : ) # Looks numeric :{ : s/([\\\'])/\\$1/g; # Escape \ and ' ... This is used when displaying function arguments in a stack trace: it's trying to show numeric values as unquoted numbers and any other defined value as a quoted string. So these are how it's meant to work: $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc")' Carp::Clan::__ANON__(): a at -e line 1 main::foo('abc') called at -e line 1 $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("abc\n")' Carp::Clan::__ANON__(): a at -e line 1 main::foo('abc\x0A') called at -e line 1 $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123")' Carp::Clan::__ANON__(): a at -e line 1 main::foo(123) called at -e line 1 $ And this one goes wrong: $ perl -MCarp::Clan=confess -we 'sub foo { confess "a" } foo("123\n")' Carp::Clan::__ANON__(): a at -e line 1 main::foo(123 ) called at -e line 1 $ OK, this one's not likely to produce a security hole, but it's just the first instance I set eyes on. It's a very common antipattern. >Plain honest-to-Ritchie file access is obsolete. I respectfully disagree. I find the Unix filesystem is an excellent abstraction, and I use it all the time. > I want to see http:// >automatically parsed. It's possible to map URIs into Unix filename space. (I had a go at this myself for Linux a while back, but didn't develop it very far.) It's also possible to map Unix filenames into URI space. If you want access to both HTTP and local files in one context, you can still get this with pure file operations or with pure URI operations, at your option. If you insist on mixed parsing, then you're a long way from providing a filename context, and you have to document the rules in order for users to be able to make intentional use of the features. > We're decades past the requirements you are >raising, The requirement to not surprise the user? -zefram http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138755.html Sat, 26 Jul 2008 04:03:28 +0000 Re: [perl #57244] crash with recursive regexp by Zefram Eric Brine wrote: >That should be No, it does work as written, provided that $datum_rx is in scope where $array_rx is executed. Your version doesn't change that: $datum_rx still needs to be in scope at execution. In the real application some of the regexps are "our" rather than "my", and the (??{}) uses a fully-qualified name: this is the only way to make the recursive reference work independently of execution context. >but that's not related to the bug in Perl. Indeed. -zefram http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138754.html Sat, 26 Jul 2008 03:14:05 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Ed Avis Mark Mielke mark.mielke.cc> writes: >What legitimate scenarios use files with names "<x"? Very few. But if such a filename does exist, you'd want the program to handle it in a sane way. Unfortunately the world is not always legitimate. Every program using while (<>) has to come with a health warning that it is unsafe to use unless you know that no ><| characters are contained in the filenames. Remember gets() in the standard C library? What's the problem with it? No legitimate use would ever enter a million characters of text when given a prompt to enter 'yes' or 'no'. And if users are so stupid as to enter too much text and overflow the fixed buffer, that is a problem with the users and they should be educated to take more care. Yet today, gets() is deprecated and nobody uses it - whether writing 'security code' or not. It is just too dangerous. >I strongly disagree. You are turning an existing feature with existing >syntax into a non-feature, because you think there MIGHT be a >theoretical somebody incompetent enough to hang themselves with a bit of >rope. First I would like to say that this is best not treated as a moral issue. Those bad programmers, deserve what they get, we should teach them a lesson, etc. If the measure of incompetence is using 'while (<>)' in a program without realizing that it is dangerous, then I have been incompetent many times and so have lots of others. Again, have a look at the examples you can find on Google Code Search. All of these people have been incompetent and written programs that are unsafe to use on arbitrary lists of files. All the programs either need to be patched to use a safe form of open, or else need to mention the gotcha in their documentation. The bar for incompetence or being a dimwit is set remarkably low IMHO. Surely you should not need to be one of the expert 10% (or even the top 90%) to read some files and count the number of lines. Yet I bet that if you get a sample of perl programmers and ask them to implement a simple 'wc -l' in perl, more than nine out of ten will write one that breaks if given a filename beginning with >, and most of those without even thinking there might be a problem. -- Ed Avis <eda@waniasset.com> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138753.html Sat, 26 Jul 2008 02:00:20 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Glenn Linderman wrote: > On approximately 7/25/2008 2:13 PM, came the following characters from > the keyboard of Mark Mielke: >> Glenn Linderman wrote: >>> One could speculate about "while(<>)" using 2-arg open if -T is set >>> and 3-arg open otherwise, with "while(<<>>)" or "use magical while;" >>> causing 2-arg open to be used even without -T. >>> >>> How else can we encourage the dimwits to continue using perl, after >>> they get burned by stuff like this, if we don't improve the language? >> >> Responding to Glenn's although not specifically to him, but to all >> with this opinion: >> >> How many people have been "burned" by this "problem"? Is the new >> class of dimwits exceedingly more dimwitted than the previous? Why >> was this not a problem 10 years ago? > > I probably shouldn't have used the word "dimwit", but... it seemed > like the argument for not breaking while(<>) is very much "elitist"... > the "I'm smart enough not to get trapped, so why aren't you?". So to > try to talk to that class of people, I used the term. Forget what term you used. Do you have an ACTUAL CASE of failure, or is this only theory? > But yes, the new class of dimwits (I might as well keep using the > term, since I've started) are more dimwitted than the previous class, > because they were raised on Windows, instead of Unix. In Unix class, > shell script handling of special characters, and the "rm *" with the > file named -f to kill you, and the file name -i to protect you, were > regularly taught to newbies. In Windows, no one teaches you anything, > you get to learn by the seat of your pants. If you take classes, > someone might think to mention stuff like that, but if you read > documentation, particularly our Perl documentation, it is easy to > overlook such an "esoteric" topic, even if it is noticed, because > really, they are looking for how to open files, not how to learn > defensive programming. I disagree. Windows and UNIX have been around for about as long as Perl 5. Again, do you have an ACTUAL CASE of failure, or is this only theory? >> I don't see the point. There is no value in restricting Perl's >> functionailty, so that some theoretical dimwit will have one less >> theoretical security hole in one theoretical scenario. Where is the >> proof that this "security problem" is causing problems, and why >> aren't these dimwits having their hands cut off to prevent them from >> programming? > > Nope, not suggesting restricting it, just making easy things safer, > and unsafe things harder. Things could be made "safe" by chmod ugo-x /usr/bin/perl. You are not answering the question. What is the proof that this "security problem" is causing problems? Where is this outbreak you are trying to prevent? > > Fortunately, a large majority of dimwits put spaces in their > filenames, and have enough problems with that, that they never think > to put in > < | and other line noise. I think it is purely that that > has kept the problem from getting out of hand. More guessing. Do you have any evidence? Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138752.html Sat, 26 Jul 2008 01:10:54 +0000 Re: [perl #2783] Security of ARGV using 2-argument open - It'safeature by Mark Mielke Zefram wrote: > Mark Mielke wrote: > >> How many people have been "burned" by this "problem"? >> > > I don't get burned by it: this kind of issue is why I avoid using such > DWIMish heuristic features. I only use <> in throwaway programs that > I'm using on known inputs, and if I want to read arbitrary files then > I use three-argument open or equivalent. It's a pity that such a short > operator isn't available to do something of wider utility. > > Same issues apply with several regexp features, such as /$/ (funny > treatment of newline at end of string) and /\s/ (utf8 flag dependence). > I see people get burned by the unexpectedly complex behaviour of > these operators all the time. I use the more verbose, more explicit, > non-obvious constructions that actually do what I mean. I don't hugely > object to the extra typing, but I wish that the operators with plainer > semantics were shorter and more accessible so that less nitpicky > programmers would use them by default. > I don't understand. What is /$/ reasonably supposed to do? Do these theoretical people really get burned? Please point out a real-life program that uses /$/ that isn't mere theory. >> I don't see the point. There is no value in restricting Perl's >> functionailty, >> > > I don't think anyone's arguing for the functionality to not be there. > Just give it a longer name, that indicates that there's more going > on than meets the eye. If we were starting from scratch, I'd say > that a plain honest-to-Ritchie file access should be called "open", > and the one that can run commands, expand shell variables, hack root, > and whatnot should be called "magic_open". See, nice descriptive names, > with added Huffman value Plain honest-to-Ritchie file access is obsolete. I want to see http:// automatically parsed. We're decades past the requirements you are raising, and in the last decade that open(2 args) has existed, I don't recall a single problem such as you describe. Cheers, mark -- Mark Mielke <mark@mielke.cc> http://www.nntp.perl.org/group/perl.perl5.porters/2008/07/msg138751.html Sat, 26 Jul 2008 01:08:04 +0000