[perl #63674] open() is not UTF-8-clean

This code

perl -lwe '$a="\x{e3}"; utf8::downgrade($a); open(my $x, ">", "x$a");
utf8::upgrade($a); open(my $y, ">", "y$a"); opendir(my $d, ".");
while(defined($_ = readdir($d))) { print unpack("H*", $_) unless /\A[
-~]*\z/ }'

actually sends different octets to open().
see here:

 perl -MDevel::Peek -lwe '$a="\x{e3}"; utf8::downgrade($a); print
Dump("x$a"); utf8::upgrade($a); print Dump("y$a");'

SV = PV(0x1b45c68) at 0x1b694e8
  REFCNT = 1
  FLAGS = (PADTMP,POK,pPOK)
  PV = 0x1b5fad0 "x\343"\0
  CUR = 2
  LEN = 8

SV = PV(0x1b45b58) at 0x1b730a0
  REFCNT = 1
  FLAGS = (PADTMP,POK,pPOK,UTF8)
  PV = 0x1b63b90 "y\303\243"\0 [UTF8 "y\x{e3}"]
  CUR = 3
  LEN = 8



On Fri Mar 06 03:03:13 2009, zefram@fysh.org wrote:
> This is a bug report for perl from zefram@fysh.org,
> generated with the help of perlbug 1.36 running under perl 5.10.0.
> 
> 
> -----------------------------------------------------------------
> [Please enter your report here]
> 
> $ perl -lwe '$a="\x{e3}"; utf8::downgrade($a); open(my $x, ">",
>    "x$a"); utf8::upgrade($a); open(my $y, ">", "y$a"); opendir(my $d,
>    "."); while(defined($_ = readdir($d))) { print unpack("H*", $_)
>    unless /\A[ -~]*\z/ }'
> 78e3
> 79c3a3
> $
> 
> Apparently open() is using, for the filename, the octet sequence used
> to represent the string internally, rather than the character sequence
> that the string actually represents.  This is a common problem with
> XS modules; I'm a bit surprised to see the core get it wrong too.
> (Not *very* surprised, though, because the way the SvUTF8 flag was
> injected invites this sort of mistake.)
> 
> [Please do not change anything below this line]
> -----------------------------------------------------------------
> ---
> Flags:
>     category=core
>     severity=medium
> ---
> Site configuration information for perl 5.10.0:
> 
> Configured by Debian Project at Thu Jan  1 12:43:38 UTC 2009.
> 
> Summary of my perl5 (revision 5 version 10 subversion 0)
>    configuration:
>   Platform:
>     osname=linux, osvers=2.6.26-1-686, archname=i486-linux-gnu-thread-
>    multi
>     uname='linux rebekka 2.6.26-1-686 #1 smp mon dec 15 18:15:07 utc
>    2008 i686 gnulinux '
>     config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN
>    -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr
>    -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10
>    -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5
>    -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local
>    -Dsitelib=/usr/local/share/perl/5.10.0
>    -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1
>    -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1
>    -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl
>    -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio
>    -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib
>    -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des'
>     hint=recommended, useposix=true, d_sigaction=define
>     useithreads=define, usemultiplicity=define
>     useperlio=define, d_sfio=undef, uselargefiles=define,
>    usesocks=undef
>     use64bitint=undef, use64bitall=undef, uselongdouble=undef
>     usemymalloc=n, bincompat5005=undef
>   Compiler:
>     cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN
>    -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE
>    -D_FILE_OFFSET_BITS=64',
>     optimize='-O2 -g',
>     cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing
>    -pipe -I/usr/local/include'
>     ccversion='', gccversion='4.3.2', gccosandvers=''
>     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
>     d_longlong=define, longlongsize=8, d_longdbl=define,
>    longdblsize=12
>     ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
>    lseeksize=8
>     alignbytes=4, prototype=define
>   Linker and Libraries:
>     ld='cc', ldflags =' -L/usr/local/lib'
>     libpth=/usr/local/lib /lib /usr/lib /usr/lib64
>     libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
>     perllibs=-ldl -lm -lpthread -lc -lcrypt
>     libc=/lib/libc-2.7.so, so=so, useshrplib=true,
>    libperl=libperl.so.5.10.0
>     gnulibc_version='2.7'
>   Dynamic Linking:
>     dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
>     cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib'
> 
> Locally applied patches:
> 
> 
> ---
> @INC for perl 5.10.0:
>     /etc/perl
>     /usr/local/lib/perl/5.10.0
>     /usr/local/share/perl/5.10.0
>     /usr/lib/perl5
>     /usr/share/perl5
>     /usr/lib/perl/5.10
>     /usr/share/perl/5.10
>     /usr/local/lib/site_perl
>     .
> 
> ---
> Environment for perl 5.10.0:
>     HOME=/home/zefram
>     LANG (unset)
>     LANGUAGE (unset)
>     LD_LIBRARY_PATH (unset)
>     LOGDIR (unset)
>     PATH=/home/zefram/pub/i686-pc-linux-
>   
gnu/bin:/home/zefram/pub/common/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/local/bin:/usr/games
>     PERL_BADLANG (unset)
>     SHELL=/usr/bin/zsh




---
via perlbug:  queue: perl5 status: open
https://rt.perl.org:443/rt3/Ticket/Display.html?id=63674

• [perl #63674] open() is not UTF-8-clean by Victor Efimov via RT
• [perl #63674] open() is not UTF-8-clean by Victor Efimov via RT