develooper Front page | perl.perl5.porters | Postings from May 2016

[perl #128143] Inconsistent behaviour when decoding in substitution

Thread Next
From:
Michael Budde
Date:
May 14, 2016 19:56
Subject:
[perl #128143] Inconsistent behaviour when decoding in substitution
Message ID:
rt-4.0.18-12885-1463128856-1334.128143-75-0@perl.org
# New Ticket Created by  Michael Budde 
# Please include the string:  [perl #128143]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=128143 >


This is a bug report for perl from mbu@jobindex.dk,
generated with the help of perlbug 1.40 running under perl 5.24.0.


-----------------------------------------------------------------
[Please describe your issue here]

     $x =~ s/(.)/$latin1->decode($1)/ge
     $x =~ s/(.)/Encode::decode('latin1', $1)/ge

These two expressions can in some cases give different results. See attached
example program for a specific example.

I'm fully aware that decoding an already decoded string is not a good idea.
With that said, I think the inconsistent behaviour shown in the example is
surprising at least. I would assume that the four versions would give the same
result (whatever that would be).

I have reproduced the issue on Perl 5.24.0, 5.22.1 and 5.18.2.

----------------- Example ---------------------------------------
use Encode;
use Devel::Peek;

my $latin1 = find_encoding('latin1');

my $x = "\N{U+00C5}\N{U+0080}";

sub version1 {
     my $in = shift;
     return Encode::decode('latin1', $in);
}

sub version2 {
     return Encode::decode('latin1', $_[0]);
}

sub version3 {
     my $in = shift;
     return $latin1->decode($in);
}

sub version4 {
     return $latin1->decode($_[0]);
}

my $a = $x =~ s/(.)/version1($1)/ger; Dump($a); # "\xC5\x80"
my $b = $x =~ s/(.)/version2($1)/ger; Dump($b); # "\xC5\x80"
my $c = $x =~ s/(.)/version3($1)/ger; Dump($c); # "\xC5\x80"
my $d = $x =~ s/(.)/version4($1)/ger; Dump($d); # "\xC3\x85\xC2\x80"
----------------- Example end -----------------------------------

----------------- Example output --------------------------------
SV = PVMG(0x24f01f0) at 0x23ff2f0
   REFCNT = 1
   FLAGS = (POK,pPOK,UTF8)
   IV = 0
   NV = 0
   PV = 0x23f8bd0 "\303\205\302\200"\0 [UTF8 "\x{c5}\x{80}"]
   CUR = 4
   LEN = 10
SV = PVMG(0x24f0220) at 0x24f6440
   REFCNT = 1
   FLAGS = (POK,pPOK,UTF8)
   IV = 0
   NV = 0
   PV = 0x2403de0 "\303\205\302\200"\0 [UTF8 "\x{c5}\x{80}"]
   CUR = 4
   LEN = 10
SV = PVMG(0x24f0280) at 0x24f62a8
   REFCNT = 1
   FLAGS = (POK,pPOK,UTF8)
   IV = 0
   NV = 0
   PV = 0x2403e40 "\303\205\302\200"\0 [UTF8 "\x{c5}\x{80}"]
   CUR = 4
   LEN = 10
SV = PVMG(0x24f02b0) at 0x24f67a0
   REFCNT = 1
   FLAGS = (POK,IsCOW,pPOK,UTF8)
   IV = 0
   NV = 0
   PV = 0x2427690 "\303\203\302\205\303\202\302\200"\0 [UTF8 "\x{c3}\x{85}\x{c2}\x{80}"]
   CUR = 8
   LEN = 10
   COW_REFCNT = 0
----------------- Example output end ----------------------------

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
     category=library
     severity=low
     module=Encode
---
Site configuration information for perl 5.24.0:

Configured by mbu at Thu May 12 11:18:57 CEST 2016.

Summary of my perl5 (revision 5 version 24 subversion 0) configuration:
    
   Platform:
     osname=linux, osvers=3.13.0-85-generic, archname=x86_64-linux
     uname='linux mbu 3.13.0-85-generic #129-ubuntu smp thu mar 17 20:50:15 utc 2016 x86_64 x86_64 x86_64 gnulinux '
     config_args='-de -Dprefix=/home/mbu/perl5/perlbrew/perls/perl-5.24.0 -Aeval:scriptdir=/home/mbu/perl5/perlbrew/perls/perl-5.24.0/bin'
     hint=recommended, useposix=true, d_sigaction=define
     useithreads=undef, usemultiplicity=undef
     use64bitint=define, use64bitall=define, uselongdouble=undef
     usemymalloc=n, bincompat5005=undef
   Compiler:
     cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
     optimize='-O2',
     cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
     ccversion='', gccversion='4.8.4', gccosandvers=''
     intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
     d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
     ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
     alignbytes=8, prototype=define
   Linker and Libraries:
     ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
     libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed 
/usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib 
/usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
     libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
     perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
     libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
     gnulibc_version='2.19'
   Dynamic Linking:
     dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
     cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'

Locally applied patches:
     Devel::PatchPerl 1.38

---
@INC for perl 5.24.0:
     /home/mbu/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0/x86_64-linux
     /home/mbu/perl5/perlbrew/perls/perl-5.24.0/lib/site_perl/5.24.0
     /home/mbu/perl5/perlbrew/perls/perl-5.24.0/lib/5.24.0/x86_64-linux
     /home/mbu/perl5/perlbrew/perls/perl-5.24.0/lib/5.24.0
     .

---
Environment for perl 5.24.0:
     HOME=/home/mbu
     LANG=en_US.UTF-8
     LANGUAGE=en_AG:en
     LD_LIBRARY_PATH (unset)
     LOGDIR (unset)
     PATH=/home/mbu/perl5/perlbrew/bin:/home/mbu/perl5/perlbrew/perls/perl-5.24.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/mbu/usr/bin:/home/mbu/.fzf/bin:/home/mbu/.cargo/bin
     PERLBREW_BASHRC_VERSION=0.75
     PERLBREW_HOME=/home/mbu/.perlbrew
     PERLBREW_MANPATH=/home/mbu/perl5/perlbrew/perls/perl-5.24.0/man
     PERLBREW_PATH=/home/mbu/perl5/perlbrew/bin:/home/mbu/perl5/perlbrew/perls/perl-5.24.0/bin
     PERLBREW_PERL=perl-5.24.0
     PERLBREW_ROOT=/home/mbu/perl5/perlbrew
     PERLBREW_VERSION=0.75
     PERL_BADLANG (unset)
     SHELL=/bin/bash


Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About