develooper Front page | perl.perl5.porters | Postings from March 2006

[perl #38812] encoding error in UTF-8 locales

Thread Previous | Thread Next
From:
Vincent Lefevre
Date:
March 30, 2006 01:00
Subject:
[perl #38812] encoding error in UTF-8 locales
Message ID:
rt-3.0.11-38812-131684.10.2029449569782@perl.org
# New Ticket Created by  Vincent Lefevre 
# Please include the string:  [perl #38812]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/rt3/Ticket/Display.html?id=38812 >


This is a bug report for perl from vincent@vinc17.org,
generated with the help of perlbug 1.35 running under perl v5.8.8.


-----------------------------------------------------------------
[Please enter your report here]

Consider the following script:

#!/usr/bin/env perl

use strict;
use encoding ':locale';
use XML::LibXML;

my $parser = XML::LibXML->new();
my $doc = $parser->parse_file('file.xml');

open OUT, '>:encoding(iso-8859-1)', 'out.xml' or die "$!";
my $string = $doc->toString(0);
$string =~ s/xml//s;
print OUT $string;
close OUT or die "$!";

where file.xml is:

<?xml version="1.0" encoding="iso-8859-1"?>
<!-- é -->
<root/>

I get the following error:

$ LC_CTYPE="en_US.UTF-8" ./encoding-bug
"\x{fffd}" does not map to iso-8859-1 at ./encoding-bug line 14.

Without the "$string =~ s/xml//s;", there are no errors. The problem
is reproducible on both Mac OS X and Linux.

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=medium
---
Site configuration information for perl v5.8.8:

Configured by vinc17 at Fri Feb 24 03:27:08 CET 2006.

Summary of my perl5 (revision 5 version 8 subversion 8) configuration:
  Platform:
    osname=darwin, osvers=8.5.0, archname=darwin-2level
    uname='darwin prunille.vinc17.org 8.5.0 darwin kernel version 8.5.0: sun jan 22 10:38:46 pst 2006; root:xnu-792.6.61.obj~1release_ppc power macintosh powerpc '
    config_args='-des -Dprefix=/opt/local -Dccflags=-I'/opt/local/include' -Dldflags=-L/opt/local/lib -Dvendorprefix=/opt/local -Dcc=/usr/bin/gcc-4.0'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='/usr/bin/gcc-4.0', ccflags ='-I/opt/local/include -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/opt/local/include',
    optimize='-O3',
    cppflags='-no-cpp-precomp -I/opt/local/include -fno-common -DPERL_DARWIN -no-cpp-precomp -fno-strict-aliasing -pipe -Wdeclaration-after-statement -I/opt/local/include'
    ccversion='', gccversion='4.0.1 (Apple Computer, Inc. build 5250)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags ='-L/opt/local/lib -L/usr/local/lib'
    libpth=/usr/local/lib /opt/local/lib /usr/lib
    libs=-ldbm -ldl -lm -lc
    perllibs=-ldl -lm -lc
    libc=/usr/lib/libc.dylib, so=dylib, useshrplib=false, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-L/opt/local/lib -bundle -undefined dynamic_lookup -L/usr/local/lib'

Locally applied patches:
    

---
@INC for perl v5.8.8:
    /Users/vinc17/lib/site_perl
    /opt/local/lib/perl5/5.8.8/darwin-2level
    /opt/local/lib/perl5/5.8.8
    /opt/local/lib/perl5/site_perl/5.8.8/darwin-2level
    /opt/local/lib/perl5/site_perl/5.8.8
    /opt/local/lib/perl5/site_perl/5.8.7/darwin-2level
    /opt/local/lib/perl5/site_perl/5.8.7
    /opt/local/lib/perl5/site_perl
    /opt/local/lib/perl5/vendor_perl/5.8.8/darwin-2level
    /opt/local/lib/perl5/vendor_perl/5.8.8
    /opt/local/lib/perl5/vendor_perl/5.8.7/darwin-2level
    /opt/local/lib/perl5/vendor_perl/5.8.7
    /opt/local/lib/perl5/vendor_perl
    .

---
Environment for perl v5.8.8:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/vinc17
    LANG=POSIX
    LANGUAGE (unset)
    LC_CTYPE=en_US.ISO8859-1
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/Users/vinc17/bin:/usr/local/bin:/opt/local/bin:/usr/bin:/bin:/opt/local/sbin:/usr/sbin:/sbin:/usr/X11R6/bin:.
    PERL5LIB=/Users/vinc17/lib/site_perl
    PERL_BADLANG (unset)
    SHELL=/opt/local/bin/zsh


Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About