develooper Front page | perl.perl5.porters | Postings from July 2005

[perl #36659] UTF-16 endianess switch after line break

From:
Jeremias Reith
Date:
July 26, 2005 04:30
Subject:
[perl #36659] UTF-16 endianess switch after line break
Message ID:
rt-3.0.11-36659-118306.16.8532791110032@perl.org
# New Ticket Created by  Jeremias Reith 
# Please include the string:  [perl #36659]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/rt3/Ticket/Display.html?id=36659 >


This is a bug report for perl from jr@terragate.net,
generated with the help of perlbug 1.35 running under perl v5.8.7.


-----------------------------------------------------------------
There is an endianess switch after each newline while outputting UTF-16 on
Win32.

Example script:

#!/usr/bin/perl

binmode(STDOUT, ':encoding(UTF-16)');
map { print $_ . "\n" } @ARGV;


This produces the following (called with "foo bar baz" as command line params):

0000000: feff 0066 006f 006f 000d 0a00 6200 6100  ...f.o.o....b.a.
0000010: 7200 0d0a 0062 0061 007a 000d 0a         r....b.a.z...

The carriage return (\r) is correctly outputted as 0xd but after that the
newline is printed in little endian (0xa00 instead of 0xa). Furthermore all
following chars are printed in LE until the next line break.

In my point of view this looks like a bug in the code that transparently adds
a \r for each newline on the windows platform.

So each line break causes a switch from BE to LE and vice versa.

Output of the same script on Mac OS X (Perl 5.8.6):

0000000: feff 0066 006f 006f 000a 0062 0061 0072  ...f.o.o...b.a.r
0000010: 000a 0062 0061 007a 000a                 ...b.a.z..

No problem here.

[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=medium
---
Site configuration information for perl v5.8.7:

Configured by builder at Mon Jun  6 13:36:05 2005.

Summary of my perl5 (revision 5 version 8 subversion 7) configuration:
  Platform:
    osname=MSWin32, osvers=5.0, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    usethreads=define use5005threads=undef useithreads=define
usemultiplicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cl', ccflags ='-nologo -Gf -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 -D_CONSOLE
-DNO_STRICT -DHAVE_DES_FCRYPT -DBUILT_BY_ACTIVESTATE -DNO_HASH_SEED
-DUSE_SITECUSTOMIZE -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO
-DPERL_MSVCRT_READFIX',
    optimize='-MD -Zi -DNDEBUG -O1',
    cppflags='-DWIN32'
    ccversion='12.00.8804', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64',
lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf 
-libpath:"C:\Perl\lib\CORE"  -machine:x86'
    libpth=\lib
    libs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib 
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib
uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib
msvcrt.lib
    perllibs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib 
comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib
uuid.lib ws2_32.lib mpr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib
msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=yes, libperl=perl58.lib
    gnulibc_version='undef'
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf 
-libpath:"C:\Perl\lib\CORE"  -machine:x86'

Locally applied patches:
    ACTIVEPERL_LOCAL_PATCHES_ENTRY
    #  if !defined(PERL_DARWIN)
    Iin_load_module moved for compatibility with build 806
    #  endif
    #  if defined(__hpux)
    Avoid signal flag SA_RESTART for older versions of HP-UX
    #  endif
    PerlEx hacks for CGI::Carp
    Less verbose ExtUtils::Install and Pod::Find
    instmodsh upgraded from ExtUtils-MakeMaker-6.25
    24699 ICMP_UNREACHABLE handling in Net::Ping
    21540 Fix backward-compatibility issues in if.pm

---
@INC for perl v5.8.7:
    C:/Perl/lib
    C:/Perl/site/lib
    .

---
Environment for perl v5.8.7:
    HOME (unset)
    LANG (unset)
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=C:\Perl\bin\;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem
    PERL_BADLANG (unset)
    SHELL (unset)




nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About