[perl #107008] UTF8 patches for 5.16

# New Ticket Created by  Father Chrysostomos 
# Please include the string:  [perl #107008]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=107008 >


Brian Fraser’s work on UTF8 support was never fully integrated into blead.  I’m creating this ticket, so I can make it a blocker for 5.16.

I really do think it should be a blocker, because without the last few patches it becomes painful to list the exceptions in perldelta.

The patches can be found here: https://github.com/Hugmeir/gsoc-pad-utf8-safety/commits/tokemess

Everything before ‘Label cleanup’ has been applied already.

The label patch has been modified to account for the unicode_eval feature and is on the sprout/tokemess branch at perl5.git.perl.org.  It is producing one failure that neither Brian Fraser nor I can fully understand.  We are awaiting feedback from Zefram, since it is in code that he wrote.  (Why is parse_label trying to take ownership of the string passed to it?  Why is it assuming it occurs at the start of a mallocked block?)

---
Flags:
    category=core
    severity=low
---
Site configuration information for perl 5.15.5:

Configured by sprout at Sun Dec 18 11:26:14 PST 2011.

Summary of my perl5 (revision 5 version 15 subversion 5) configuration:
  Snapshot of: 5dca8ed9d28127c9f7a2e7ce5f8ba970da3608cd
  Platform:
    osname=darwin, osvers=10.5.0, archname=darwin-2level
    uname='darwin pint.local 10.5.0 darwin kernel version 10.5.0: fri nov 5 23:20:39 pdt 2010; root:xnu-1504.9.17~1release_i386 i386 '
    config_args='-de -Dusedevel -DDEBUGGING=-g'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=undef, usemultiplicity=undef
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
    optimize='-O3 -g',
    cppflags='-fno-common -DPERL_DARWIN -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.2.1 (Apple Inc. build 5664)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -fstack-protector -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib
    libs=-ldbm -ldl -lm -lutil -lc
    perllibs=-ldl -lm -lutil -lc
    libc=, so=dylib, useshrplib=false, libperl=libperl.a
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector'

Locally applied patches:
    

---
@INC for perl 5.15.5:
    /usr/local/lib/perl5/site_perl/5.15.5/darwin-2level
    /usr/local/lib/perl5/site_perl/5.15.5
    /usr/local/lib/perl5/5.15.5/darwin-2level
    /usr/local/lib/perl5/5.15.5
    /usr/local/lib/perl5/site_perl
    .

---
Environment for perl 5.15.5:
    DYLD_LIBRARY_PATH (unset)
    HOME=/Users/sprout
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin:/usr/local/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

• [perl #107008] UTF8 patches for 5.16 by Father Chrysostomos