[perl #24526] Combination of rindex() and index() is non-deterministic with LANG=en_GB.UTF-8

# New Ticket Created by  martin@xanalys.com 
# Please include the string:  [perl #24526]
# in the subject line of all future correspondence about this issue. 
# <URL: http://rt.perl.org/rt2/Ticket/Display.html?id=24526 >



This is a bug report for perl from martin@xanalys.com,
generated with the help of perlbug 1.34 running under perl v5.8.0.


-----------------------------------------------------------------
[Please enter your report here]

Calling index() after calling rindex() when LANG is en_GB.UTF-8 can give
non-deterministic results.  In particular, index() will return the wrong value
unless a dummy call to index() with position 0 is made first.

The attached script and input file show the problem.  Here are two runs
showing different output on a stock Red Hat 9 installation.  Note the values
114 and 115, which should both be 112.  Apart from the machinery around the
calls to rindex() and index(), the calls themselves are the same in the
initial calls and the while loop.  Removing the # from the dummy call to
index($_, "\n", 0) makes the problem do away.

% cat input.txt | env LANG=en_GB.UTF-8 perl script.pl
22
69
72
114
-- 37 4
69
-- 101 4
115
% cat input.txt | env LANG=C perl script.pl
22
69
72
112
-- 37 4
69
-- 101 4
112
% 

begin 666 bug.tar.gz
M'XL(`%*3NS\``^W42T_C,!``X)S]*V9+!+%(:SMMDD,!L<M[V??RVA42E&)H
M)!-,'E`)\=]Q$JBVV\"I](#F.S2N/3,=JYVF_23264LKZ^UPP7D0!!;G7`C/
M+YY<=#KELQ`$;8N'H1^$0G1"S\1[P@\LX&_8TTB>9KT$P+KJ)5D4OQ8GDW06
M#<W6W`>6IPE3U_V>8F=1S+1,%"%Y?"XOP&9=L$]@&996S(*91>,X;G0)T4D4
M9V!>SN70L4_<<M^%=DA;5405\/]Y1XR?3Q00?!0Q-Y'-Z4MU!?>?T\CJT+3I
MF%!YVU.PD+*;.YDPG:<#9W7H*AE?9@/'/J6CY3RE77N>R<L%DWTWB)0$Q[&U
M"[:)H*98JE74ET4V<!<\2N&>`%2=-)I-L'496GX\F'>Q,DD35S-131#TI8L]
M%ZQ)6ZPZ>;K@`YGJ]Q_%.L]:V3";:M5QY?Q7\UX[_YXG1O/O\W8Y_^9O`.=_
M!CY^6EO?V-S:WOF\^^7KM^\_?O[ZO;=_0,:WBQ$JM@^/_ORM2S#AI+Y0E5'D
I/YW\&UJ7,]U?-T(((8000@@AA!!"""&$$$(((830^_<(*P,YH@`H````
`
end


[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=high
---
Site configuration information for perl v5.8.0:

Configured by bhcompile'
cf_email='bhcompile at Tue Feb 18 22:17:47 EST 2003.

Summary of my rderl (revision 5.0 version 8 subversion 0) configuration:
  Platform:
    osname=linux, osvers=2.4.20-2.48smp, archname=i386-linux-thread-multi
    uname='linux str'
    config_args='-des -Doptimize=-O2 -march=i386 -mcpu=i686 -g -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dinstallprefix=/usr -Dprefix=/usr -Darchname=i386-linux -Dvendorprefix=/usr -Dsiteprefix=/usr -Dotherlibdirs=/usr/lib/perl5/5.8.0 -Duseshrplib -Dusethreads -Duseithreads -Duselargefiles -Dd_dosuid -Dd_semctl_semun -Di_db -Ui_ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef'
 useithreads=define usemultiplicity=
    useperlio= d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=un uselongdouble=
    usemymalloc=, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DDEBUGGING -fno-strict-aliasing -I/usr/local/include -I/usr/include/gdbm'
    ccversion='', gccversion='3.2.2 20030213 (Red Hat Linux 8.0 3.2.2-1)', gccosandvers=''
gccversion='3.2.2 200302'
    intsize=e, longsize= , ptrsize=p, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
    ivtype='long'
k', ivsize=4'
ivtype='long'
known_ext, nvtype='double'
o_nonbl', nvsize=, Off_t='', lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc'
l', ldflags =' -L/usr/local/lib'
ldf'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt -lutil
    perllibs=
    libc=/lib/libc-2.3.1.so, so=so, useshrplib=true, libperl=libper
    gnulibc_version='2.3.1'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so', d_dlsymun=undef, ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5/5.8.0/i386-linux-thread-multi/CORE'
    cccdlflags='-fPIC'
ccdlflags='-rdynamic -Wl,-rpath,/usr/lib/perl5', lddlflags='s Unicode/Normalize XS/A'

Locally applied patches:
    MAINT18379

---
@INC for perl v5.8.0:
    /u/roy/perl5/lib/site_perl/5.00555/i386-linux
    /usr/lib/perl5/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/5.8.0
    /usr/lib/perl5/site_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/site_perl/5.8.0
    /usr/lib/perl5/site_perl
    /usr/lib/perl5/vendor_perl/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.8.0
    /usr/lib/perl5/vendor_perl
    /usr/lib/perl5/5.8.0/i386-linux-thread-multi
    /usr/lib/perl5/5.8.0
    .

---
Environment for perl v5.8.0:
    HOME=/u/martin
    LANG=en_GB.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=.:/u/martin/bin:/u/martin/nfs-files/bin/Linux:/u/martin/nfs-files/bin:/usr/local/bin:/usr/ucb:/bin:/usr/bin:/sbin:/usr/sbin:/usr/bin/X11:/usr/X11R6/bin:/usr/local/X11R5/bin:/usr/openwin/bin:/usr/dt/bin:/etc:/usr/lib:/usr/etc:/usr/bsd:/usr/hlh:/usr/new:/usr/contrib/bin:/usr/ccs/bin:/sp/lispsrc/clc/pcl/scripts:/sp/lispsrc/misc/admin/utils
    PERLLIB=/u/roy/perl5/lib/site_perl/5.00555/i386-linux
    PERL_BADLANG (unset)
    SHELL=/bin/csh
    dlflags='-share (unset)

• [perl #24526] Combination of rindex() and index() is non-deterministic with LANG=en_GB.UTF-8 by perlbug-followup
• Re: [perl #24526] Combination of rindex() and index() is non-deterministic with LANG=en_GB.UTF-8 by Dave Mitchell