[perl #122950] Off-by-one when truncating *shared* array by: $#ary -= $chop_this_many

# New Ticket Created by   
# Please include the string:  [perl #122950]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=122950 >



This is a bug report for perl from andy@hmc.edu,
generated with the help of perlbug 1.39 running under perl 5.18.2.

Off-by-one when truncating *shared* array by: $#ary -= $chop_this_many

What should happen: Shared array max index should drop to a new, lower value.

What does happen: Actual new max index is *too large by one*  (Too few elements chopped off)
See the 4th line in the sample output; "...index is:  5"; It should say "...index is:  4".

See additional notes and environment details after example code and example output below.



$ ./perl-shared-array-truncate.pl
Before truncation, ordinary array highest index is:  9
After removing five, ordinary array highest index is:  4

Before truncation, shared array highest index is:  9
After removing five, shared array highest index is:  5

Shared array now contains: 0 | 1 | 2 | 3 | 4 | 5



$ cat perl-shared-array-truncate.pl
#!/usr/bin/perl -w
use strict;

use threads;
use threads::shared;

### First with an ordinary array
my @ordinary_array;
@ordinary_array = (0,1,2,3,4,5,6,7,8,9);

printf "Before truncation, ordinary array highest index is:  %s\n", $#ordinary_array;
$#ordinary_array = $#ordinary_array - 5;
printf "After removing five, ordinary array highest index is:  %s\n\n", $#ordinary_array;


### Then try it with a shared array
my @shared_array :shared;
lock @shared_array; # Same with or without this line
@shared_array = (0,1,2,3,4,5,6,7,8,9);

printf "Before truncation, shared array highest index is:  %s\n", $#shared_array;
$#shared_array = $#shared_array - 5;
printf "After removing five, shared array highest index is:  %s\n\n", $#shared_array;
printf "Shared array now contains: %s\n", join(' | ', @shared_array);


----- environment and notes -----

$ uname -a
Linux andy.xxxxxxx 3.15.10-201.fc20.x86_64 #1 SMP Wed Aug 27 21:10:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ /usr/bin/perl -v
This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux-thread-multi
(with 19 registered patches, see perl -V for more detail)


Notes:

Although I am using the "threads" module in this example script, I am not spawning
any threads here (although I do in the real code in which this problem surfaced).
This is not a concurrency problem. I guess that was obvious. :-)

I am aware that for removing a small number of elements I can "pop" a few times however
I am using this to implement an LRU queue of keys to track the order of elements added
to a hash. I want to occasionally remove ~1000 elements from the end of a ~10000 element
array and I fear that pop on that scale will be slow and inefficient.
Yes, I can compensate for the off-by-one in the meantime.

I am aware that splice is not allowed with shared arrays. This truncation does not appear
to violate (my reading of) the rules for shared arrays, and one of the test scripts at:

   cpansearch.perl.org/src/RJBS/perl-5.17.6/dist/threads-shared/t/av_simple.t

uses this method to truncate an array.

I am aware of Thread::Queue. 

I have also repeated this on these environments with same results:

Linux xxxxxxxx 2.6.32-358.2.1.el6.centos.plus.x86_64 #1 SMP Wed Mar 13 02:09:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
This is perl, v5.10.1 (*) built for x86_64-linux-thread-multi

Linux andy2 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
This is perl 5, version 14, subversion 2 (v5.14.2) built for x86_64-linux-gnu-thread-multi

Thank you very much.

Andy


---
Flags:
    category=library
    severity=medium
    module=threads::shared
---
Site configuration information for perl 5.18.2:

Configured by Red Hat, Inc. at Tue Jan  7 14:45:19 UTC 2014.

Summary of my perl5 (revision 5 version 18 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=3.11.9-200.fc19.x86_64, archname=x86_64-linux-thread-multi
    uname='linux buildvm-12.phx2.fedoraproject.org 3.11.9-200.fc19.x86_64 #1 smp wed nov 20 21:22:24 utc 2013 x86_64 x86_64 x86_64 gnulinux '
    config_args='-des -Doptimize=-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Dccdlflags=-Wl,--enable-new-dtags -Dlddlflags=-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches  -m64 -mtune=generic -Wl,-z,relro  -Dshrpdir=/usr/lib64 -DDEBUGGING=-g -Dversion=5.18.2 -Dmyhostname=localhost -Dperladmin=root@localhost -Dcc=gcc -Dcf_by=Red Hat, Inc. -Dprefix=/usr -Dvendorprefix=/usr -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl5 -Dsitearch=/usr/local/lib64/perl5 -Dprivlib=/usr/share/perl5 -Dvendorlib=/usr/share/perl5/vendor_perl -Darchlib=/usr/lib64/perl5 -Dvendorarch=/usr/lib64/perl5/vendor_perl -Darchname=x86_64-linux-thread-multi -Dlibpth=/usr/local/lib64 /lib64 /usr/lib64 -Duseshrplib -Dusethreads -Duseithreads -Dusedtrace=/usr/bin/dtrace -Duselargefiles -Dd_semctl_semun -Di_db -Ui
 _ndbm -Di_gdbm -Di_shadow -Di_syslog -Dman3ext=3pm -Duseperlio -Dinstallusrbinperl=n -Ubincompat5005 -Uversiononly -Dpager=/usr/bin/less -isr -Dd_gethostent_r_proto -Ud_endhostent_r_proto -Ud_sethostent_r_proto -Ud_endprotoent_r_proto -Ud_setprotoent_r_proto -Ud_endservent_r_proto -Ud_setservent_r_proto -Dscriptdir=/usr/bin -Dusesitecustomize'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
    ccversion='', gccversion='4.8.2 20131212 (Red Hat 4.8.2-7)', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -fstack-protector'
    libpth=/usr/local/lib64 /lib64 /usr/lib64
    libs=-lresolv -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lpthread -lc -lgdbm_compat
    perllibs=-lresolv -lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.18'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,--enable-new-dtags'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -Wl,-z,relro '

Locally applied patches:
    Fedora Patch1: Removes date check, Fedora/RHEL specific
    Fedora Patch3: support for libdir64
    Fedora Patch4: use libresolv instead of libbind
    Fedora Patch5: USE_MM_LD_RUN_PATH
    Fedora Patch6: Skip hostname tests, due to builders not being network capable
    Fedora Patch7: Dont run one io test due to random builder failures
    Fedora Patch9: Fix find2perl to translate ? glob properly (RT#113054)
    Fedora Patch10: Update h2ph(1) documentation (RT#117647)
    Fedora Patch11: Update pod2html(1) documentation (RT#117623)
    Fedora Patch12: Disable ornaments on perl5db AutoTrace tests (RT#118817)
    Fedora Patch14: Do not use system Term::ReadLine::Gnu in tests (RT#118821)
    Fedora Patch15: Define SONAME for libperl.so
    Fedora Patch16: Install libperl.so to -Dshrpdir value
    Fedora Patch18: Fix crash with \\&$glob_copy (RT#119051)
    Fedora Patch19: Fix coreamp.t rand test (RT#118237)
    Fedora Patch20: Reap child in case where exception has been thrown (RT#114722)
    Fedora Patch21: Fix using regular expressions containing multiple code blocks (RT#117917)
    Fedora Patch200: Link XS modules to libperl.so with EU::CBuilder on Linux
    Fedora Patch201: Link XS modules to libperl.so with EU::MM on Linux

---
@INC for perl 5.18.2:
    /usr/local/lib64/perl5
    /usr/local/share/perl5
    /usr/lib64/perl5/vendor_perl
    /usr/share/perl5/vendor_perl
    /usr/lib64/perl5
    /usr/share/perl5
    .

---
Environment for perl 5.18.2:
    HOME=/home/andy
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/home/andy/.local/bin:/home/andy/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash

• [perl #122950] Off-by-one when truncating *shared* array by: $#ary -= $chop_this_many by perlbug-followup
• Re: [perl #122950] Off-by-one when truncating *shared* array by:$#ary -= $chop_this_many by Dave Mitchell