develooper Front page | perl.perl5.porters | Postings from July 2018

[perl #133314] in-place editing in 5.28 quickly exhausts open filedescriptor limit

Thread Previous | Thread Next
From:
perlbug-followup
Date:
July 1, 2018 05:57
Subject:
[perl #133314] in-place editing in 5.28 quickly exhausts open filedescriptor limit
Message ID:
rt-4.0.24-5426-1530410923-543.133314-75-0@perl.org
# New Ticket Created by   
# Please include the string:  [perl #133314]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org/Ticket/Display.html?id=133314 >



This is a bug report for perl from mtmiller@debian.org,
generated with the help of perlbug 1.41 running under perl 5.28.0.


-----------------------------------------------------------------
The in-place editing feature in perl 5.28 dies with an error if called
with 1020 or more files on the command line. This is a regression from
5.26, which was able to operate on any number of files on its command
line.

By way of example:

    $ for n in $(seq 1 1019); do mktemp XXXXXX > /dev/null; done
    $ ls | wc -l
    1019
    $ perl -pi -e s/a/b/ *
    $ rm *
    $ for n in $(seq 1 1020); do mktemp XXXXXX > /dev/null; done
    $ ls | wc -l
    1020
    $ perl -pi -e s/a/b/ *
    Cannot complete in-place edit of ZwuzEb: failed to rename work file
    'XXVtDC4r' to 'ZwuzEb': Bad file descriptor.

With the following modified program, one can look into the /proc
filesystem and see that by the end of the program, the perl process is
holding 1019 open file descriptors to the working directory.

    $ perl -pi -e 's/a/b/; BEGIN {print "$$\n"} END {sleep 60}' *

It looks like perl is doing an opendir on the parent directory of each
file edited with the -i option, but not closing that descriptor before
moving on to the next file in the argument list.

I have verified that this is related to the "safer in-place editing"
feature from #127663. At commit 3ff4feb5bd22 perl is able to operate on
a list of 100,000 files, and at commit 9c6681cc159f it dies on a list of
1020 files.
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=medium
---
Site configuration information for perl 5.28.0:

Configured by mike at Sat Jun 30 16:56:57 PDT 2018.

Summary of my perl5 (revision 5 version 28 subversion 0) configuration:
  Commit id: 6cb72a3d7fbefb477b95588e2fd81e33073da8b3
  Platform:
    osname=linux
    osvers=4.16.0-2-amd64
    archname=x86_64-linux
    uname='linux galago.mtmxr.com 4.16.0-2-amd64 #1 smp debian 4.16.12-1 (2018-05-27) x86_64 gnulinux '
    config_args='-des -Dprefix=/tmp/perl'
    hint=recommended
    useposix=true
    d_sigaction=define
    useithreads=undef
    usemultiplicity=undef
    use64bitint=define
    use64bitall=define
    uselongdouble=undef
    usemymalloc=n
    default_inc_excludes_dot=define
    bincompat5005=undef
  Compiler:
    cc='cc'
    ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64'
    optimize='-O2'
    cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
    ccversion=''
    gccversion='8.1.0'
    gccosandvers=''
    intsize=4
    longsize=8
    ptrsize=8
    doublesize=8
    byteorder=12345678
    doublekind=3
    d_longlong=define
    longlongsize=8
    d_longdbl=define
    longdblsize=16
    longdblkind=3
    ivtype='long'
    ivsize=8
    nvtype='double'
    nvsize=8
    Off_t='off_t'
    lseeksize=8
    alignbytes=8
    prototype=define
  Linker and Libraries:
    ld='cc'
    ldflags =' -fstack-protector-strong -L/usr/local/lib'
    libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
    libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc
    perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
    libc=libc-2.27.so
    so=so
    useshrplib=false
    libperl=libperl.a
    gnulibc_version='2.27'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs
    dlext=so
    d_dlsymun=undef
    ccdlflags='-Wl,-E'
    cccdlflags='-fPIC'
    lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'


---
@INC for perl 5.28.0:
    /tmp/perl/lib/site_perl/5.28.0/x86_64-linux
    /tmp/perl/lib/site_perl/5.28.0
    /tmp/perl/lib/5.28.0/x86_64-linux
    /tmp/perl/lib/5.28.0

---
Environment for perl 5.28.0:
    HOME=/home/mike
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LD_LIBRARY_PATH (unset)
    LOGDIR (unset)
    PATH=/bin:/usr/bin
    PERL_BADLANG (unset)
    SHELL=/bin/bash


Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About