develooper Front page | perl.perl5.porters | Postings from August 2013

[perl #119475] :mmap flag in open cause hang in eof function on files >=2G

Thread Next
From:
Linda Walsh
Date:
August 26, 2013 23:33
Subject:
[perl #119475] :mmap flag in open cause hang in eof function on files >=2G
Message ID:
rt-3.6.HEAD-1873-1377559967-924.119475-75-0@perl.org
# New Ticket Created by  Linda Walsh 
# Please include the string:  [perl #119475]
# in the subject line of all future correspondence about this issue. 
# <URL: https://rt.perl.org:443/rt3/Ticket/Display.html?id=119475 >



This is a bug report for perl from perl-diddler@tlinx.org,
generated with the help of perlbug 1.39 running under perl 5.16.2.


-----------------------------------------------------------------
[Please describe your issue here]

perl -we'use strict;
open (my $h, "<:raw:mmap", "2G") or "die";
use P;
P "eof?";
if (eof $h) {
P "yes"} else {P "no"}
'
-----
"2G" is a 2G file (dd if={source} of=2G bs=1G count=2)

it works at 2GB-1, but at 2GB and greater, it hits the
eof function and never comes back.

Without the mmap, it works (at a 50% or greater speed penalty).

(FWIW, the machine I am running on has:
> free -h
             total       used       free     shared    buffers     cached
Mem:           94G        92G       2.0G         0B       456K        76G
-/+ buffers/cache:        15G        78G
Swap:         8.0G        95M       7.9G
----
alot of "free" memory.  It may be OS or driver related, I don't
know, but since it works w/o mmap, that is a fallback position --
though certainly not a desirable one.

I map pairs of files & compare them.  For larger files, I try a 
sparse compare first -- like only comparing maybe 4M out of every 64.

If all the segments compare equal, then I have to do a full file
compare, but in doing the I/O, there is alot of seeking as well
as many files being mapped, then released.  If they are read
into buffers first, there is a large performance penalty.




[Please do not change anything below this line]
-----------------------------------------------------------------
---
Flags:
    category=core
    severity=high
---
This perlbug was built using Perl 5.16.2 - Fri Feb 15 01:17:37 UTC 2013
It is being executed now by  Perl 5.16.2 - Fri Feb 15 01:12:05 UTC 2013.

Site configuration information for perl 5.16.2:

Configured by abuild at Fri Feb 15 01:12:05 UTC 2013.

Summary of my perl5 (revision 5 version 16 subversion 2) configuration:
   
  Platform:
    osname=linux, osvers=3.4.6-2.10-default, archname=x86_64-linux-thread-multi
    uname='linux build34 3.4.6-2.10-default #1 smp thu jul 26 09:36:26 utc 2012 (641c197) x86_64 x86_64 x86_64 gnulinux '
    config_args='-ds -e -Dprefix=/usr -Dvendorprefix=/usr -Dinstallusrbinperl -Dusethreads -Di_db -Di_dbm -Di_ndbm -Di_gdbm -Dd_dbm_open -Duseshrplib=true -Doptimize=-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe -Accflags=-DPERL_USE_SAFE_PUTENV -Dotherlibdirs=/usr/lib/perl5/site_perl'
    hint=recommended, useposix=true, d_sigaction=define
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=define, use64bitall=define, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
    optimize='-fmessage-length=0 -O2 -Wall -D_FORTIFY_SOURCE=2 -fstack-protector -funwind-tables -fasynchronous-unwind-tables -g -Wall -pipe',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -fstack-protector'
    ccversion='', gccversion='4.7.2 20130108 [gcc-4_7-branch revision 195012]', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib64 -fstack-protector'
    libpth=/lib64 /usr/lib64 /usr/local/lib64
    libs=-lm -ldl -lcrypt -lpthread
    perllibs=-lm -ldl -lcrypt -lpthread
    libc=/lib64/libc-2.17.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version='2.17'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E -Wl,-rpath,/usr/lib/perl5/5.16.2/x86_64-linux-thread-multi/CORE'
    cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib64 -fstack-protector'

Locally applied patches:
    

---
@INC for perl 5.16.2:
    /home/law/bin/lib
    /usr/lib/perl5/site_perl/5.16.2/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.16.2
    /usr/lib/perl5/vendor_perl/5.16.2/x86_64-linux-thread-multi
    /usr/lib/perl5/vendor_perl/5.16.2
    /usr/lib/perl5/5.16.2/x86_64-linux-thread-multi
    /usr/lib/perl5/5.16.2
    /usr/lib/perl5/site_perl/5.16.2/x86_64-linux-thread-multi
    /usr/lib/perl5/site_perl/5.16.2
    /usr/lib/perl5/site_perl
    .

---
Environment for perl 5.16.2:
    HOME=/home/law
    LANG=en_US.UTF-8
    LANGUAGE (unset)
    LC_COLLATE=C
    LC_CTYPE=en_US.UTF-8
    LD_LIBRARY_PATH=/usr/lib64/mpi/gcc/openmpi/lib64
    LOGDIR (unset)
    PATH=/home/law/bin/lib:/sbin:/usr/local/sbin:/usr/lib64/mpi/gcc/openmpi/bin:/home/law/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/opt/kde3/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:.:/usr/lib/qt3/bin:/opt/dell/srvadmin/bin:/usr/sbin:/etc/local/func_lib:/home/law/lib
    PERL5OPT=-Mutf8 -CSA -I/home/law/bin/lib
    PERL_BADLANG (unset)
    SHELL=/bin/bash


Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About