develooper Front page | perl.perl5.porters | Postings from May 2004

RE: [perl #29873] seg fault w/v5.8.4 on IBM RS/6k when populating large hash

Thread Previous | Thread Next
From:
al.danial
Date:
May 28, 2004 01:28
Subject:
RE: [perl #29873] seg fault w/v5.8.4 on IBM RS/6k when populating large hash
Message ID:
48C9AB115A3DEB44AA5CA4B56BB6E4B502EEB66F@MBSP02.services.sp.trw.com
=> -----Original Message-----
=> From: H.Merijn Brand [mailto:h.m.brand@hccnet.nl]
=> Sent: Thursday, May 27, 2004 2:19 AM
=> To: Danial, Al
=> Cc: Perl 5 Porters
=> Subject: Re: [perl #29873] seg fault w/v5.8.4 on IBM RS/6k when
=> populating large hash 
=> 
=> 
=> On Wed 26 May 2004 18:37, <al.danial@ngc.com> wrote:
=> > Merijn, thanks for looking into the problem so quickly.
=> > I neglected to mention that the other machines on which
=> > the program runs all have much less memory than the 8 GB
=> > on the IBM.  The Linux box has (only) 1 GB RAM and has 
=> > no problems with the code.
=> 
=> Linux is able to give you more memory than AIX. Look at your limits.


Memory limits on our AIX boxes, both hard and soft, are "unlimited".  

% limit -h
cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       unlimited
coredumpsize    unlimited
memoryuse       unlimited
% limit
cputime         unlimited
filesize        unlimited
datasize        unlimited
stacksize       unlimited
coredumpsize    1024 kbytes
memoryuse       unlimited


/etc/security/limits has this (and all users get the default):

default:
        fsize = -1
        core = 2048
        cpu = -1
        data = -1
        rss = -1
        stack = -1
        nofiles = 2000

If we had memory limits in place our work with large finite element 
models would me be disrupted.  Other programs such as matlab and Patran
have no difficulty consuming huge amounts of memory.

A difference between our setups is your perl was built with an IBM compiler 
while mine was built with GCC.  I rebuilt v5.8.4 with IBM's xlc compiler 
but the seg fault happens with that perl too.

I agree the evidence suggests an environment issue, and not a fault
with Perl.  Thanks for your assistance in any event.         -- Al

 
=> On my Linux box I've all my limits set to unlimited, but on AIX
=> 
=> i2:/u/usr/merijn 101 > limit
=> cputime      unlimited
=> filesize     unlimited
=> datasize     131072 kbytes
=> stacksize    32768 kbytes
=> coredumpsize 1024 kbytes
=> memoryuse    32768 kbytes
=> descriptors  2000
=> i2:/u/usr/merijn 102 >
=> 
=> And then you also have to check the system imposed (hard) limits
=> 
=> i2:/u/usr/merijn 102 > limit -h
=> cputime      unlimited
=> filesize     unlimited
=> datasize     unlimited
=> stacksize    unlimited
=> coredumpsize unlimited
=> memoryuse    unlimited
=> descriptors  unlimited
=> i2:/u/usr/merijn 103 >
=> 
=> You could try to increase the personal limits, but you're
=> not always allowed to:
=> 
=> i2:/u/usr/merijn 103 > limit memoryuse unlimited
=> i2:/u/usr/merijn 104 > limit
=> cputime      unlimited
=> filesize     unlimited
=> datasize     131072 kbytes
=> stacksize    32768 kbytes
=> coredumpsize 1024 kbytes
=> memoryuse    unlimited
=> descriptors  2000
=> i2:/u/usr/merijn 105 >
=> 
=> Playing around and setting more to unlimited, makes your
=> script most likely pass:
=> 
=> i2:/tmp 113 > limit
=> cputime      unlimited
=> filesize     unlimited
=> datasize     unlimited
=> stacksize    unlimited
=> coredumpsize 1024 kbytes
=> memoryuse    unlimited
=> descriptors  2000
=> i2:/tmp 114 > perl test.pl
=> i2:/tmp 115 >
=> 
=> This is the same script that previously gave me out of memory
=> 
=> As a sysamin, you could check out /etc/security/limits
=> 
=> As this is no bug in Perl, I will close it
=> 
=> > Here is my rough estimate of the memory that would be
=> > taken by the scalar data within the hash of hash of lists:
=> > 
=> >     3 *        # number of sets
=> >     300000 *   # number of ID's
=> >     7 *        # number of rand's
=> >    (9*4        # number of int's per scalar variable (??)
=> >                # times number of bytes per int plus
=> >     + 8)       # number of bytes for the rand double 
=> > 
=> >   = 277 MB
=> > 
=> > I have no idea how much additional space is needed by the hash 
=> > tables but even if it is double the amount above, the code 
=> > should be able to (and does on Linux) run comfortably within 
=> > 1 GB.          -- Al
=> > 
=> >  -+- -+- -+- -+- -+- -+- -+- -+- -+- -+- -+- -+- -+- -+-
=> > Albert Danial  al.danial@ngc.com  310-813-9516 fax: -4467
=> > NGST R10/1328 One Space Park, Redondo Bch., CA 90278, USA 
=> > 
=> > 
=> > 
=> > => -----Original Message-----
=> > => From: H. Merijn Brand via RT [mailto:perlbug-followup@perl.org]
=> > => Sent: Wednesday, May 26, 2004 1:50 AM
=> > => To: Danial, Al
=> > => Subject: Re: [perl #29873] seg fault w/v5.8.4 on IBM RS/6k when
=> > => populating large hash 
=> > => 
=> > => 
=> > => On Wed 26 May 2004 02:41, "al.danial@ngc.com (via RT)" 
=> > => <perlbug-followup@perl.org> wrote:
=> > => > # New Ticket Created by  al.danial@ngc.com 
=> > => > # Please include the string:  [perl #29873]
=> > => > # in the subject line of all future correspondence about 
=> > => this issue. 
=> > => > # <URL: 
=> http://rt.perl.org:80/rt3/Ticket/Display.html?id=29873 >
=> > => > 
=> > => > 
=> > => > This is a bug report for perl from al.danial@ngc.com,
=> > => > generated with the help of perlbug 1.35 running under 
=> perl v5.8.4.
=> > => > 
=> > => > 
=> > => > 
=> -----------------------------------------------------------------
=> > => > This program causes a seg fault on an IBM RS/6000 
=> with AIX 4.3.3.
=> > => > 
=> > => >     #!/usr/local/perl-5.8.4/bin/perl -w
=> > => >     my $upper_limit;
=> > => >     $upper_limit = 275000;  # works
=> > => >     $upper_limit = 300000;  # fails
=> > => >     my %data = ();
=> > => >     foreach my $set (qw( A B C )) {
=> > => >         foreach my $id (1..$upper_limit) {
=> > => >             $data{$set}{$id} = [ rand, rand, rand, rand, 
=> > => rand, rand, rand, ];
=> > => >         }
=> > => >     }
=> > => 
=> > => That fully depends on your system configuration. On my 
=> > => system, the first limit
=> > => also causes problems on both recent and older perl's:
=> > => 
=> > => i2:/tmp 103 > perl test.pl
=> > => Out of memory!
=> > => Exit 1
=> > => i2:/tmp 104 >
=> > => 
=> > => i2:/tmp 104 > perl5.6.1 test.pl
=> > => Out of memory!
=> > => Exit 1
=> > => i2:/tmp 105 >
=> > => 
=> > => No seg faults though. Mahor difference between my perl's and 
=> > => yours is that
=> > => myine are built with vac, and your's are built with gcc.
=> > => 
=> > => > The failure happens with Perl version 5.6.1 on the 
=> IBM as well.
=> > => > 
=> > => > The workstation has 8 GB of memory so I don't think 
=> it is because
=> > => > the code is using too much space.  The above program 
=> works fine
=> > => > on Linux, Sun, and SGI workstations using Perl 5.8.0.
=> > => > 
=> > => > 
=> > => > [Please do not change anything below this line]
=> > => > 
=> -----------------------------------------------------------------
=> > => > ---
=> > => > Flags:
=> > => >     category=core
=> > => >     severity=high
=> > => > ---
=> > => > Site configuration information for perl v5.8.4:
=> > => > 
=> > => > Configured by adanial at Thu May 20 11:47:39 PDT 2004.
=> > => > 
=> > => > Summary of my perl5 (revision 5 version 8 subversion 4) 
=> > => configuration:
=> > => >   Platform:
=> > => >     osname=aix, osvers=4.3.3.0, archname=aix
=> > => >     uname='aix mohr 3 4 00041bcf4c00 '
=> > => >     config_args='-Dprefix=/usr/local/perl-5.8.4 -Dcc=gcc -de'
=> > => >     hint=recommended, useposix=true, d_sigaction=define
=> > => >     usethreads=undef use5005threads=undef 
=> > => useithreads=undef usemultiplicity=undef
=> > => >     useperlio=define d_sfio=undef uselargefiles=define 
=> > => usesocks=undef
=> > => >     use64bitint=undef use64bitall=undef uselongdouble=undef
=> > => >     usemymalloc=n, bincompat5005=undef
=> > => >   Compiler:
=> > => >     cc='gcc', ccflags ='-D_ALL_SOURCE -D_ANSI_C_SOURCE 
=> > => -D_POSIX_SOURCE -DUSE_NATIVE_DLOPEN -fno-strict-aliasing 
=> > => -I/usr/local/include -D_LARGE_FILES',
=> > => >     optimize='-O',
=> > => >     cppflags='-D_ALL_SOURCE -D_ANSI_C_SOURCE 
=> > => -D_POSIX_SOURCE -DUSE_NATIVE_DLOPEN -fno-strict-aliasing 
=> > => -I/usr/local/include'
=> > => >     ccversion='', gccversion='3.3.3', 
=> gccosandvers='aix4.3.3.0'
=> > => >     intsize=4, longsize=4, ptrsize=4, doublesize=8, 
=> byteorder=4321
=> > => >     d_longlong=define, longlongsize=8, d_longdbl=define, 
=> > => longdblsize=8
=> > => >     ivtype='long', ivsize=4, nvtype='double', nvsize=8, 
=> > => Off_t='off_t', lseeksize=8
=> > => >     alignbytes=8, prototype=define
=> > => >   Linker and Libraries:
=> > => >     ld='gcc', ldflags =' -Wl,-brtl -L/usr/local/lib -Wl,-b32'
=> > => >     libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib
=> > => >     libs=-lbind -lnsl -lgdbm -ldbm -ldb -ldl -lld -lm 
=> > => -lcrypt -lc -lbsd
=> > => >     perllibs=-lbind -lnsl -ldl -lld -lm -lcrypt -lc -lbsd
=> > => >     libc=/lib/libc.a, so=a, useshrplib=false, 
=> libperl=libperl.a
=> > => >     gnulibc_version=''
=> > => >   Dynamic Linking:
=> > => >     dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, 
=> > => ccdlflags='-Xlinker 
=> > => -bE:/usr/local/perl-5.8.4/lib/5.8.4/aix/CORE/perl.exp'
=> > => >     cccdlflags=' ', lddlflags='  -Wl,-bhalt:4 -Wl,-bM:SRE 
=> > => -Wl,-bI:$(PERL_INC)/perl.exp -Wl,-bE:$(BASEEXT).exp 
=> > => -Wl,-bnoentry -lc -L/usr/local/lib'
=> > => > 
=> > => > Locally applied patches:
=> > => >     
=> > => > 
=> > => > ---
=> > => > @INC for perl v5.8.4:
=> > => >     /usr/local/perl-5.8.4/lib/5.8.4/aix
=> > => >     /usr/local/perl-5.8.4/lib/5.8.4
=> > => >     /usr/local/perl-5.8.4/lib/site_perl/5.8.4/aix
=> > => >     /usr/local/perl-5.8.4/lib/site_perl/5.8.4
=> > => >     /usr/local/perl-5.8.4/lib/site_perl
=> > => >     .
=> > => > 
=> > => > ---
=> > => > Environment for perl v5.8.4:
=> > => >     HOME=/home/adanial
=> > => >     LANG=en_US
=> > => >     LANGUAGE (unset)
=> > => >     LC__FASTMSG=true
=> > => >     
=> > => LD_LIBRARY_PATH=/opt/platform/lsf/4.2/aix4/lib:/usr/local/kma
=> > => tplot-0.2.2/lib
=> > => >     LIBPATH=/usr/lib::/opt/matlab53/extern/lib/ibm_rs
=> > => >     LOGDIR (unset)
=> > => >     
=> > => PATH=/opt/platform/lsf/4.2/aix4/bin:/opt/platform/lsf/4.2/aix
=> > 
=> 4/etc:/opt/IDEAS7/ideas/ms7/bin:/home/adanial/bin:/bin:/usr/b
=> in:/usr/ccs/bin:/usr/ucb:/usr/bin/X11:/usr/=> 
=> dt/bin:/usr/local/bin:.:/opt/sysnoise/5.5/bin:/opt/matlab53/b
=> > in:/opt/langtools/bin:/opt/linux/IBMJava2-13/bin
=> > => >     PERL_BADLANG (unset)
=> > => >     SHELL=/bin/bash
=> > => > 
=> > => 
=> > => -- 
=> > => H.Merijn Brand        Amsterdam Perl Mongers 
=> > (http://amsterdam.pm.org/)
=> > using perl-5.6.1, 5.8.3, & 5.9.x, and 809 on  HP-UX 10.20 
=> & 11.00, 11i,
=> >    AIX 4.3, SuSE 9.0, and Win2k.           
=> http://www.cmve.net/~merijn/
=> > http://archives.develooper.com/daily-build@perl.org/   
=> perl-qa@perl.org
=> > send smoke reports to: smokers-reports@perl.org, QA: 
http://qa.perl.org
> 
> 

-- 
H.Merijn Brand        Amsterdam Perl Mongers (http://amsterdam.pm.org/)
using perl-5.6.1, 5.8.3, & 5.9.x, and 809 on  HP-UX 10.20 & 11.00, 11i,
   AIX 4.3, SuSE 9.0, and Win2k.           http://www.cmve.net/~merijn/
http://archives.develooper.com/daily-build@perl.org/   perl-qa@perl.org
send smoke reports to: smokers-reports@perl.org, QA: http://qa.perl.org


Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About