develooper Front page | perl.perl5.porters | Postings from May 2004

Re: [perl #29873] seg fault w/v5.8.4 on IBM RS/6k when populating large hash

Thread Previous | Thread Next
From:
H.Merijn Brand
Date:
May 27, 2004 02:19
Subject:
Re: [perl #29873] seg fault w/v5.8.4 on IBM RS/6k when populating large hash
Message ID:
20040527105931.6F18.H.M.BRAND@hccnet.nl
On Wed 26 May 2004 18:37, <al.danial@ngc.com> wrote:
> Merijn, thanks for looking into the problem so quickly.
> I neglected to mention that the other machines on which
> the program runs all have much less memory than the 8 GB
> on the IBM.  The Linux box has (only) 1 GB RAM and has 
> no problems with the code.

Linux is able to give you more memory than AIX. Look at your limits.

On my Linux box I've all my limits set to unlimited, but on AIX

i2:/u/usr/merijn 101 > limit
cputime      unlimited
filesize     unlimited
datasize     131072 kbytes
stacksize    32768 kbytes
coredumpsize 1024 kbytes
memoryuse    32768 kbytes
descriptors  2000
i2:/u/usr/merijn 102 >

And then you also have to check the system imposed (hard) limits

i2:/u/usr/merijn 102 > limit -h
cputime      unlimited
filesize     unlimited
datasize     unlimited
stacksize    unlimited
coredumpsize unlimited
memoryuse    unlimited
descriptors  unlimited
i2:/u/usr/merijn 103 >

You could try to increase the personal limits, but you're
not always allowed to:

i2:/u/usr/merijn 103 > limit memoryuse unlimited
i2:/u/usr/merijn 104 > limit
cputime      unlimited
filesize     unlimited
datasize     131072 kbytes
stacksize    32768 kbytes
coredumpsize 1024 kbytes
memoryuse    unlimited
descriptors  2000
i2:/u/usr/merijn 105 >

Playing around and setting more to unlimited, makes your
script most likely pass:

i2:/tmp 113 > limit
cputime      unlimited
filesize     unlimited
datasize     unlimited
stacksize    unlimited
coredumpsize 1024 kbytes
memoryuse    unlimited
descriptors  2000
i2:/tmp 114 > perl test.pl
i2:/tmp 115 >

This is the same script that previously gave me out of memory

As a sysamin, you could check out /etc/security/limits

As this is no bug in Perl, I will close it

> Here is my rough estimate of the memory that would be
> taken by the scalar data within the hash of hash of lists:
> 
>     3 *        # number of sets
>     300000 *   # number of ID's
>     7 *        # number of rand's
>    (9*4        # number of int's per scalar variable (??)
>                # times number of bytes per int plus
>     + 8)       # number of bytes for the rand double 
> 
>   = 277 MB
> 
> I have no idea how much additional space is needed by the hash 
> tables but even if it is double the amount above, the code 
> should be able to (and does on Linux) run comfortably within 
> 1 GB.          -- Al
> 
>  -+- -+- -+- -+- -+- -+- -+- -+- -+- -+- -+- -+- -+- -+-
> Albert Danial  al.danial@ngc.com  310-813-9516 fax: -4467
> NGST R10/1328 One Space Park, Redondo Bch., CA 90278, USA 
> 
> 
> 
> => -----Original Message-----
> => From: H. Merijn Brand via RT [mailto:perlbug-followup@perl.org]
> => Sent: Wednesday, May 26, 2004 1:50 AM
> => To: Danial, Al
> => Subject: Re: [perl #29873] seg fault w/v5.8.4 on IBM RS/6k when
> => populating large hash 
> => 
> => 
> => On Wed 26 May 2004 02:41, "al.danial@ngc.com (via RT)" 
> => <perlbug-followup@perl.org> wrote:
> => > # New Ticket Created by  al.danial@ngc.com 
> => > # Please include the string:  [perl #29873]
> => > # in the subject line of all future correspondence about 
> => this issue. 
> => > # <URL: http://rt.perl.org:80/rt3/Ticket/Display.html?id=29873 >
> => > 
> => > 
> => > This is a bug report for perl from al.danial@ngc.com,
> => > generated with the help of perlbug 1.35 running under perl v5.8.4.
> => > 
> => > 
> => > -----------------------------------------------------------------
> => > This program causes a seg fault on an IBM RS/6000 with AIX 4.3.3.
> => > 
> => >     #!/usr/local/perl-5.8.4/bin/perl -w
> => >     my $upper_limit;
> => >     $upper_limit = 275000;  # works
> => >     $upper_limit = 300000;  # fails
> => >     my %data = ();
> => >     foreach my $set (qw( A B C )) {
> => >         foreach my $id (1..$upper_limit) {
> => >             $data{$set}{$id} = [ rand, rand, rand, rand, 
> => rand, rand, rand, ];
> => >         }
> => >     }
> => 
> => That fully depends on your system configuration. On my 
> => system, the first limit
> => also causes problems on both recent and older perl's:
> => 
> => i2:/tmp 103 > perl test.pl
> => Out of memory!
> => Exit 1
> => i2:/tmp 104 >
> => 
> => i2:/tmp 104 > perl5.6.1 test.pl
> => Out of memory!
> => Exit 1
> => i2:/tmp 105 >
> => 
> => No seg faults though. Mahor difference between my perl's and 
> => yours is that
> => myine are built with vac, and your's are built with gcc.
> => 
> => > The failure happens with Perl version 5.6.1 on the IBM as well.
> => > 
> => > The workstation has 8 GB of memory so I don't think it is because
> => > the code is using too much space.  The above program works fine
> => > on Linux, Sun, and SGI workstations using Perl 5.8.0.
> => > 
> => > 
> => > [Please do not change anything below this line]
> => > -----------------------------------------------------------------
> => > ---
> => > Flags:
> => >     category=core
> => >     severity=high
> => > ---
> => > Site configuration information for perl v5.8.4:
> => > 
> => > Configured by adanial at Thu May 20 11:47:39 PDT 2004.
> => > 
> => > Summary of my perl5 (revision 5 version 8 subversion 4) 
> => configuration:
> => >   Platform:
> => >     osname=aix, osvers=4.3.3.0, archname=aix
> => >     uname='aix mohr 3 4 00041bcf4c00 '
> => >     config_args='-Dprefix=/usr/local/perl-5.8.4 -Dcc=gcc -de'
> => >     hint=recommended, useposix=true, d_sigaction=define
> => >     usethreads=undef use5005threads=undef 
> => useithreads=undef usemultiplicity=undef
> => >     useperlio=define d_sfio=undef uselargefiles=define 
> => usesocks=undef
> => >     use64bitint=undef use64bitall=undef uselongdouble=undef
> => >     usemymalloc=n, bincompat5005=undef
> => >   Compiler:
> => >     cc='gcc', ccflags ='-D_ALL_SOURCE -D_ANSI_C_SOURCE 
> => -D_POSIX_SOURCE -DUSE_NATIVE_DLOPEN -fno-strict-aliasing 
> => -I/usr/local/include -D_LARGE_FILES',
> => >     optimize='-O',
> => >     cppflags='-D_ALL_SOURCE -D_ANSI_C_SOURCE 
> => -D_POSIX_SOURCE -DUSE_NATIVE_DLOPEN -fno-strict-aliasing 
> => -I/usr/local/include'
> => >     ccversion='', gccversion='3.3.3', gccosandvers='aix4.3.3.0'
> => >     intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321
> => >     d_longlong=define, longlongsize=8, d_longdbl=define, 
> => longdblsize=8
> => >     ivtype='long', ivsize=4, nvtype='double', nvsize=8, 
> => Off_t='off_t', lseeksize=8
> => >     alignbytes=8, prototype=define
> => >   Linker and Libraries:
> => >     ld='gcc', ldflags =' -Wl,-brtl -L/usr/local/lib -Wl,-b32'
> => >     libpth=/usr/local/lib /lib /usr/lib /usr/ccs/lib
> => >     libs=-lbind -lnsl -lgdbm -ldbm -ldb -ldl -lld -lm 
> => -lcrypt -lc -lbsd
> => >     perllibs=-lbind -lnsl -ldl -lld -lm -lcrypt -lc -lbsd
> => >     libc=/lib/libc.a, so=a, useshrplib=false, libperl=libperl.a
> => >     gnulibc_version=''
> => >   Dynamic Linking:
> => >     dlsrc=dl_aix.xs, dlext=so, d_dlsymun=undef, 
> => ccdlflags='-Xlinker 
> => -bE:/usr/local/perl-5.8.4/lib/5.8.4/aix/CORE/perl.exp'
> => >     cccdlflags=' ', lddlflags='  -Wl,-bhalt:4 -Wl,-bM:SRE 
> => -Wl,-bI:$(PERL_INC)/perl.exp -Wl,-bE:$(BASEEXT).exp 
> => -Wl,-bnoentry -lc -L/usr/local/lib'
> => > 
> => > Locally applied patches:
> => >     
> => > 
> => > ---
> => > @INC for perl v5.8.4:
> => >     /usr/local/perl-5.8.4/lib/5.8.4/aix
> => >     /usr/local/perl-5.8.4/lib/5.8.4
> => >     /usr/local/perl-5.8.4/lib/site_perl/5.8.4/aix
> => >     /usr/local/perl-5.8.4/lib/site_perl/5.8.4
> => >     /usr/local/perl-5.8.4/lib/site_perl
> => >     .
> => > 
> => > ---
> => > Environment for perl v5.8.4:
> => >     HOME=/home/adanial
> => >     LANG=en_US
> => >     LANGUAGE (unset)
> => >     LC__FASTMSG=true
> => >     
> => LD_LIBRARY_PATH=/opt/platform/lsf/4.2/aix4/lib:/usr/local/kma
> => tplot-0.2.2/lib
> => >     LIBPATH=/usr/lib::/opt/matlab53/extern/lib/ibm_rs
> => >     LOGDIR (unset)
> => >     
> => PATH=/opt/platform/lsf/4.2/aix4/bin:/opt/platform/lsf/4.2/aix
> 4/etc:/opt/IDEAS7/ideas/ms7/bin:/home/adanial/bin:/bin:/usr/bin:/usr/ccs/bin:/usr/ucb:/usr/bin/X11:/usr/=> dt/bin:/usr/local/bin:.:/opt/sysnoise/5.5/bin:/opt/matlab53/b
> in:/opt/langtools/bin:/opt/linux/IBMJava2-13/bin
> => >     PERL_BADLANG (unset)
> => >     SHELL=/bin/bash
> => > 
> => 
> => -- 
> => H.Merijn Brand        Amsterdam Perl Mongers 
> (http://amsterdam.pm.org/)
> using perl-5.6.1, 5.8.3, & 5.9.x, and 809 on  HP-UX 10.20 & 11.00, 11i,
>    AIX 4.3, SuSE 9.0, and Win2k.           http://www.cmve.net/~merijn/
> http://archives.develooper.com/daily-build@perl.org/   perl-qa@perl.org
> send smoke reports to: smokers-reports@perl.org, QA: http://qa.perl.org
> 
> 

-- 
H.Merijn Brand        Amsterdam Perl Mongers (http://amsterdam.pm.org/)
using perl-5.6.1, 5.8.3, & 5.9.x, and 809 on  HP-UX 10.20 & 11.00, 11i,
   AIX 4.3, SuSE 9.0, and Win2k.           http://www.cmve.net/~merijn/
http://archives.develooper.com/daily-build@perl.org/   perl-qa@perl.org
send smoke reports to: smokers-reports@perl.org, QA: http://qa.perl.org



Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About