develooper Front page | perl.ithreads | Postings from January 2009

Re: No gain in speed with threads

Thread Previous | Thread Next
From:
Hendrik Visage
Date:
January 2, 2009 17:03
Subject:
Re: No gain in speed with threads
Message ID:
ff6ed0780901020041x6326ebefg3205e88d1b1b51b9@mail.gmail.com
On Wed, Dec 31, 2008 at 15:32, Blanchette, Marco
<MAB@stowers-institute.org>wrote:

> Dear all,
>
> I am trying to speed up a very long procedure that I need to run on
> multiple files and though that I could multithread different jobs on
> different files across multiple CPUs. For some reason that I don't really
> get, I only achieve very small time gain. I have included my script which
> essentially repeat the same function, extractSeq() on multiple files using a
> maximum of four threads.
>
> I would really appreciate if I could finally understand how to use threads
> to speed up some of my lengthy scripts.


Yeah, the problem is these things are let lose without some nice computer
Science back grounds ;(

The "better" method in my opinion (and sorry for the pseudo code) is:


main_begin
 init_list(@file_list)

 for i in 1 2 3 4
 do
  thread_create(&worker())
 done

 join_threads

end_main

Procedure worker()
start
 local flag=1;

 while(flag)
   do
    mutex_lock(lock)
    next=read_and_remove_from_list(@file_list)
    mutex_unlock(lock)
    if (next == NULL)
     then
       flag=0
     else
       process(next)
     fi
  done
  cleanup_thread_stuff()
return


Notice that you don't have any delays like sleep etc. (which could also
cause bad lags if the actuall processing for hundreds or more files takes
less than a second per file..) and keep the read_and_remove_from_list short
and simple, then you'll see optimal speed increase.. also since disk reads
are involved, you might find that a number_of_cpus+extra_threads (given RAM
is sufficient) would give better results... the number for extra_threads to
be determine with emperical evaluation on the specific system etc.

Thread Previous | Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About