perl.ithreads

perl.ithreads http://www.nntp.perl.org/group/perl.ithreads/ ... Copyright 1998-2008 perl.org Tue, 07 Oct 2008 23:57:02 +0000 ask@perl.org Re: Lots of context switches with shared array by Dean Arnold Thomas Karcher wrote: > Hi Dean, > > > If someone from the ithreads development team is listening: I guess this > is an issue worth considering to have a solution for ... e. g. a > thread-safe array (similar to Thread::Queue) or a more fine-granular > locking mechanism for shared arrays ... not that I really thought it > through :) > FWIW: Thread::Queue *is* a thread::shared array, w/ just a few methods added. And it suffers from the same issues as your sort-merge, assuming enough threads are accessing the queue concurrently. The core issue is the global lock on the shared interpretter; until thats removed (which doesn't seem likely in the near future), shared data will remain a "contentious" issue. http://www.nntp.perl.org/group/perl.ithreads/2008/09/msg1327.html Tue, 23 Sep 2008 12:18:58 +0000 Re: Lots of context switches with shared array by Thomas Karcher Hi Dean, > > Thanks for the enlightening answer. Do you think it is possible to > > "trick" threads::shared's lock with a reference to the array, i. e. the > > reference is shared but the array itself isn't? > Not certain how you intend to do that ? Are you intending to write your > own XS implementation ? Keep in mind that a shared ref must point to No, that's too much and not an option for me. > a shared variable, ie, a shared variable can't be used as a ref to > a private variable. Right, that kills the idea ... too bad. Thanks for your comments. If someone from the ithreads development team is listening: I guess this is an issue worth considering to have a solution for ... e. g. a thread-safe array (similar to Thread::Queue) or a more fine-granular locking mechanism for shared arrays ... not that I really thought it through :) Regards, Thomas http://www.nntp.perl.org/group/perl.ithreads/2008/09/msg1326.html Tue, 23 Sep 2008 10:42:46 +0000 Re: Lots of context switches with shared array by Dean Arnold Thomas Karcher wrote: > Hi Dean, > > > Thanks for the enlightening answer. Do you think it is possible to > "trick" threads::shared's lock with a reference to the array, i. e. the > reference is shared but the array itself isn't? > Not certain how you intend to do that ? Are you intending to write your own XS implementation ? Keep in mind that a shared ref must point to a shared variable, ie, a shared variable can't be used as a ref to a private variable. Dean Arnold Presicient Corp. http://www.nntp.perl.org/group/perl.ithreads/2008/09/msg1325.html Tue, 23 Sep 2008 10:33:27 +0000 Re: Lots of context switches with shared array by Thomas Karcher Hi Dean, > > I'm writing a multithreaded merge sort that suffers from massive amounts > > of context switches (as seen by vmstat). The outline of the program is > > roughly as follows: > > > > ==== > > my @sortdata : shared; // the values to be sorted > > [...] > > // starting two threads > > new threads( \&mergesort_string, $begin, $half, $threaddepth - 1 ); > > new threads( \&mergesort_string, $half, $end, $threaddepth - 1 ); > > [...] > > > > sub mergesort_string { > > my ( $begin, $end, $threaddepth ) = @_; > > > > // sorting "my" part of @sortdata, as delimited > > // by $begin and $end > > } > > ==== > > > > By playing around, I discovered that the "shared" is what causes the > > context switches. They take up about 40% of the duration of the program > > for an array of 10,000 elements. > > > > I don't need any locking on the elements itself because the merge sort > > algorithm itself guarantees in my opinion that there is no conflicting > > access to them. > > > > Am I missing something here? How can I avoid those context switches that > > make the program essentially useless? > I'm afraid there isn't much you can do short of writing your own XS code that > bypasses threads::shared. Due to the global lock on the shared interpretter (where > shared variables are actually stored), each thread that touches the array is going > to be trying to acquire the global lock, ergo lots of lock contention, leading to lots > of probable context switches. > You may be able to improve performance by copying parts of the array out to a private > version, but that may not adapt to your algorithm. Thanks for the enlightening answer. Do you think it is possible to "trick" threads::shared's lock with a reference to the array, i. e. the reference is shared but the array itself isn't? Thanks, Thomas http://www.nntp.perl.org/group/perl.ithreads/2008/09/msg1324.html Tue, 23 Sep 2008 08:32:54 +0000 Re: Lots of context switches with shared array by Dean Arnold Thomas Karcher wrote: > Hi there, > > I'm writing a multithreaded merge sort that suffers from massive amounts > of context switches (as seen by vmstat). The outline of the program is > roughly as follows: > > ==== > my @sortdata : shared; // the values to be sorted > [...] > // starting two threads > new threads( \&mergesort_string, $begin, $half, $threaddepth - 1 ); > new threads( \&mergesort_string, $half, $end, $threaddepth - 1 ); > [...] > > sub mergesort_string { > my ( $begin, $end, $threaddepth ) = @_; > > // sorting "my" part of @sortdata, as delimited > // by $begin and $end > } > ==== > > By playing around, I discovered that the "shared" is what causes the > context switches. They take up about 40% of the duration of the program > for an array of 10,000 elements. > > I don't need any locking on the elements itself because the merge sort > algorithm itself guarantees in my opinion that there is no conflicting > access to them. > > Am I missing something here? How can I avoid those context switches that > make the program essentially useless? > I'm afraid there isn't much you can do short of writing your own XS code that bypasses threads::shared. Due to the global lock on the shared interpretter (where shared variables are actually stored), each thread that touches the array is going to be trying to acquire the global lock, ergo lots of lock contention, leading to lots of probable context switches. You may be able to improve performance by copying parts of the array out to a private version, but that may not adapt to your algorithm. Dean Arnold Presicient Corp. http://www.nntp.perl.org/group/perl.ithreads/2008/09/msg1323.html Tue, 23 Sep 2008 08:21:11 +0000 Lots of context switches with shared array by Thomas Karcher Hi there, I'm writing a multithreaded merge sort that suffers from massive amounts of context switches (as seen by vmstat). The outline of the program is roughly as follows: ==== my @sortdata : shared; // the values to be sorted [...] // starting two threads new threads( \&mergesort_string, $begin, $half, $threaddepth - 1 ); new threads( \&mergesort_string, $half, $end, $threaddepth - 1 ); [...] sub mergesort_string { my ( $begin, $end, $threaddepth ) = @_; // sorting "my" part of @sortdata, as delimited // by $begin and $end } ==== By playing around, I discovered that the "shared" is what causes the context switches. They take up about 40% of the duration of the program for an array of 10,000 elements. I don't need any locking on the elements itself because the merge sort algorithm itself guarantees in my opinion that there is no conflicting access to them. Am I missing something here? How can I avoid those context switches that make the program essentially useless? Thanks! Thomas http://www.nntp.perl.org/group/perl.ithreads/2008/09/msg1322.html Tue, 23 Sep 2008 08:10:31 +0000 Re: XS callbacks from different threads. by Josef Wolf On Thu, Aug 07, 2008 at 10:02:29PM +0300, Shmuel Fomberg wrote: > >>>static SV *callback_ref = (SV*)NULL; > >>Reality check: you do know that "static" means here, right? > >I think so: I have exactly one instance of this variable. Do you see > >any problems with this? > > Um, you are thinking C++. No, I'm thinking pure C here. I don't line C++ very much. (please feel free to check http://yosefk.com/c++fqa/ for some reasons why I don't like C++) > declaring static variable outside object or > function says something about it's scope. (you still get one instance of > this, even without the static) Since we were talking about threads, I assumed that your question was about having one instance (shared by all threads) against having one instance per thread. I need to pass the callback reference from the main thread to the callback thread, so having a shared variable for this is exactly what I want. The scope should surely be limited to my file, because I don't want to export objects without a reason. I still don't see any problem with this. > >>>int call_perl (int cnt, ...) > >>>{ > >>> dTHX; > >>> PERL_SET_CONTEXT(my_perl); > >>I hate the "my_perl" parameter. Who defined it? what is it set to? > >AFAIK, this is created by dTHX. > > Correct. but from where does the dTHX take it? from the connection > between the thread and one perl interpreter. who makes the connection? > PERL_SET_CONTEXT. but the PERL_SET_CONTEXT is inited with my_perl... see > the loop here? I agree that the dTHXa() method you mentioned is the cleaner method. BTW: I can't find any reference of this macro in the documentation. Are you sure this is officially part of the API? http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1321.html Fri, 08 Aug 2008 08:10:29 +0000 Re: XS callbacks from different threads. by Shmuel Fomberg Josef Wolf wrote: >> Well, maybe, but there is already one thread that is passing through >> perl, so calling PERL_SET_CONTEXT for this perl on other thread is >> pulling the rug under the first thread's legs. it can't be good. > I've got it working now. The solution is to PERL_SET_CONTEXT() > _before_ declaring the context with dTHX. The code looks like this now: Interesting to know that it's working. Thanks. >>> static SV *callback_ref = (SV*)NULL; >> Reality check: you do know that "static" means here, right? > I think so: I have exactly one instance of this variable. Do you see > any problems with this? Um, you are thinking C++. declaring static variable outside object or function says something about it's scope. (you still get one instance of this, even without the static) >>> int call_perl (int cnt, ...) >>> { >>> dTHX; >>> PERL_SET_CONTEXT(my_perl); >> I hate the "my_perl" parameter. Who defined it? what is it set to? > AFAIK, this is created by dTHX. Correct. but from where does the dTHX take it? from the connection between the thread and one perl interpreter. who makes the connection? PERL_SET_CONTEXT. but the PERL_SET_CONTEXT is inited with my_perl... see the loop here? Shmuel. http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1320.html Thu, 07 Aug 2008 12:04:26 +0000 Re: XS callbacks from different threads. by Josef Wolf Thanks for your help and your passience, Shmuel! On Tue, Aug 05, 2008 at 11:03:58PM +0300, Shmuel Fomberg wrote: > Well, maybe, but there is already one thread that is passing through > perl, so calling PERL_SET_CONTEXT for this perl on other thread is > pulling the rug under the first thread's legs. it can't be good. I've got it working now. The solution is to PERL_SET_CONTEXT() _before_ declaring the context with dTHX. The code looks like this now: static PerlInterpreter *orig_perl; static SV *callback_ref = (SV*)NULL; static int do_call_perl (size_t argc, char *argv[]) { dTHX; dSP; I32 ax; /* ... snipped here ... */ } static int call_perl_argv (size_t argc, char *argv[]) { int ret; static pthread_mutex_t mutex=PTHREAD_MUTEX_INITIALIZER; pthread_mutex_lock(&mutex); PERL_SET_CONTEXT(orig_perl); ret = do_call_perl (argc, argv); pthread_mutex_unlock(&mutex); return ret; } void register_callback (SV *callback) { dTHX; if (callback_ref == (SV*)NULL) { callback_ref = newSVsv (callback); /* first time, create new SV */ } else { SvSetSV (callback_ref, callback); /* been here, overwrite */ } orig_perl = my_perl; do { sleep (3); } while (call_perl (1, "check")); shutdown_threads (); } > Maybe you should use two threads: one of the calling perl, and one slave > perl. because as said two paragraphs ago, I don't think that using one > perl interpreter will work. Now I have multiple C-threads calling one perl interpreter. Multiple entries into the interpreter are protected by a mutex. That's exactly what I've tried to achieve (at least for now). > >static SV *callback_ref = (SV*)NULL; > > Reality check: you do know that "static" means here, right? I think so: I have exactly one instance of this variable. Do you see any problems with this? > >int call_perl (int cnt, ...) > >{ > > dTHX; > > PERL_SET_CONTEXT(my_perl); > > I hate the "my_perl" parameter. Who defined it? what is it set to? AFAIK, this is created by dTHX. > Especially if you have more then one perl interpreter in you code. I used: > dTHXa(ph->perl); > PERL_SET_CONTEXT(ph->perl); > where ph was some struct that held the perl interpreter for me... Ah, I did not know about dTHXa(). http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1319.html Wed, 06 Aug 2008 12:00:44 +0000 Re: XS callbacks from different threads. by Shmuel Fomberg Josef Wolf wrote: >> For this one you need a real expert. It is possible to call into Perl >> from an XS sub. but to call into Perl from other threads while it is >> inside a XS call? I have no idea. > > As far as I understand the corresponding section in perlguts, it should > be possible _if_ PERL_SET_CONTEXT() is called before. But for some > reason it keeps crashing :-( Well, maybe, but there is already one thread that is passing through perl, so calling PERL_SET_CONTEXT for this perl on other thread is pulling the rug under the first thread's legs. it can't be good. >>> 2. I need to create a second perl interpreter (or clone the first one). >>> Can somebody point me to example code how to do that? >> That's actually no big deal. look at perldoc page perlembed, there are >> examples. > What I am missing from those examples is the explanation how the > different interpreters would communicate. There are two ways to communicate between two thread: using Perl's mechanizem, or using C. either possible. just stick to one and don't mix. For example, if you choose to use C communication, then after initizing the lib, the master perl should call some C function that waits until the lib finished working / need restart / etc. The lib will use the same system to release the master from it's sleep, either directly or even an XS sub that the perl code will call. (C calling Perl calling C...) > And there's one more complication: the low-level library don't even > state how many threads will call my callback. Therefore I would > prefer to have only _one_ interpreter and use a mutex to keep the > threads from entering the interpreter simultaneously. Maybe you should use two threads: one of the calling perl, and one slave perl. because as said two paragraphs ago, I don't think that using one perl interpreter will work. > static SV *callback_ref = (SV*)NULL; Reality check: you do know that "static" means here, right? > int call_perl (int cnt, ...) > { > dTHX; > PERL_SET_CONTEXT(my_perl); I hate the "my_perl" parameter. Who defined it? what is it set to? Especially if you have more then one perl interpreter in you code. I used: dTHXa(ph->perl); PERL_SET_CONTEXT(ph->perl); where ph was some struct that held the perl interpreter for me... Good luck, anyway. Shmuel. http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1318.html Tue, 05 Aug 2008 13:04:20 +0000 Re: XS callbacks from different threads. by Josef Wolf Thanks for your efforts, Shmuel! On Mon, Aug 04, 2008 at 08:10:31PM +0300, Shmuel Fomberg wrote: > >>IANAE, (I am not an expert) but if you say that the main thread returns > >>immediately and waiting in a loop for the lib, then you already have one > >>thread inside Perl. So the background threads shouldn't call into your > >>perl interpreter. > > > >If I understand correctly, I have two options then: > > > >1. _not_ let the main thread return back from the XS. That is, I > > move the sleep(0) from the perl script into the init() function. > > Would that break anything withhin perl? > > For this one you need a real expert. It is possible to call into Perl > from an XS sub. but to call into Perl from other threads while it is > inside a XS call? I have no idea. As far as I understand the corresponding section in perlguts, it should be possible _if_ PERL_SET_CONTEXT() is called before. But for some reason it keeps crashing :-( > >2. I need to create a second perl interpreter (or clone the first one). > > Can somebody point me to example code how to do that? > > That's actually no big deal. look at perldoc page perlembed, there are > examples. What I am missing from those examples is the explanation how the different interpreters would communicate. And there's one more complication: the low-level library don't even state how many threads will call my callback. Therefore I would prefer to have only _one_ interpreter and use a mutex to keep the threads from entering the interpreter simultaneously. This is how my code (still crashing) looks right now: # define PERL_NO_GET_CONTEXT # include "defs.h" static SV *callback_ref = (SV*)NULL; static pthread_mutex_t mutex=PTHREAD_MUTEX_INITIALIZER; /* This function works OK when called from module_config() in main thread, but it crashes on PUSHMARK(SP) when called from other threads */ int call_perl (int cnt, ...) { dTHX; dSP; int i; I32 numargs; va_list args; pthread_mutex_lock(&mutex); PERL_SET_CONTEXT(my_perl); ENTER; SAVETMPS; PUSHMARK(SP); /* Here it crashes when called from different thread */ va_start (args, cnt); while (cnt-- > 0) { char *str = va_arg(args, char*); XPUSHs(sv_2mortal (newSVpv (str, strlen (str)))); } va_end (args); PUTBACK; numargs = call_sv (callback_ref, G_SCALAR); if (numargs != 1) croak("Big trouble\n"); SPAGAIN; for (i=0; i<numargs; i++) { printf ("ret %i\n", POPi); } PUTBACK; FREETMPS; LEAVE; pthread_mutex_unlock(&mutex); return numargs; } void module_config (SV *callback) { dTHX; if (callback_ref == (SV*)NULL) { callback_ref = newSVsv (callback); /* first time, create new SV */ } else { SvSetSV (callback_ref, callback); /* been here, overwrite */ } start_threads (); do { sleep (180); } while (call_perl (1, "check")); /* works OK when called from here */ } http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1317.html Mon, 04 Aug 2008 15:10:26 +0000 Re: XS callbacks from different threads. by Shmuel Fomberg Hi Josef. >> IANAE, (I am not an expert) but if you say that the main thread returns >> immediately and waiting in a loop for the lib, then you already have one >> thread inside Perl. So the background threads shouldn't call into your >> perl interpreter. > > If I understand correctly, I have two options then: > > 1. _not_ let the main thread return back from the XS. That is, I > move the sleep(0) from the perl script into the init() function. > Would that break anything withhin perl? For this one you need a real expert. It is possible to call into Perl from an XS sub. but to call into Perl from other threads while it is inside a XS call? I have no idea. > 2. I need to create a second perl interpreter (or clone the first one). > Can somebody point me to example code how to do that? That's actually no big deal. look at perldoc page perlembed, there are examples. Shmuel. http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1316.html Mon, 04 Aug 2008 10:12:06 +0000 Re: XS callbacks from different threads. by Josef Wolf On Sun, Aug 03, 2008 at 10:07:26PM +0300, Shmuel Fomberg wrote: > Josef Wolf wrote: > > > IIUC, I don't need to create multiple PerlInterpreter's. I just need > > to set up the TLS properly in each thread. But it is not clear to me > > how to do that. > > # Main thread returns immediately after creating background threads > > # For now we only sit and wait until we get killed. In a future > > # improvement we might periodically check for configuration > > # changes and restart. > > IANAE, (I am not an expert) but if you say that the main thread returns > immediately and waiting in a loop for the lib, then you already have one > thread inside Perl. So the background threads shouldn't call into your > perl interpreter. Thanks for your answer, Shmuel! If I understand correctly, I have two options then: 1. _not_ let the main thread return back from the XS. That is, I move the sleep(0) from the perl script into the init() function. Would that break anything withhin perl? 2. I need to create a second perl interpreter (or clone the first one). Can somebody point me to example code how to do that? http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1315.html Sun, 03 Aug 2008 23:10:27 +0000 Re: XS callbacks from different threads. by Shmuel Fomberg Josef Wolf wrote: > IIUC, I don't need to create multiple PerlInterpreter's. I just need > to set up the TLS properly in each thread. But it is not clear to me > how to do that. > # Main thread returns immediately after creating background threads > # For now we only sit and wait until we get killed. In a future > # improvement we might periodically check for configuration > # changes and restart. IANAE, (I am not an expert) but if you say that the main thread returns immediately and waiting in a loop for the lib, then you already have one thread inside Perl. So the background threads shouldn't call into your perl interpreter. Shmuel. http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1314.html Sun, 03 Aug 2008 12:07:38 +0000 XS callbacks from different threads. by Josef Wolf Hello, I am trying to write a XS module for a C library that creates its own threads from which I have to pass callbacks back into perl. Unfortunately, I am pretty much confused by the discussion of threads in perlguts. IIUC, I don't need to create multiple PerlInterpreter's. I just need to set up the TLS properly in each thread. But it is not clear to me how to do that. From the perl point of view, this library would be used like this: use Mylib; if (Mylib::init()) { # this will create the additional threads # Main thread returns immediately after creating background threads # For now we only sit and wait until we get killed. In a future # improvement we might periodically check for configuration # changes and restart. # sleep (0); Mylib::shutdown(); # Cleanup } sub callback { # this will be called when the other threads want to notify # any events } The XS part of the interface seems to be easy: #include "EXTERN.h" #include "perl.h" #include "XSUB.h" #include "ppport.h" #include </m/l/misc/mylib/mylib.h> MODULE = Mylib PACKAGE = Mylib void deinit() int init() And here is the C part of the interface into the library: /* Protects against concurrent calls from different threads into perl */ static pthread_mutex_t mutex=PTHREAD_MUTEX_INITIALIZER; /* This function will be called by the threads created in lib_init() */ void test_callback (void) { dTHX; dSP; pthread_mutex_lock(&mutex); PUSHMARK(SP); XPUSHs(sv_2mortal(newSVpv("some", 4))); XPUSHs(sv_2mortal(newSVpv("test", 4))); XPUSHs(sv_2mortal(newSVpv("vals", 4))); PUTBACK; call_pv("callback", G_DISCARD); pthread_mutex_unlock(&mutex); } int init (void) { /* For test purposes we call this from the main thread before any other threads are created */ test_callback(); /* the lib_init() function will create the additional threads which in turn will be calling test_callback() */ return lib_init (); } void deinit (void) { lib_deinit (); } Any hints? Or maybe a pointer to an example? http://www.nntp.perl.org/group/perl.ithreads/2008/08/msg1313.html Fri, 01 Aug 2008 18:00:31 +0000 Re: Thread::pool by Javier Hi!, I understand that, to initialize the $pool ideally between daemon() and scan_files(), just after the fork. The code: .... daemon(); $pool = Thread::Pool->new( { optimize => 'memory', do => \&check_system, pre => sub {shift; print "starting a new work with\n" }, post => sub { print "stopping the work\n" }, frequency => 10, autoshutdown => 1, workers => 20, maxjobs => 100, minjobs => 10, } ); scan_files(); And the daemon() function implement the fork. Well, so the program does not work. The program execute, but does nothing. It was stopped without doing anything. Just run once while(1) of scan_files(). I think that is how should the $pool did not exist. 2008/7/29 Jerry D. Hedden <jdhedden@cpan.org>: >> First i initialice the pool and then i set the process as a daemon. Then i >> call the scan_files() function. > > Nope. That's no good. > >> I also test to initialice the pool inside the daemon() function > > Depends on where in daemon you do it - it must come after the fork. > >> and between daemon() and scan_files(). > > Better. > >> Doing that, the output is blank, but it doesn't work. > > Define "doesn't work". > -- A greeting, Javier. http://www.nntp.perl.org/group/perl.ithreads/2008/07/msg1312.html Wed, 30 Jul 2008 01:11:56 +0000 Re: Thread::pool by Javier Hi! scan_files() is a function. Where is? This function is present in same file. This code is very simple: = = = = * * * * = = = = sub scan_files { my $DIR = "/var/workq/"; while(1) { opendir(DIRHANDLE, $DIR) || die "ERROR: $DIR not read\n"; foreach (readdir(DIRHANDLE)) { if (!-d $_) { $pool->job($_); } } closedir DIRHANDLE; sleep($Fax::Config::param{'queuescanperiod'}); } } = = = = * * * * = = = = 2008/7/29 Chris Fowler <cfowler@outpostsentinel.com>: > Where is scan_files()? > > -- > Chris Fowler > OutPost Sentinel, LLC > AIM: strtok2002 > Support @ SIP/support@pbx.opsdc.com > or 678-804-8193 > Email Support @ support@outpostsentinel.com > > > -- A greeting, Javier. http://www.nntp.perl.org/group/perl.ithreads/2008/07/msg1311.html Wed, 30 Jul 2008 01:00:35 +0000 Re: Thread::pool by Jerry D. Hedden > First i initialice the pool and then i set the process as a daemon. Then i > call the scan_files() function. Nope. That's no good. > I also test to initialice the pool inside the daemon() function Depends on where in daemon you do it - it must come after the fork. > and between daemon() and scan_files(). Better. > Doing that, the output is blank, but it doesn't work. Define "doesn't work". http://www.nntp.perl.org/group/perl.ithreads/2008/07/msg1310.html Tue, 29 Jul 2008 05:39:28 +0000 Re: Thread::pool by Chris Fowler Where is scan_files()? -- Chris Fowler OutPost Sentinel, LLC AIM: strtok2002 Support @ SIP/support@pbx.opsdc.com or 678-804-8193 Email Support @ support@outpostsentinel.com http://www.nntp.perl.org/group/perl.ithreads/2008/07/msg1309.html Tue, 29 Jul 2008 05:28:25 +0000 Thread::pool by Javier Hi, I have a problem with the module Thread::Pool-0.32. I need to set my script as a daemon, but when i use this module the program kill the threads. The output of my script is: starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with starting a new work with Perl exited with active threads: 20 running and unjoined 0 finished and unjoined 0 running and detached And the source code is: = = = = = = = = * * * * * * * * * * = = = = = = = = my $pool = Thread::Pool->new( { optimize => 'memory', do => \&check_system, pre => sub {shift; print "starting a new work with @_\n" }, post => sub { print "stopping the work\n" }, frequency => 10, autoshutdown => 1, workers => 20, maxjobs => 100, minjobs => 10, } ); sub daemon { chdir "/"; open STDIN, '/dev/null'; fork && exit; setsid(); if (open PID, '>', '/var/run/faxqr.pid') { print PID "$$\n"; close PID; } } daemon(); scan_files(); = = = = = = = = * * * * * * * * * * = = = = = = = = First i initialice the pool and then i set the process as a daemon. Then i call the scan_files() function. I also test to initialice the pool inside the daemon() function and between daemon() and scan_files(). Doing that, the output is blank, but it doesn't work. Thank you for your help. -- A greeting, Javier. http://www.nntp.perl.org/group/perl.ithreads/2008/07/msg1308.html Tue, 29 Jul 2008 05:24:08 +0000 Re: Calling a object method in a create() by Valter Douglas Lisbôa Jr. Thanks, it worked! On Monday 14 July 2008 12:41:28 Jerry D. Hedden wrote: > Valter Douglas Lisbôa Jr. wrote: > > The code bellow functions very nice. > > > > --------------------- > > while(my ($id, $ep, $local, $tolerance, $lasterr, $hostscount) = > > $sthLocals->fetchrow_array()) { > > $self->localPolling($id, $ep, $local, $tolerance, $lasterr, > > $hostscount); } > > --------------------- > > > > But if I try to execute with a thread: > > > > --------------------- > > while(my ($id, $ep, $local, $tolerance, $lasterr, $hostscount) = > > $sthLocals->fetchrow_array()) { > > my $thr = threads->create($self->localPooling, $id, $ep, $local, > > $tolerance, $lasterr, $hostscount); > > } > > --------------------- > > > > It returns a error: > > Can't locate object method "localPooling" via package "Poller::Check"! > > Try > > while (my ($id, $ep, $local, $tolerance, $lasterr, $hostscount) > = $sthLocals->fetchrow_array()) > { > my $thr = threads->create(sub { > $self->localPooling($id, $ep, $local, $tolerance, $lasterr, > $hostscount); > }); > } -- Valter Douglas Lisbôa Jr. Sócio-Diretor Trenix - IT Solutions "Nossas Idéias, suas Soluções!" www.trenix.com.br contato@trenix.com.br Tel. +55 19 3402.2957 Cel. +55 19 9183.4244 http://www.nntp.perl.org/group/perl.ithreads/2008/07/msg1307.html Mon, 14 Jul 2008 09:33:37 +0000 Re: Calling a object method in a create() by Jerry D. Hedden Valter Douglas Lisbôa Jr. wrote: > The code bellow functions very nice. > > --------------------- > while(my ($id, $ep, $local, $tolerance, $lasterr, $hostscount) = > $sthLocals->fetchrow_array()) { > $self->localPolling($id, $ep, $local, $tolerance, $lasterr, $hostscount); > } > --------------------- > > But if I try to execute with a thread: > > --------------------- > while(my ($id, $ep, $local, $tolerance, $lasterr, $hostscount) = > $sthLocals->fetchrow_array()) { > my $thr = threads->create($self->localPooling, $id, $ep, $local, $tolerance, > $lasterr, $hostscount); > } > --------------------- > > It returns a error: > Can't locate object method "localPooling" via package "Poller::Check"! Try while (my ($id, $ep, $local, $tolerance, $lasterr, $hostscount) = $sthLocals->fetchrow_array()) { my $thr = threads->create(sub { $self->localPooling($id, $ep, $local, $tolerance, $lasterr, $hostscount); }); } http://www.nntp.perl.org/group/perl.ithreads/2008/07/msg1306.html Mon, 14 Jul 2008 08:41:37 +0000 Calling a object method in a create() by Valter Douglas Lisbôa Jr. Hi all, I new on threads, but not so on perl. I trying to execute a thread inside a object method of a class that I make to consult SNMP quests and feed a rrddatabase. The code bellow functions very nice. --------------------- while(my ($id, $ep, $local, $tolerance, $lasterr, $hostscount) = $sthLocals->fetchrow_array()) { $self->localPolling($id, $ep, $local, $tolerance, $lasterr, $hostscount); } --------------------- But if I try to execute with a thread: --------------------- while(my ($id, $ep, $local, $tolerance, $lasterr, $hostscount) = $sthLocals->fetchrow_array()) { my $thr = threads->create($self->localPooling, $id, $ep, $local, $tolerance, $lasterr, $hostscount); } --------------------- It returns a error: Can't locate object method "localPooling" via package "Poller::Check"! I try to put a \& before $self-> and take another error: Not a code reference Are there a way to call a class $self->method in a thread? I search across Google and not find any clue. Thanks in advance. -- Valter Douglas LisbÃ´a Jr. SÃ³cio-Diretor Trenix - IT Solutions "Nossas IdÃ©ias, suas SoluÃ§Ãµes!" www.trenix.com.br contato@trenix.com.br Tel. +55 19 3402.2957 Cel. +55 19 9183.4244 http://www.nntp.perl.org/group/perl.ithreads/2008/07/msg1305.html Mon, 14 Jul 2008 07:22:53 +0000 environment variables by Wong, Danny H. Hi All, I have a script that set environment variables for 10 plus configuration and each configuration share some environment variable and have some different or specific environment variables for a particular configuration. I set the env variables before calling each thread, but I notice that the environment variable inherits the previous configurations environment setting. Example: configure 1: set 11 variables Configure 2: set 10 variables, all of of configure minus 1 specific variable But configure 2 has all eleven variables. Is there a way to delete the environment variables before setting each configuration? I tried undef (%ENV) SETENV (); but no luck? Also, I read that environment variables in Linux are cached? If it true, is there a way to clear the cache without rebooting the system? Any ideas how I can delete all the environment variables before setting them again for the next configuration? thanks in advance. - - - - - Cisco - - - - - This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer. http://www.nntp.perl.org/group/perl.ithreads/2008/06/msg1304.html Fri, 13 Jun 2008 17:15:54 +0000 Re: Memory usage by Javier Terceiro 2008/6/11 Jerry D. Hedden <jdhedden@cpan.org>: > On Wed, Jun 11, 2008 at 9:26 AM, Javier Terceiro <correolista@gmail.com> wrote: >> Hello list, >> >> I am designed a new software. I am using threads. The operation is >> correct, but memory consumption is excessive. The system is using 100 >> MB and when the software is working, 500 Mb. The memory is not >> released, why? > > You need to either join the threads after they're done working, or > else detach them. See below. > >> >> The source code is: >> >> ----------------------------------------------------------------------------- >> >> #!/usr/bin/perl -w >> >> use strict; >> use warnings; >> use AnyData; >> use Sys::Syslog; >> use Sys::Syslog qw(:DEFAULT setlogsock); >> use Sys::Syslog qw(:standard :macros); >> use POSIX qw(setsid); >> use threads; >> use threads::shared; >> >> sub mail >> { >> print ("The mail thread\n"); >> return 0; >> } >> >> sub mf >> { >> print ("The mf thread\n"); >> return 0; >> } >> >> sub scan_files >> { >> while(1) >> { >> foreach (readdir(DIRHANDLE)) >> { >> if (!-d $_) >> { >> my $opt = >> choose_file_option("workq/",$_,"Delivered"); >> switch ($opt) >> { >> case "mail" >> { >> threads->new(\&mail, $_); > > threads->new(\&main, $_)->detach(); > >> } >> case "mf" >> { >> threads->new(\&mf, $_); > > threads->new(\&mf, $_)->detach(); > >> } >> else >> { >> syslog ('daemon|info', >> "Send mail. Failure detected"); >> } >> } >> } >> } >> sleep(60); >> } >> } >> >> sub daemon >> { >> chdir "/"; >> open STDIN, '/dev/null'; >> open STDOUT, '>/dev/null'; >> open STDERR, '>/dev/null'; >> fork && exit; >> setsid(); >> if (open PID, '>', '/var/run/program.pid') >> { >> print PID "$$\n"; >> close PID; >> } >> } >> >> setlogmask( LOG_MASK($Fax::Config::param{'log_info'}) ); >> openlog 'faxqr', 'pid', LOG_DAEMON; >> daemon(); >> >> $SIG{"TERM"} = \&notify_exit; >> >> scan_files(); >> >> ----------------------------------------------------------------------------- >> >> Any idea that there is a very high consumption of memory. >> >> -- >> A greeting, >> >> Javier. >> > Hi, I aply this changes, but now I obtein a fatal error: *** glibc detected *** double free or corruption (!prev): 0x08fea490 *** And the process is cancel :( Any idea? -- A greeting, Javier. http://www.nntp.perl.org/group/perl.ithreads/2008/06/msg1303.html Wed, 11 Jun 2008 07:59:01 +0000 Re: Memory usage by Jerry D. Hedden On Wed, Jun 11, 2008 at 9:26 AM, Javier Terceiro <correolista@gmail.com> wrote: > Hello list, > > I am designed a new software. I am using threads. The operation is > correct, but memory consumption is excessive. The system is using 100 > MB and when the software is working, 500 Mb. The memory is not > released, why? You need to either join the threads after they're done working, or else detach them. See below. > > The source code is: > > ----------------------------------------------------------------------------- > > #!/usr/bin/perl -w > > use strict; > use warnings; > use AnyData; > use Sys::Syslog; > use Sys::Syslog qw(:DEFAULT setlogsock); > use Sys::Syslog qw(:standard :macros); > use POSIX qw(setsid); > use threads; > use threads::shared; > > sub mail > { > print ("The mail thread\n"); > return 0; > } > > sub mf > { > print ("The mf thread\n"); > return 0; > } > > sub scan_files > { > while(1) > { > foreach (readdir(DIRHANDLE)) > { > if (!-d $_) > { > my $opt = > choose_file_option("workq/",$_,"Delivered"); > switch ($opt) > { > case "mail" > { > threads->new(\&mail, $_); threads->new(\&main, $_)->detach(); > } > case "mf" > { > threads->new(\&mf, $_); threads->new(\&mf, $_)->detach(); > } > else > { > syslog ('daemon|info', > "Send mail. Failure detected"); > } > } > } > } > sleep(60); > } > } > > sub daemon > { > chdir "/"; > open STDIN, '/dev/null'; > open STDOUT, '>/dev/null'; > open STDERR, '>/dev/null'; > fork && exit; > setsid(); > if (open PID, '>', '/var/run/program.pid') > { > print PID "$$\n"; > close PID; > } > } > > setlogmask( LOG_MASK($Fax::Config::param{'log_info'}) ); > openlog 'faxqr', 'pid', LOG_DAEMON; > daemon(); > > $SIG{"TERM"} = \&notify_exit; > > scan_files(); > > ----------------------------------------------------------------------------- > > Any idea that there is a very high consumption of memory. > > -- > A greeting, > > Javier. > http://www.nntp.perl.org/group/perl.ithreads/2008/06/msg1302.html Wed, 11 Jun 2008 06:32:32 +0000 Memory usage by Javier Terceiro Hello list, I am designed a new software. I am using threads. The operation is correct, but memory consumption is excessive. The system is using 100 MB and when the software is working, 500 Mb. The memory is not released, why? The source code is: ----------------------------------------------------------------------------- #!/usr/bin/perl -w use strict; use warnings; use AnyData; use Sys::Syslog; use Sys::Syslog qw(:DEFAULT setlogsock); use Sys::Syslog qw(:standard :macros); use POSIX qw(setsid); use threads; use threads::shared; sub mail { print ("The mail thread\n"); return 0; } sub mf { print ("The mf thread\n"); return 0; } sub scan_files { while(1) { foreach (readdir(DIRHANDLE)) { if (!-d $_) { my $opt = choose_file_option("workq/",$_,"Delivered"); switch ($opt) { case "mail" { threads->new(\&mail, $_); } case "mf" { threads->new(\&mf, $_); } else { syslog ('daemon|info', "Send mail. Failure detected"); } } } } sleep(60); } } sub daemon { chdir "/"; open STDIN, '/dev/null'; open STDOUT, '>/dev/null'; open STDERR, '>/dev/null'; fork && exit; setsid(); if (open PID, '>', '/var/run/program.pid') { print PID "$$\n"; close PID; } } setlogmask( LOG_MASK($Fax::Config::param{'log_info'}) ); openlog 'faxqr', 'pid', LOG_DAEMON; daemon(); $SIG{"TERM"} = \&notify_exit; scan_files(); ----------------------------------------------------------------------------- Any idea that there is a very high consumption of memory. -- A greeting, Javier. http://www.nntp.perl.org/group/perl.ithreads/2008/06/msg1301.html Wed, 11 Jun 2008 06:27:07 +0000 Re: thread hash bug? by Jerry D. Hedden Wong, Danny H. wrote: > I'm trying to created a hash of array inside a thread, but the hash > loses all its keys/values when leaving the function. Is there a bug in > using a hash inside a thread? Any ideas? 1. use strict; 2. use warnings; 3. Read 'perldoc threads::shared' http://www.nntp.perl.org/group/perl.ithreads/2008/06/msg1300.html Wed, 04 Jun 2008 06:15:17 +0000 thread hash bug? by Wong, Danny H. Hi GURUS, I'm trying to created a hash of array inside a thread, but the hash loses all its keys/values when leaving the function. Is there a bug in using a hash inside a thread? Any ideas? foreach my $i ( 1..3){ push (@BUILD_THREADS, threads->create(\&COPY, $i)); } foreach my $THREAD (@BUILD_THREADS) { my @DATA = $THREAD->join(); } sub COPY { my ($i) = @_; copy ( ${SOURCE}, "${DESTINATION}_${i}"); push @{$STORE_BUILD_RESULTS{"flintstones"}}, "${DESTINATION}_${i}"; while ( my ($KEY, $VALUE) = each (%STORE_BUILD_RESULTS)) { print "KEY, VALUE is $KEY, @$VALUE\n"; #I get correct print out. # $COMPLETED_BUILDS = STORE_BUILD_RESULTS } } while ( my ($KEY, $VALUE) = each (%STORE_BUILD_RESULTS)) #The hash is empty. Nothing gets printed out. { print "KEY, VALUE is $KEY, $VALUE\n"; # $COMPLETED_BUILDS = STORE_BUILD_RESULTS } - - - - - Cisco - - - - - This e-mail and any attachments may contain information which is confidential, proprietary, privileged or otherwise protected by law. The information is solely intended for the named addressee (or a person responsible for delivering it to the addressee). If you are not the intended recipient of this message, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer. http://www.nntp.perl.org/group/perl.ithreads/2008/06/msg1299.html Wed, 04 Jun 2008 02:08:54 +0000 Re: p5p summary: Improving threads::shared ? by Jerry D. Hedden Jerry D. Hedden wrote: > > perhaps we could just provide a function: > > $y = shared_clone($x); I uploaded threads::shared 1.21 which now has this functionality. http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1298.html Sat, 17 May 2008 04:59:27 +0000 Problem in build perl5.8.7 on OpenVMS using threads by ravi kumar Hi, I am getting the fallowing error while building the perl5.8.7 on OpenVMS I64 v.8.3 system using threads MMK 15-MAY-2008 16:40:40 15-MAY-2008 16:40:40 15-MAY-2008 16:40:40 CC/DECC/NOANSI_ALIAS /Include=[]/Standard=Relaxed_ANSI/Prefix=All/Obj=.obj/List/Machine/Show=Expan/float=ieee/ieee=denorm/Define=(PE RL_CORE,_LARGEFILE=1) MINIPERLMAIN.C int Perl_vms_decc_feature_get_index(aTHX_ const char *) ^ %CC-E-NOSEMI, Missing ";". at line number 846 in file PERL_BUILD_ROOT:[000000]vmsish.h;1 I32 unlnk (char*); ....^ %CC-E-PARNOIDENT, Missing identifier. at line number 3188 in file PERL_BUILD_ROOT:[000000]perl.h;1 struct group _grent_struct; ........................^ %CC-E-INCOMPMEM, The member "_grent_struct" has an incomplete type. at line number 623 in file PERL_BUILD_ROOT:[000000]reentr.h;1 PL_perl_destruct_level = 0; ........^ %CC-W-INCOMPDEREF, In this statement, "my_perl" is a pointer to an incomplete struct or union and should not be used as the left ope rand of a member dereference. at line number 93 in file PERL_BUILD_ROOT:[000000]miniperlmain.c;1 PL_exit_flags |= PERL_EXIT_DESTRUCT_END; ....^ %CC-W-INCOMPDEREF, In this statement, "my_perl" is a pointer to an incomplete struct or union and should not be used as the left ope rand of a member dereference. at line number 95 in file PERL_BUILD_ROOT:[000000]miniperlmain.c;1 %MMK-F-ERRUPD, error status %X10B91262 occurred when updating target MINIPERLMAIN.OBJ 15-MAY-2008 16:40:42 $ I have changed the configure.com file by changing the usethreads="T" and useithreads="Y" also i used configure option as fallows configure.com - "-Dunlink_all_versions" "-Duselargefiles" "-Dcf_email=SYSTEM" "-Dusethreads" "-de" I have built the same without enabling threads and it went fine. so could you help me out? -- Ravikumar karnati http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1297.html Fri, 16 May 2008 04:22:37 +0000 Re: p5p summary: Improving threads::shared ? by Dean Arnold Jerry D. Hedden wrote: >> perhaps we could just provide a function: >> >> $y = shared_clone($x); > > What name should this function have? > > shared_clone() > clone() > shared_copy() > copy() > make_shared() > > Or something else? > I'd vote against clone() or copy(), as they're too general. Otherwise, I personally have no opinion, tho clone has come to mean this sort of deepcopy (see the various Clone modules), so I suppose shared_clone() may make sense. - Dean http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1296.html Thu, 08 May 2008 08:54:08 +0000 Re: p5p summary: Improving threads::shared ? by Jerry D. Hedden > perhaps we could just provide a function: > > $y = shared_clone($x); What name should this function have? shared_clone() clone() shared_copy() copy() make_shared() Or something else? http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1295.html Thu, 08 May 2008 07:27:09 +0000 Re: p5p summary: Improving threads::shared ? by Jerry D. Hedden > For any existing apps (eg, Data::Dumper) that want to > deal with it, they could always fallback to detecting > something as shared and saving its id (ie, the shared > interpretter version's address) to detect cycles. Not > pretty, but effective. I looked into fixing Data::Dumper for this. The circular reference checking is done both in Perl (for the pure-Perl version) and in XS (the usual version). It'll take me awhile to figure out how to do it in XS. http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1294.html Thu, 08 May 2008 07:17:28 +0000 Re: p5p summary: Improving threads::shared ? by Jerry D. Hedden > > It means that something like this would DWIM: > > > > my $x : shared; > > $x = [ { 'complex' => 'aggregate' }, [ qw/ currently not sharable / ] > ]; > > > > > > Then we just need a tie-in with the assignment operator. Or > > perhaps we could just provide a function: > > > > $y = shared_clone($x); > > > > > > Would an assignment op overload work ? > I.e., if the LHS was already shared(), then the = overload > would do the deepcopy ? Or would that break the > XS tie/magic side of the code ? Oops. Shared variables aren't objects. So I don't think we can use 'overload' on '='. Is that correct? http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1293.html Thu, 08 May 2008 07:13:11 +0000 Re: p5p summary: Improving threads::shared ? by Jerry D. Hedden On Mon, May 5, 2008 at 1:53 PM, Jerry D. Hedden <jdhedden@cpan.org> wrote: > > My sample worked OK *after* I removed the Data::Dumper > > print; apparently, it has some issues w/ circular > > references as well (I tried setting Deepcopy and > > Purity, to no avail); I suspect its due to the fact > > that references to shared elements don't have the same > > stringified value, so it doesn't actually look like > > a circular ref. > > Oh, yes. That's part of what I reported here: > > http://rt.perl.org/rt3/Public/Bug/Display.html?id=37946 > > The following highlights the problem: > > > use strict; > use warnings; > > use threads; > use threads::shared; > > my $x; > $x = \$x; > share($x); > > print("Look at \$x:\n"); > print($x, "\n"); > print($$x, "\n"); > print($$$x, "\n"); > print($$$$x, "\n"); > print($$$$$x, "\n"); > print($$$$$$x, "\n"); > > my @q :shared = ($x); > > my $y = $q[0]; > > print("\nFirst look at \$y:\n"); > print($y, "\n"); > print($$y, "\n"); > print($$$y, "\n"); > print($$$$y, "\n"); > print($$$$$y, "\n"); > print($$$$$$y, "\n"); > > print("\nSecond look at \$y:\n"); > print($y, "\n"); > print($$y, "\n"); > print($$$y, "\n"); > print($$$$y, "\n"); > print($$$$$y, "\n"); > print($$$$$$y, "\n"); > > This outputs: > > Look at $x: > REF(0x144f8f0) > REF(0x144f8f0) > REF(0x144f8f0) > REF(0x144f8f0) > REF(0x144f8f0) > REF(0x144f8f0) > > First look at $y: > SCALAR(0x1423c70) > SCALAR(0x1423ad8) > SCALAR(0x14bd968) > SCALAR(0x14bd980) > SCALAR(0x14bd998) > SCALAR(0x14bd9b0) > > Second look at $y: > REF(0x1423c70) > REF(0x1423ad8) > REF(0x14bd968) > REF(0x14bd980) > REF(0x14bd998) > SCALAR(0x14bd9b0) > > Seems to me that this is a bug. It shows that > threads::shared isn't detecting that it's dealing with a > circular reference. With each level that the app traverses, > threads::shared "unrolls" another version of the shared > variable. I just posted a patch to blead that fixes this. If the patch passes muster, I'll release an update for threads::shared to CPAN. http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1292.html Wed, 07 May 2008 11:57:58 +0000 Re: p5p summary: Improving threads::shared ? by Dean Arnold Dean Arnold wrote: > Jerry D. Hedden wrote: >> Conceptually, I think it would require keeping a weak >> reference of each private SV associated with a shared SV. >> This would need to be done on a per-thread basis. Then when >> a thread tries to reference a shared SV, a lookup would be >> made to see if a private SV already exists (and still >> exists) for that thread. If so, that is returned (and the >> ref count of the weak ref is incremented?). If not, a new >> private SV is created, and a weak ref to it is appropriately >> stored. >> >> Is this a logical approach? If so, is it doable? >> > > For threads::shared, the only solution I can think of is > adding a fieldhash to the thread-private my_ctx structure > keyed by the address of the referent variable's > shared version. Then each attempt to create a new proxy > would lookup any existing persistent proxy for the shared > SV, and return it if found (currently, each new reference > to a shared variable creates a new proxy, which is what causes > this mess). > I forgot one not-so-minor detail: clone processing would need to be updated to detect and replace the fieldhash'd proxies, and update each SV that invoked the magic dup() method - Dean http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1291.html Tue, 06 May 2008 14:00:47 +0000 Re: p5p summary: Improving threads::shared ? by Dean Arnold Jerry D. Hedden wrote: >> Is this a logical approach? If so, is it doable? > > If circular references can't be fully supported in > threads::shared, then I need to document this in its POD and > in Thread::Queue's POD, too. Do you agree? > I don't know that they can't be "supported"; but they do need to be explained. I also don't see how it effects T::Q: since you always skip over anything thats already shared, existing circular refs aren't an issue. Its only detecting and creating new shared circular refs that causes a problem, and thats fixed w/ the fieldhash lookup (which is keyed on the private address, not the shared version). For any existing apps (eg, Data::Dumper) that want to deal with it, they could always fallback to detecting something as shared and saving its id (ie, the shared interpretter version's address) to detect cycles. Not pretty, but effective. - Dean http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1290.html Mon, 05 May 2008 12:42:58 +0000 Re: p5p summary: Improving threads::shared ? by Dean Arnold Jerry D. Hedden wrote: > erry D. Hedden wrote: >> Look at $x: >> REF(0x144f8f0) >> REF(0x144f8f0) >> REF(0x144f8f0) >> REF(0x144f8f0) >> REF(0x144f8f0) >> REF(0x144f8f0) >> >> First look at $y: >> SCALAR(0x1423c70) >> SCALAR(0x1423ad8) >> SCALAR(0x14bd968) >> SCALAR(0x14bd980) >> SCALAR(0x14bd998) >> SCALAR(0x14bd9b0) >> >> Second look at $y: >> REF(0x1423c70) >> REF(0x1423ad8) >> REF(0x14bd968) >> REF(0x14bd980) >> REF(0x14bd998) >> SCALAR(0x14bd9b0) >> >> Seems to me that this is a bug. It shows that >> threads::shared isn't detecting that it's dealing with a >> circular reference. With each level that the app traverses, >> threads::shared "unrolls" another version of the shared >> variable. > >> The more I think about it, the more I'm convinced this needs >> to be fixed. The original variable $x is a circular >> reference, and it's finiteness is detectable. The copy made >> to $y has become an infinitely deep reference with lazy >> evaluation. > > I've been looking at how to fix this, but know understanding > of the internals of threads::shared is weak. > > Conceptually, I think it would require keeping a weak > reference of each private SV associated with a shared SV. > This would need to be done on a per-thread basis. Then when > a thread tries to reference a shared SV, a lookup would be > made to see if a private SV already exists (and still > exists) for that thread. If so, that is returned (and the > ref count of the weak ref is incremented?). If not, a new > private SV is created, and a weak ref to it is appropriately > stored. > > Is this a logical approach? If so, is it doable? > By coincidence, I've been doing related work on Thread::Sociable's STM implementation. It has to keep a "shadow" proxy of each variable in order to avoid the same referencing issue (otherwise, it could create multiple different transactional versions of the same referenced variable) For threads::shared, the only solution I can think of is adding a fieldhash to the thread-private my_ctx structure keyed by the address of the referent variable's shared version. Then each attempt to create a new proxy would lookup any existing persistent proxy for the shared SV, and return it if found (currently, each new reference to a shared variable creates a new proxy, which is what causes this mess). It may create refcounting issues; the private proxy would need to be refcounted every time the shared variable was refcounted in the same thread. (Alternately, the thread could refcount only the proxy, and then decrement the shared versions refcount only when the private proxy's refcount dropped to zero) I also don't know how it would effect taking ref's of shared array/hash elements. And this will probably slow things down even further. - Dean http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1289.html Mon, 05 May 2008 12:36:19 +0000 Re: p5p summary: Improving threads::shared ? by Jerry D. Hedden > Is this a logical approach? If so, is it doable? If circular references can't be fully supported in threads::shared, then I need to document this in its POD and in Thread::Queue's POD, too. Do you agree? http://www.nntp.perl.org/group/perl.ithreads/2008/05/msg1288.html Mon, 05 May 2008 12:14:10 +0000