perl.ipc.dirqueue

perl.ipc.dirqueue http://www.nntp.perl.org/group/perl.ipc.dirqueue/ ... Copyright 1998-2008 perl.org Sun, 20 Jul 2008 11:05:54 +0000 ask@perl.org released version 1.0 of IPC::DirQueue by jm Hi -- Version 1.0 of IPC::DirQueue is now released, and should be showing up on CPAN rsn. Changes since 0.09: - worker_still_working() is broken; it always claimed that the worker was no longer running. Fix thanks to Brian Duggan <bduggan at mutatu dot org> - use buffered print instead of unbuffered syswrite() when creating data files using enqueue_sub() - fix code to match POD documentation regarding the range of chars permitted in metadata during enqueueing, thanks to Brian Grossman for report - update POD docs: visit_all_jobs() does not return a value. thanks to Simon on AnnoCPAN: http://annocpan.org/%7eJMASON/IPC-DirQueue-0.09/lib/IPC/DirQueue.pm#note_1570 [the 1.0 version bump is just because -- no good reason. ;) this isn't really more stable than any other version!] --j. http://www.nntp.perl.org/group/perl.ipc.dirqueue/2008/04/msg29.html Fri, 18 Apr 2008 08:45:05 +0000 released version 0.09 of IPC::DirQueue by jm Hi -- Version 0.09 of IPC::DirQueue is now released, and should be showing up on CPAN rsn. Changes since 0.08: ------------------------------------------------------------------------ r8418 | jmason | 2006-12-21 19:00:03 +0000 (Thu, 21 Dec 2006) | 1 line add IPC::DirQueue::Job::get_data() API, to access queued job data directly as a string instead of as a filename, thanks to Collin Winter <collinw at gmail.com> ------------------------------------------------------------------------ r8379 | jmason | 2006-12-11 20:32:13 +0000 (Mon, 11 Dec 2006) | 1 line fix documentation; it was unclear for beginners as to how job data got into and out of IPC::DirQueue, and that reading/writing files is required ------------------------------------------------------------------------ r8103 | jmason | 2006-11-24 14:27:37 +0000 (Fri, 24 Nov 2006) | 1 line fix platform-specific tests: must use 'use constant' in the conditional variables, otherwise the tests appear to be failing on win32 platforms when in fact we're just trying to skip them (see http://www.nntp.perl.org/group/perl.cpan.testers/377405 for example) (by the way, IPC::DirQueue users might be interested in this account of how qpsmtpd and IPC::DirQueue powers the traps.SpamAssassin.org spamtraps: http://taint.org/2007/04/17/132339a.html ) --j. http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/04/msg28.html Fri, 27 Apr 2007 13:40:48 +0000 Re: cancelling queued items by jm Brian Duggan writes: > On Wednesday, March 21, Justin Mason wrote: > > It sounds pretty good -- just one thing -- how is a caller supposed to get > > the datapath? Presumably they don't construct it themselves... > > >>From the $job object (obtainable by sending a callback > to visit_all_jobs()). Though actually, it looks like using > $job->{pathqueue} instead of the datapath might be easier, > so that pickup_queued_job() doesn't have to read the > queuefile (to find out the datapath). > > > that's ok, that suggestion is a one-liner. Having said that -- a test case > > "t" script would rock though :) > > Okay, I've attached one. Cool. That makes the idea pretty clear, too. I had to move a little stuff around to make it more efficient, but it works fine; I also copied the test script for fanout and ordered queue modes, too. Checked in: : jm 516...; svn commit -m "added support to pickup a queued item by its data path, thanks to Brian Duggan <bduggan at matatu.org>" Sending MANIFEST Sending lib/IPC/DirQueue.pm Adding t/60pickup_path_basic.t Adding t/60pickup_path_fanout.t Adding t/60pickup_path_ordered.t Transmitting file data ..... Committed revision 9297. > thanks! > Brian > > ps On a separate note, I noticed that read_control_file() is always > called with two arguments, but the second argument is ignored. Is > this perhaps an alternative to saying 'local *IN' in read_control_file()? oops! bug ;) Thanks for spotting this. fixed: : jm 745...; svn commit -m "read_control_file() was always opening the control file for read twice, redundantly; fixed" lib/IPC/DirQueue.pm Sending lib/IPC/DirQueue.pm Transmitting file data . Committed revision 9296. --j. http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/03/msg27.html Thu, 22 Mar 2007 12:32:29 +0000 Re: cancelling queued items by Brian Duggan On Wednesday, March 21, Justin Mason wrote: > It sounds pretty good -- just one thing -- how is a caller supposed to get > the datapath? Presumably they don't construct it themselves... From the $job object (obtainable by sending a callback to visit_all_jobs()). Though actually, it looks like using $job->{pathqueue} instead of the datapath might be easier, so that pickup_queued_job() doesn't have to read the queuefile (to find out the datapath). > that's ok, that suggestion is a one-liner. Having said that -- a test case > "t" script would rock though :) Okay, I've attached one. thanks! Brian ps On a separate note, I noticed that read_control_file() is always called with two arguments, but the second argument is ignored. Is this perhaps an alternative to saying 'local *IN' in read_control_file()? http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/03/msg26.html Wed, 21 Mar 2007 17:35:12 +0000 Re: cancelling queued items by jm Brian Duggan writes: > Hello, > > I'm using IPC::DirQueue -- very nice module, thanks. > > I wanted the ability to cancel not-yet-active items from > anywhere in the queue (a la lprm), and was trying > to think of a safe way to do it. It occurred to me > that one easy modification to IPC::DirQueue that would > allow this would be to have pickup_queued_job() take > an optional parameter which is the name of the data path, > and then just add something like > > next if exists($args{datapath}) && > $nextfile ne $args{datapath}; > > to the while loop. > > Any chance this could be incorporated? Or is there > already another safe way to cancel an item? It sounds pretty good -- just one thing -- how is a caller supposed to get the datapath? Presumably they don't construct it themselves... > thanks > Brian > > (p.s. happy to send a patch if you prefer) that's ok, that suggestion is a one-liner. Having said that -- a test case "t" script would rock though :) --j. http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/03/msg25.html Wed, 21 Mar 2007 08:50:52 +0000 Re: usage question by Thomas J Pinkl On Mon, Mar 05, 2007 at 11:28:03AM -0500, Thomas J Pinkl wrote: > Justin, here's a patch which adds an optional "readonly" parameter > to visit_all_jobs(). If the parameter isn't passed, then it defaults > to true, thus maintaining backward compatibility. The patch > includes changes to the POD as well. It's against the current SVN > source. Perhaps I whipped this up too quickly. The read-only flag works, as far as it goes, but jobs are not considered "active" when read-only is false. If I understand the code correctly, then I'll also have to do something to make the job "active" from within visit_all_jobs(). Is that correct? -- Thomas J. Pinkl | T: 215-442-9300 Senior Systems Architect | 800-444-1427 Health Business Systems, Inc | F: 215-442-7555 An SXC Company | 738 Louis Drive | http://www.hbsrx.com/ Warminster, PA 18974 | http://www.sxc.com/ http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/03/msg24.html Mon, 05 Mar 2007 09:06:57 +0000 Re: usage question by Thomas J Pinkl On Mon, Mar 05, 2007 at 11:24:07AM +0000, Justin Mason wrote: > Alternatively, I'd be happy to apply a patch that implements > a version of visit_all_jobs() which allows writes somehow, or > a way to access a job returned by visit_all_jobs() and render > it writable. Justin, here's a patch which adds an optional "readonly" parameter to visit_all_jobs(). If the parameter isn't passed, then it defaults to true, thus maintaining backward compatibility. The patch includes changes to the POD as well. It's against the current SVN source. -- Thomas J. Pinkl | T: 215-442-9300 Senior Systems Architect | 800-444-1427 Health Business Systems, Inc | F: 215-442-7555 An SXC Company | 738 Louis Drive | http://www.hbsrx.com/ Warminster, PA 18974 | http://www.sxc.com/ http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/03/msg23.html Mon, 05 Mar 2007 08:28:23 +0000 Re: usage question by jm hi Thomas -- an interesting use-case I hadn't considered! ;) The problem is that when the queue is accessed in ordered mode, it's simply a lexical sort by filename; and when accessed in unordered mode, it's simply the OS-based ordering rather than in any way "random". What about re-queueing the deferred jobs? ie. my $dq = IPC::DirQueue->new( \%dq_opts ); while (1) { my $job = $dq->pickup_queued_job(); last if (! $job); my $rc = &process_job( $job ); if (!$rc) { $job->enqueue_file($job->get_data_path()); } $job->finish(); } in other words, if a job is to be deferred, take it off the queue and re-queue it under a new name. Alternatively, I'd be happy to apply a patch that implements a version of visit_all_jobs() which allows writes somehow, or a way to access a job returned by visit_all_jobs() and render it writable. --j. Thomas J Pinkl writes: > I'm attempting to use IPC::DirQueue 0.08 for a project where messages > are queued for later processing. However, when dequeuing these > messages, it will be possible that a message cannot be processed > immediately, but could be processed later. > > I thought I'd be able to use IPC::DirQueue like this: > > my $dq = IPC::DirQueue->new( \%dq_opts ); > while (1) { > my $job = $dq->pickup_queued_job(); > last if (! $job); > > my $rc = &process_job( $job ); > if ($rc) { > $job->finish(); > } else { > $job->return_to_queue(); > } > } > > But if, say, the first job is returned to the queue, then a subsequent > call to pickup_queued_job() returns that same job. Turning the loop > above into an infinite loop (absent an external event which suddenly > causes process_job() to return true). > > I've read through the IPC::DirQueue and IPC::DirQueue::Job pod pages, > but I'm not seeing any way to iterate through the queue the way that I > need. The closest method I can see is visit_all_jobs(), but that only > provides read-only Job objects. I suppose I could play with the > read-only flag within the callback function, but that doesn't seem > right. > > Does anyone have any suggestions? > > -- > Thomas J. Pinkl | T: 215-442-9300 > Senior Systems Architect | 800-444-1427 > Health Business Systems, Inc | F: 215-442-7555 > An SXC Company | > 738 Louis Drive | http://www.hbsrx.com/ > Warminster, PA 18974 | http://www.sxc.com/ http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/03/msg22.html Mon, 05 Mar 2007 03:25:14 +0000 usage question by Thomas J Pinkl I'm attempting to use IPC::DirQueue 0.08 for a project where messages are queued for later processing. However, when dequeuing these messages, it will be possible that a message cannot be processed immediately, but could be processed later. I thought I'd be able to use IPC::DirQueue like this: my $dq = IPC::DirQueue->new( \%dq_opts ); while (1) { my $job = $dq->pickup_queued_job(); last if (! $job); my $rc = &process_job( $job ); if ($rc) { $job->finish(); } else { $job->return_to_queue(); } } But if, say, the first job is returned to the queue, then a subsequent call to pickup_queued_job() returns that same job. Turning the loop above into an infinite loop (absent an external event which suddenly causes process_job() to return true). I've read through the IPC::DirQueue and IPC::DirQueue::Job pod pages, but I'm not seeing any way to iterate through the queue the way that I need. The closest method I can see is visit_all_jobs(), but that only provides read-only Job objects. I suppose I could play with the read-only flag within the callback function, but that doesn't seem right. Does anyone have any suggestions? -- Thomas J. Pinkl | T: 215-442-9300 Senior Systems Architect | 800-444-1427 Health Business Systems, Inc | F: 215-442-7555 An SXC Company | 738 Louis Drive | http://www.hbsrx.com/ Warminster, PA 18974 | http://www.sxc.com/ http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/03/msg21.html Sat, 03 Mar 2007 08:37:09 +0000 Re: getting: Illegal octal digit '8' by Thomas J Pinkl On Tue, Jan 16, 2007 at 07:04:44PM +0000, Justin Mason wrote: > Are you providing values for data_file_mode or queue_file_mode in > the constructor? Yes, both are being set. > It's worth noting that they should be formatted as octal-formatted > *strings*, not numbers. e.g. '0666', not 0666. Could that be it? Ah. I'll try that. Thank you. -- Thomas J. Pinkl | T: 215-442-9300 Senior Systems Architect | 800-444-1427 Health Business Systems, Inc | F: 215-442-7555 An SXC Company | 738 Louis Drive | http://www.hbsrx.com/ Warminster, PA 18974 | http://www.sxc.com/ http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/01/msg20.html Tue, 16 Jan 2007 11:23:32 +0000 Re: getting: Illegal octal digit '8' by jm Thomas J Pinkl writes: > I've just started using IPC::DirQueue 0.08. Thanks to the author(s) > for a module that appears to deliver exactly what I need. You're welcome! ;) > I received these warnings: > > Illegal octal digit '8' ignored at \ > /usr/lib/perl5/site_perl/5.6.1/IPC/DirQueue.pm line 146. > Illegal octal digit '8' ignored at \ > /usr/lib/perl5/site_perl/5.6.1/IPC/DirQueue.pm line 148. > > which corresponds to this section of code: > > $self->{data_file_mode} ||= '0666'; > $self->{data_file_mode} = oct ($self->{data_file_mode}); > $self->{queue_file_mode} ||= '0666'; > $self->{queue_file_mode} = oct ($self->{queue_file_mode}); > > in sub new(). Lines 146 and 148 are the ones that use 'oct'. Are you providing values for data_file_mode or queue_file_mode in the constructor? It's worth noting that they should be formatted as octal-formatted *strings*, not numbers. e.g. '0666', not 0666. Could that be it? --j. > The fix would appear to be: > > 146: $self->{data_file_mode} = oct sprintf "0%o",$self->{data_file_mode}; > > 148: $self->{queue_file_mode} = oct sprintf "0%o",$self->{queue_file_mode}; > > Can anyone else confirm this? > > -- > Thomas J. Pinkl | T: 215-442-9300 > Senior Systems Architect | 800-444-1427 > Health Business Systems, Inc | F: 215-442-7555 > An SXC Company | > 738 Louis Drive | http://www.hbsrx.com/ > Warminster, PA 18974 | http://www.sxc.com/ http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/01/msg19.html Tue, 16 Jan 2007 11:05:15 +0000 getting: Illegal octal digit '8' by Thomas J Pinkl I've just started using IPC::DirQueue 0.08. Thanks to the author(s) for a module that appears to deliver exactly what I need. I received these warnings: Illegal octal digit '8' ignored at \ /usr/lib/perl5/site_perl/5.6.1/IPC/DirQueue.pm line 146. Illegal octal digit '8' ignored at \ /usr/lib/perl5/site_perl/5.6.1/IPC/DirQueue.pm line 148. which corresponds to this section of code: $self->{data_file_mode} ||= '0666'; $self->{data_file_mode} = oct ($self->{data_file_mode}); $self->{queue_file_mode} ||= '0666'; $self->{queue_file_mode} = oct ($self->{queue_file_mode}); in sub new(). Lines 146 and 148 are the ones that use 'oct'. The fix would appear to be: 146: $self->{data_file_mode} = oct sprintf "0%o",$self->{data_file_mode}; 148: $self->{queue_file_mode} = oct sprintf "0%o",$self->{queue_file_mode}; Can anyone else confirm this? -- Thomas J. Pinkl | T: 215-442-9300 Senior Systems Architect | 800-444-1427 Health Business Systems, Inc | F: 215-442-7555 An SXC Company | 738 Louis Drive | http://www.hbsrx.com/ Warminster, PA 18974 | http://www.sxc.com/ http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/01/msg18.html Tue, 16 Jan 2007 10:41:08 +0000 Re: [PATCH] Advanced tagging and filtering by jm hey, btw -- something's up with the latest version, at least; when I apply the patches and make test, it hangs in t/10enq_string as soon as pickup_queued_job() is called. SVN trunk works fine, as does the first patch, but as soon as the second patch is applied this happens. Here's the current rev of the combined patch against SVN: http://taint.org/x/2007/filter.patch That contains your two patches, my documentation changes, and a "warn" in the test script to illustrate this hang. --j. Malte S. Stretz writes: > On Wednesday 03 January 2007 16:38 CET Justin Mason wrote: > > backwards compatibility is not a problem here, so go ahead and > > add the tag in addition to the hash. > > > > I think the best location is just before the hash, e.g. > > > > 50.20040909232529941258.TAG.HASH[.PID.RAND] > > > > (vs the simple > > > > 50.20040909232529941258.HASH[.PID.RAND] > > > > when tags are not in use.) > > > > > Actually, when people start to filter the files based on their name, > > > the format must not change anymore in future. Maybe the filter should > > > apply to the tag only... > > > > Yes, I think that's a good idea... people shouldn't have to worry about > > the rest of the filename changing. > > Ok, attached is a patch on top of the last one which changes the code as you > described it above. Additionally there's a revamped test. > > While I was hunting another totally stupid bug, I found out that I actually > forgot to add the $filter in the second call to pickup_job in wait_for_job. > Gave some nasty surprises. And visit_all_jobs now supports $filter, too. > > The syntax for $filter has changed a bit: If it is a string, it must be > equal the tag, if it is a RE created with qr// it is matched as a RE. > > Can you update the QUEUE DIRECTORY STRUCTURE section for me? Your English > is a lot better than mine :) > > Cheers, > Malte > part 3 text/x-diff 7472 > --- DirQueue.pm.patch1 2006-12-30 01:27:31.000000000 +0100 > +++ DirQueue.pm 2007-01-03 21:08:17.000000000 +0100 > @@ -173,8 +173,10 @@ > $self->{ordered} = 1; > } > > - $self->{tag} ||= hash_string_to_filename($self->gethostname().$$); > - $self->{tag_sub} ||= sub { return $self->{tag}; }; > + $self->{hash} ||= hash_string_to_filename($self->gethostname().$$); > + if (defined $self->{tag}) { > + $self->{tag_sub} ||= sub { return $self->{tag}; }; > + } > $self->{tag_max_length} ||= 128; > if (!defined $self->{tag_warn}) { > $self->{tag_warn} = 1; > @@ -456,9 +458,10 @@ > > Pick up the next job in the queue, so that it can be processed. > > -The parameter C<$filter> can be used to specify a regular expression which > -is matched against the queued filename. All files which don't match will be > -skipped. > +The parameter C<$filter> can be used to specify either a string or a regular > +expression (with qr//) which is compared (the the first case) or matched > +(in the latter case) against the tag part of the queued filename. All files > +which don't match will be skipped. > > If no job is available for processing, either because the queue is > empty or because other worker processes are already working on > @@ -709,7 +712,7 @@ > while (time == $qdirlaststat) { > Time::HiRes::usleep ($pollintvl); > dbg "wait_for_queued_job: spinning until time != stat $qdirlaststat"; > - my $job = $self->pickup_queued_job(); > + my $job = $self->pickup_queued_job($filter); > if ($job) { return $job; } > } > > @@ -739,7 +742,7 @@ > > ########################################################################### > > -=item $job = $dq->visit_all_jobs($visitor, $visitcontext); > +=item $job = $dq->visit_all_jobs($visitor, $visitcontext, $filter); > > Visit all the jobs in the queue, in a read-only mode. Used to list > the entire queue. > @@ -759,15 +762,17 @@ > 'active_host': the hostname on which the job is active > 'active_pid': the process ID of the process which picked up the job > > +The jobs can be filtered with C<$filter> as in C<pickup_queued_job()>. > + > =cut > > sub visit_all_jobs { > - my ($self, $visitor, $visitcontext) = @_; > + my ($self, $visitor, $visitcontext, $filter) = @_; > > my $pathqueuedir = $self->q_subdir('queue'); > my $pathactivedir = $self->q_subdir('active'); > > - my $iter = $self->queue_iter_start($pathqueuedir); > + my $iter = $self->queue_iter_start($pathqueuedir, $filter); > > my $nextfile; > while (1) { > @@ -1118,30 +1123,39 @@ > > my @gmt = gmtime ($job->{time_submitted_secs}); > > - # NN.20040718140300MMMM.tag[.rand] > + # NN.20040718140300MMMM[.tag].hash[.rand] > # > # NN = priority, default 50 > # MMMM = microseconds from Time::HiRes::gettimeofday() > - # tag = some base64-ish string, default hash(hostname.$$) > + # hash = hash(hostname.$$) > + # tag = some base64-ish string > # hostname = current hostname > > - my $base = sprintf ("%02d.%04d%02d%02d%02d%02d%02d%06d.", > + my $file = sprintf ("%02d.%04d%02d%02d%02d%02d%02d%06d", > $job->{pri}, > $gmt[5]+1900, $gmt[4]+1, $gmt[3], $gmt[2], $gmt[1], $gmt[0], > $job->{time_submitted_msecs}); > > + # add tag (including leading dot, only if wanted) and hash > + $file .= $self->get_q_filename_tag($job).".".$self->{hash}; > + > # normally, this isn't used. but if there's a collision, > # all retries after that will do this; in this case, the > # extra anti-collision stuff is useful > - my $extra = $addextra ? ".".$$.".".$self->get_random_int() : ""; > + if ($addextra) { > + $file .= ".".$$.".".$self->get_random_int(); > + } > > - return $base.$self->get_q_filename_tag($job, $base, $extra).$extra; > + return $file; > } > > sub get_q_filename_tag { > - my($self, $job, $base, $extra) = @_; > - # create a (new?) tag > - my $str = $self->{tag_sub}->($job, $base, $extra) || ''; > + my($self, $job) = @_; > + # return an empty string if there's nothing to do > + return '' unless defined $self->{tag_sub}; > + # create a (new?) tag, possibly empty > + my $str = $self->{tag_sub}->($job); > + return '' unless defined $str; > # weed out all dangerous chars > my $tag = filter_unsafe_chars($str); > # limit the length > @@ -1150,7 +1164,7 @@ > if ($self->{tag_warn} && $tag ne $str) { > warn "IPC::DirQueue: the tag was filtered\n"; > } > - return $tag; > + return ".$tag"; > } > > sub hash_string_to_filename { > @@ -1242,23 +1256,37 @@ > > sub queue_iter_start { > my ($self, $pathqueuedir, $filter, $type) = @_; > - > - $filter ||= qr/^/; > + > + $filter = qr// unless defined $filter; > dbg ("queue iter: filter $filter in $pathqueuedir"); > unless (ref $filter eq 'CODE') { > # we need to copy $filter here else the closure will get annoyed > - my $f = $filter; > + my $re = $filter; > + unless (ref ($re) eq 'Regexp') { > + $re = quotemeta($re); > + $re = qr/^${re}$/; > + } > $filter = sub { > - if (wantarray) { # grep is picky about list context > - return grep { /^\d/ && /$f/ } @_; > - } > - else { > - $_ = shift; > - return unless defined; > - return unless /^\d/; > - return unless /$f/; > - return $_; > + # we can't use grep because it is picky about > + # list context and this sub can be called for > + # single files and grep would return the count > + # in those cases > + my @r; > + while (@_) { > + my $f = shift; > + next unless defined $f; > + next unless $f =~ /^\d/; > + # we've got to go split here to make the ^ > + # and $ anchors work > + my @f = split(/\./, $f); > + # does this file have a tag (ie. even number > + # of elements)? > + next if scalar (@f) % 2; > + # apply the tag > + next unless $f[2] =~ $re; > + push(@r, $f); > } > + return wantarray ? @r : $r[0]; > }; > } > > @@ -1358,11 +1386,11 @@ > @files = sort $iter->{filter}->(readdir(DIR)); > closedir DIR; > } > - > + > if (scalar @files <= 0) { > return if $self->queuedir_is_bad($iter->{dir}); > } > - > + > $iter->{type} = 'files'; > $iter->{files} = \@files; > return $iter; > @@ -1555,8 +1583,7 @@ > > The filename format is as follows: > > - 50.20040909232529941258.TAG[.PID.RAND] > - | base |tag | extra | > + 50.20040909232529941258[.TAG].HASH[.PID.RAND] > > The first two digits (C<50>) are the priority of the job. Lower priority > numbers are run first. C<20040909232529> is the current date and time when the > @@ -1568,10 +1595,8 @@ > possible to set this manually with the C<tag> parameter of the constructor, or even > to have it generated dynamically by the routine set with C<tag_sub>. > > -The sub routine is called with the parameters C<$job>, C<$base> and C<$extra>. The > -first is the job currently enqueued and thus allows access to the metadata. The > -latter two parameters correspond to the already known parts of the filename as > -shown above (including the dots). > +The sub routine is called with a single parameter C<$job>. This is the job > +currently enqueued and thus allows access to the metadata. > > Note that only characters from the set [A-Za-z0-9+_] are allowed and the length > of the string is limited to 128 characters. If any of these restrictions are http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/01/msg17.html Thu, 04 Jan 2007 05:34:52 +0000 Re: [PATCH] Advanced tagging and filtering by Malte S. Stretz On Wednesday 03 January 2007 16:38 CET Justin Mason wrote: > backwards compatibility is not a problem here, so go ahead and > add the tag in addition to the hash. > > I think the best location is just before the hash, e.g. > > 50.20040909232529941258.TAG.HASH[.PID.RAND] > > (vs the simple > > 50.20040909232529941258.HASH[.PID.RAND] > > when tags are not in use.) > > > Actually, when people start to filter the files based on their name, > > the format must not change anymore in future. Maybe the filter should > > apply to the tag only... > > Yes, I think that's a good idea... people shouldn't have to worry about > the rest of the filename changing. Ok, attached is a patch on top of the last one which changes the code as you described it above. Additionally there's a revamped test. While I was hunting another totally stupid bug, I found out that I actually forgot to add the $filter in the second call to pickup_job in wait_for_job. Gave some nasty surprises. And visit_all_jobs now supports $filter, too. The syntax for $filter has changed a bit: If it is a string, it must be equal the tag, if it is a RE created with qr// it is matched as a RE. Can you update the QUEUE DIRECTORY STRUCTURE section for me? Your English is a lot better than mine :) Cheers, Malte http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/01/msg16.html Wed, 03 Jan 2007 12:16:38 +0000 Re: [PATCH] Advanced tagging and filtering by jm Malte S. Stretz writes: > On Wednesday 03 January 2007 13:02 CET Justin Mason wrote: > > hi Malte! > > > > Thanks for this, it looks very useful indeed. One tweak, though: > > > b) Makes the TAG part of the filename configurable. Ah, yeah, TAG was > > > called HASH before :) I added the parameters tag (static tag), tag_sub > > > (callback to generate a dynamic tag) and two more (tag_max_length and > > > tag_warn, cf. inline doc). > > > > I would prefer to *add* the TAG part *as well as* the HASH stuff. > > The hash is useful, since it avoids collisions -- TAG doesn't make > > that guarantee (nor should it have to). > > > > That should be OK, right? > > Yeah, that was my original plan. But I wanted to keep the filename format > backwards compatible. Not sure if thats really needed, I also wasn't sure > if I should add the tag in the front, the end or in between. In the end I > went for compatibility and assumed that in most cases the tags will still > be quite unique -- at least in the producer/consumer case where you've got > most probably some kind of pid in there they will be at least as unique as > the current hostname/pid based ones. And if not, we've got the timestamp > plus the additional really random part at the end :) backwards compatibility is not a problem here, so go ahead and add the tag in addition to the hash. I think the best location is just before the hash, e.g. 50.20040909232529941258.TAG.HASH[.PID.RAND] (vs the simple 50.20040909232529941258.HASH[.PID.RAND] when tags are not in use.) > Actually, when people start to filter the files based on their name, the > format must not change anymore in future. Maybe the filter should apply to > the tag only... Yes, I think that's a good idea... people shouldn't have to worry about the rest of the filename changing. --j. http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/01/msg15.html Wed, 03 Jan 2007 07:39:31 +0000 Re: [PATCH] Advanced tagging and filtering by Malte S. Stretz On Wednesday 03 January 2007 13:02 CET Justin Mason wrote: > hi Malte! > > Thanks for this, it looks very useful indeed. One tweak, though: > > b) Makes the TAG part of the filename configurable. Ah, yeah, TAG was > > called HASH before :) I added the parameters tag (static tag), tag_sub > > (callback to generate a dynamic tag) and two more (tag_max_length and > > tag_warn, cf. inline doc). > > I would prefer to *add* the TAG part *as well as* the HASH stuff. > The hash is useful, since it avoids collisions -- TAG doesn't make > that guarantee (nor should it have to). > > That should be OK, right? Yeah, that was my original plan. But I wanted to keep the filename format backwards compatible. Not sure if thats really needed, I also wasn't sure if I should add the tag in the front, the end or in between. In the end I went for compatibility and assumed that in most cases the tags will still be quite unique -- at least in the producer/consumer case where you've got most probably some kind of pid in there they will be at least as unique as the current hostname/pid based ones. And if not, we've got the timestamp plus the additional really random part at the end :) Actually, when people start to filter the files based on their name, the format must not change anymore in future. Maybe the filter should apply to the tag only... Cheers, Malte http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/01/msg14.html Wed, 03 Jan 2007 07:28:22 +0000 Re: [PATCH] Advanced tagging and filtering by jm hi Malte! Thanks for this, it looks very useful indeed. One tweak, though: > b) Makes the TAG part of the filename configurable. Ah, yeah, TAG was > called HASH before :) I added the parameters tag (static tag), tag_sub > (callback to generate a dynamic tag) and two more (tag_max_length and > tag_warn, cf. inline doc). I would prefer to *add* the TAG part *as well as* the HASH stuff. The hash is useful, since it avoids collisions -- TAG doesn't make that guarantee (nor should it have to). That should be OK, right? otherwise, all looks good... --j. http://www.nntp.perl.org/group/perl.ipc.dirqueue/2007/01/msg13.html Wed, 03 Jan 2007 06:13:42 +0000 [PATCH] Advanced tagging and filtering by Malte S. Stretz Hi Justin, I was looking for a nice IPC module on CPAN and what did I find? Yet another Perl Module created by you :) Unfortunately was IPC::DirQueue really slow in one of the cases I tried to use it for. Thats why I wrote the attached patch. But first: What do I need? I have a server which has a tcpdump running, listening for UDP packets, and where you can connect via TCP to request notifications about some kind of packets. The notifications are based on ids which are inside the UDP packets. (This is actually used to do some UDP hole punching as a University project.) So we've got two parallel queues, one for the incoming requests and one for the outgoing replies. On the one side we can have n processes pushing requests into the incoming queue and waiting for replies on the outgoing one. On the other side is just a single consumer/producer, which parses the tcpdump output, looks wether there's a request for the id in the incoming queue and if so, puts a reply into the outgoing queue which is then sent via TCP to the user. That worked pretty well but at some point I got an almost-deadlock: If one of the clients pushes loads of requests into the incoming queue, it can overwhelm the consumer which can happen to look at all those new packets first before it sees the request it was actually looking for. Additionally is there quite some overhead because each time the metadata has to be read first before I can identify the owner. Actually the following simple code can have the effect I described above: use IPC::DirQueue; my $dq = IPC::DirQueue->new({ dir => 'dq' }); $dq->enqueue_string("foo!", {id => 'foo'}); $dq->enqueue_string("bar!", {id => 'bar'}); while (1) { my $job = $dq->wait_for_queued_job(0, 0.1); my $id = $job->{metadata}->{id}; #print $id; if ($id eq 'foo') { my $data = $job->get_data(); $job->finish(); print $data; } else { $job->return_to_queue(); } } If started often enough, at some point we will have loads of bars which block the foos. Just uncomment the print $id and the app will scream bar all the time but never foo. Hmmm... I guess there's actually another bug hidden because sometimes I see no foo at all; maybe the dir iterator is reset or something. Whatever, I actually went to increase the speed for applications like this. The attached patch does the following: a) Adds a parameter file_mode to the constructor; doesn't really increase speed but I guess in most cases one wants to change both modes and this way its less code :) b) Makes the TAG part of the filename configurable. Ah, yeah, TAG was called HASH before :) I added the parameters tag (static tag), tag_sub (callback to generate a dynamic tag) and two more (tag_max_length and tag_warn, cf. inline doc). c) Adds $filter parameter to wait_for_queued_job() and pickup_queued_job(). These accept a regular expression which is matched against the queue file name. I don't really like the position of the $filter parameter in the wait method, I'd prefer to have it as the first parameter and experimented with a 'ref $_[1] eq Regexp' to keep it backwards compatible but then just went the simple way and put it to the end :) d) The actual filtering stuff is done in the iterators. I guess I did quite some refactoring/rewriting there. The most important thing is the $iter->{filter} code, that I made the iterator more or less independent from the $self object and thus could use it to rewrite the fanout stuff to use a recursive (unordered) iterator, thus reducing code duplication. The tests all work (again) and I attached another one for the stuff I did. So, with the patch, change the code above into the following and its fast as... dunno, but something really fast :) use IPC::DirQueue; my $dq = IPC::DirQueue->new({ dir => 'dq', tag_sub => sub { return $_[0]->{metadata}->{id}; }}); $dq->enqueue_string("foo!", {id => 'foo'}); $dq->enqueue_string("bar!", {id => 'bar'}); while (1) { my $job = $dq->wait_for_queued_job(0, 0.1, qr/\.foo(\..*|)$/); my $id = $job->{metadata}->{id}; #print $id; my $data = $job->get_data(); print $data; $job->finish(); } Cheers, Malte http://www.nntp.perl.org/group/perl.ipc.dirqueue/2006/12/msg12.html Fri, 29 Dec 2006 12:51:56 +0000 Re: released version 0.08 of IPC::DirQueue by jm ha -- that's version 0.08, not "x.xx" ;) --j. Justin Mason writes: > Hi -- > > Version 0.08 of IPC::DirQueue is now released, and should be showing > up on CPAN rsn. Changes since 0.07: > > ------------------------------------------------------------------------ > r8093 | jmason | 2006-11-21 13:04:53 +0000 (Tue, 21 Nov 2006) | 1 line > > add t/12_no_dir.t, to test for bug 21312 (no queue directory causes die()) > ------------------------------------------------------------------------ > r6840 | jmason | 2006-09-04 11:18:38 +0100 (Mon, 04 Sep 2006) | 1 line > > calling visit_all_jobs() with a nonexistent queue directory triggered some broken err > or-handling code, resulting in a die(); fixed. thanks to Keith Amling, http://rt.cpa > n.org/Public/Bug/Display.html?id=21312 > ------------------------------------------------------------------------ > r6839 | jmason | 2006-09-04 11:16:10 +0100 (Mon, 04 Sep 2006) | 1 line > > documentation clarification on queue structure; some of the grammar was poor > ------------------------------------------------------------------------ > r6694 | jmason | 2006-07-24 14:10:27 +0100 (Mon, 24 Jul 2006) | 1 line > > adding CHANGES file to svn http://www.nntp.perl.org/group/perl.ipc.dirqueue/2006/11/msg11.html Tue, 21 Nov 2006 06:21:10 +0000 released version x.xx of IPC::DirQueue by jm Hi -- Version 0.08 of IPC::DirQueue is now released, and should be showing up on CPAN rsn. Changes since 0.07: ------------------------------------------------------------------------ r8093 | jmason | 2006-11-21 13:04:53 +0000 (Tue, 21 Nov 2006) | 1 line add t/12_no_dir.t, to test for bug 21312 (no queue directory causes die()) ------------------------------------------------------------------------ r6840 | jmason | 2006-09-04 11:18:38 +0100 (Mon, 04 Sep 2006) | 1 line calling visit_all_jobs() with a nonexistent queue directory triggered some broken err or-handling code, resulting in a die(); fixed. thanks to Keith Amling, http://rt.cpa n.org/Public/Bug/Display.html?id=21312 ------------------------------------------------------------------------ r6839 | jmason | 2006-09-04 11:16:10 +0100 (Mon, 04 Sep 2006) | 1 line documentation clarification on queue structure; some of the grammar was poor ------------------------------------------------------------------------ r6694 | jmason | 2006-07-24 14:10:27 +0100 (Mon, 24 Jul 2006) | 1 line adding CHANGES file to svn http://www.nntp.perl.org/group/perl.ipc.dirqueue/2006/11/msg10.html Tue, 21 Nov 2006 05:51:32 +0000 Re: Changing metadata inside of a job runner by jm Duncan Hill writes: > I'm looking at your IPC::DirQueue module as a solution to my current > home-grown queue runner, and have a few questions that a brief check of the > module doesn't answer. > > 1) Can the metadata be written to from a job runner? There's no function for > this, but there's nothing obvious saying 'don't do that'. > 2) If not, would you consider this feature? > > Essentially, the code I'm fitting dq into talks to an SQL database a lot. If > a log file encounters a failure being parsed, I'd like to be able to tag the > metadata with a 'don't touch, leper' flag so that other queue runners don't > try to handle it when it gets returned to the queue. Without the ability to > change the metadata, all I can do is crank the priority down, and have a > specialised runner that only picks up low priority items and moves them out > of the main queue (which would work I suppose). hi -- Unfortunately not :( Right now, the metadata is written to only once, at enqueue time, and only read thereafter. It'd make a good patch, though ;) --j. http://www.nntp.perl.org/group/perl.ipc.dirqueue/2006/05/msg9.html Wed, 10 May 2006 14:46:50 +0000 CPAN Upload: J/JM/JMASON/IPC-DirQueue-0.07.tar.gz (fwd) by Justin Mason changes: - support running the tests with a perl that isn't /usr/bin/perl, e.g. after running '/usr/local/perl561/bin/perl Makefile.PL; make; make test' - The utime() idiom being used to touch files and directories was only added in perl 5.7.2. Add backwards-compat code to deal with the 5.6.x callers - oops, indexd t scripts required POE, so the tests failed if that module wasn't installed. make them optional and not run if POE isn't installed --j. ------- Forwarded Message Date: Mon, 10 Apr 2006 22:30:27 +0200 From: PAUSE <upload@pause.perl.org> To: "Justin Mason" <jm-cpan@jmason.org> Subject: CPAN Upload: J/JM/JMASON/IPC-DirQueue-0.07.tar.gz The uploaded file IPC-DirQueue-0.07.tar.gz has entered CPAN as file: $CPAN/authors/id/J/JM/JMASON/IPC-DirQueue-0.07.tar.gz size: 25658 bytes md5: 2b8ffecc3ab15c4610c459fd3c038434 http://www.nntp.perl.org/group/perl.ipc.dirqueue/2006/04/msg8.html Mon, 10 Apr 2006 13:34:21 +0000 IPC::DirQueue 0.06 released by jm -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hey all -- I've just uploaded v0.06 of IPC::DirQueue. This has a few interesting changes; lemme quote CHANGES on the big ones: - API CHANGE: invalid metadata in an enqueued job will now causes die() to be called, instead of silent ignore. - add dq-indexd, an optional, experimental central server to maintain DQ indexes, instead of using readdir() filesystem APIs. It requires POE - fix race condition in wait_for_queued_job() that can result in missing jobs enqueued inside the same 1-second window. This would only manifest if no further files were enqueued to the same dir. - make the 600-second active-lock timeout configurable; document it better; and reconcile code behaviour with what the documentation says it does re behaviour when a stale lock occurs on a task supposedly active on a remote system. (The code was changed to match the documentation.) - fixed a long-running bug; left-over control files in 'active'. Turns out these were a side effect of qproc A completing a task before qproc B even started creating a lockfile; in that case, qproc B would get a lock on the now-complete task, then find that the queue control file was nonexistent and give up, attempting to remove the 'active' file, but instead unlinking the now-already-unlinked temporary filename, and leaving the real 'active' file behind. Totally harmless, fixed anyway. - support Reiserfs and XFS queue dirs, which do not update directory mtimes when a file is creating within them. - patch from Anton Berezin <tobez.at.tobez.org>: return reference of sorted files list internally, and use built-in sort instead of a { $a cmp $b } callback. Both are good for speed of ordered dequeuing It's filtering through CPAN now. I wouldn't recommend using dq-indexd for anything serious just yet -- it's still experimental! -- but the rest of those features are solid, and the fix for the "leftover control files" bug is welcome. ;) for what it's worth, here's some dq-indexd benchmarks from a mixed ethernet/wireless (home) network, in case you all are curious ;) -- LOCAL TIMINGS: 85changing_indexd.t real 0m53.525s user 0m5.228s sys 0m0.446s real 0m53.452s user 0m5.113s sys 0m0.536s real 0m53.437s user 0m5.411s sys 0m0.454s 85changing_ordered.t real 0m52.359s user 0m0.350s sys 0m0.121s real 0m52.432s user 0m0.352s sys 0m0.104s real 0m51.588s user 0m0.256s sys 0m0.077s 85changing_basic.t real 0m52.314s user 0m0.143s sys 0m0.043s real 0m51.337s user 0m0.346s sys 0m0.106s real 0m52.480s user 0m0.345s sys 0m0.122s REMOTE TIMINGS: 85changing_indexd.t real 3m07.718s user 0m3.191s sys 0m1.065s real 3m06.452s user 0m3.881s sys 0m1.324s real 3m09.563s user 0m3.532s sys 0m1.244s 85changing_ordered.t real 3m21.705s user 0m0.646s sys 0m0.836s real 3m31.031s user 0m0.630s sys 0m1.002s real 3m26.674s user 0m0.645s sys 0m1.027s 85changing_basic.t real 3m46.062s user 0m0.585s sys 0m0.764s real 3m40.202s user 0m0.465s sys 0m0.609s real 3m49.065s user 0m0.558s sys 0m0.854s cheers, - --j. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Exmh CVS iD8DBQFDwx+rMJF5cimLx9ARAkphAJ4xSceOTjc/6q55vlrgvXCt0EKOrACfXYXk inMnkoDQb2XRo7k7m6XrGfw= =TGm3 -----END PGP SIGNATURE----- http://www.nntp.perl.org/group/perl.ipc.dirqueue/2006/01/msg7.html Mon, 09 Jan 2006 18:47:11 +0000 Re: [PATCH] speed up ordered dequeueing in IPC-DirQueue 0.05 by jm -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (cc'd mailing list) Anton Berezin writes: > Justin, > > On Thu, Dec 29, 2005 at 02:18:06PM +0100, Anton Berezin wrote: > > > I don't think it is possibly to do much with the algorithm as such, > > since the sorting needs to be done, and the results cannot really be > > cached if we want the ordered queue to be really ordered, but I found a > > definite thing to improve. > > On the further thought, even the existing ordered method does not > provide us with an absolute guarantee that the things are going to be > processed in order, since there is a race between readdir and newer > enqueue operations. So it is possible to somewhat relax it even further > and achieve a really good speed up for some, but not all, cases. > > The idea is to keep a last_enqueued file in the queue dir. We are only > interested in its mtime. Then, on a dequeue operation, we cache the > result of readdir and only re-read the file list if the cached file list > is empty or if the mtime of last_enqueued file has changed. > > This obviously won't help much if there are a lot of concurrent enqueue > operations, but for the case "first enqueue a bunch of jobs, then launch > several processors, then repeat after some time" it helps absolutely > tremendously. Hi Anton -- many thanks for this work! The patches are welcome. I've applied the ordered-dequeueing speedup patch from the previous mail - -- with a few changes to match latest SVN (see below). It's in as r2353. I never thought the interpreter-level code path would have such a large effect on execution speed; I was assuming it'd all be system calls, which *should* be a lot slower. Looks like I was wrong. ;) However, regarding the patch from *this* mail, it's important to note that the underlying code changed in SVN at the start of December; r2255 added a "last-enqueue" flag file, initially at least. I then realised that the queue directory *itself* can be used for this purpose, in a backwards-compatible way, so now it just adds similar semantics to the directory inode itself. I found that this change was necessitated to support Reiserfs and XFS-hosted queue directories, since those filesystems didn't modify the mtime of a directory when changes were made within it, as ext3 did. (I was assuming that this was a POSIX requirement; again, looks like I was wrong. ;) There were then several more changes since then to SVN trunk, made to fix important bugs (including another race condition) and add new, experimental functionality. This patch looks very useful -- I'd be happy to add it, as a patch against current SVN, if that'd be possible. Also, it may be worthwhile making it the ordered default, assuming I can't think of a good reason why not ;) Right now though, it doesn't apply cleanly to SVN. by the way there is a mailing list; <ipc-dirqueue-subscribe@perl.org> to subscribe. it might be worth sending future patches there so other users can comment on them too. - --j. > I did not think it was a good idea to modify the default "ordered" > behavior, so my patch makes "ordered" options to be a tri-value, with > "2" denoting this "approximately ordered" way of doing things. > > The result of the same tests as yesterday, with ordered => 2 is > > tobez@heechee ~/_> time perl eq 10000 > perl eq 10000 1.73s user 1.90s system 44% cpu 8.180 total > tobez@heechee ~/_> time perl deq 10000 > perl deq 10000 1.39s user 2.12s system 94% cpu 3.726 total > > This represents a 30-fold improvement for the dequeueing case in > comparison to the yesterdays "optimized" version. > > Below is the patch which also incorporates my previous changes. > > --- DirQueue.pm.orig Thu Dec 29 14:14:10 2005 > +++ DirQueue.pm Fri Dec 30 12:53:41 2005 > @@ -101,10 +101,12 @@ The C<chmod>-style file mode for queue c > specified as a string with a leading 0. It will be affected by the > current process C<umask>. > > -=item ordered => { 0 | 1 } (default: 1) > +=item ordered => { 0 | 1 | 2 } (default: 1) > > Whether the jobs should be processed in order of submission, or > -in no particular order. > +in no particular order. The special value of 2 denotes "approximately > +ordered", which does not make as strong guarantees about processing > +order as simply ordered queue, but is much faster in a number of cases. > > =item buf_size => $number (default: 65536) > > @@ -344,6 +346,7 @@ sub enqueue_backend { > goto failure; > } > > + $self->touch_last_enqueued() if $self->{ordered} == 2; > # my $pathqueue_created = 1; # not required, we're done! > return 1; > > @@ -360,6 +363,17 @@ failure: > return; > } > > +sub touch_last_enqueued > +{ > + my ($self) = @_; > + my $last_enqueued_file = $self->q_dir().SLASH."last_enqueued"; > + if (sysopen(LAST, $last_enqueued_file, O_WRONLY|O_CREAT, 0666)) { > + my $s = "X"; > + syswrite(LAST, $s, 1); > + close LAST; > + } > +} > + > ########################################################################### > > =item $job = $dq->pickup_queued_job(); > @@ -384,12 +398,12 @@ sub pickup_queued_job { > $self->ensure_dir_exists ($pathactivedir); > > my $ordered = $self->{ordered}; > - my @files; > + my $files; > my $dirfh; > > if ($ordered) { > - @files = $self->get_dir_filelist_sorted($pathqueuedir); > - if (scalar @files <= 0) { > + $files = $self->get_dir_filelist_sorted($pathqueuedir); > + if (@$files <= 0) { > return if $self->queuedir_is_bad($pathqueuedir); > } > } else { > @@ -406,7 +420,7 @@ sub pickup_queued_job { > my $pathtmpactive; > > if ($ordered) { > - $nextfile = shift @files; > + $nextfile = shift @$files; > } else { > $nextfile = readdir($dirfh); > } > @@ -638,12 +652,12 @@ sub visit_all_jobs { > my $pathactivedir = $self->q_subdir('active'); > > my $ordered = $self->{ordered}; > - my @files; > + my $files; > my $dirfh; > > if ($ordered) { > - @files = $self->get_dir_filelist_sorted($pathqueuedir); > - if (scalar @files <= 0) { > + $files = $self->get_dir_filelist_sorted($pathqueuedir); > + if (@$files <= 0) { > return if $self->queuedir_is_bad($pathqueuedir); > } > } else { > @@ -658,7 +672,7 @@ sub visit_all_jobs { > my $nextfile; > while (1) { > if ($ordered) { > - $nextfile = shift @files; > + $nextfile = shift @$files; > } else { > $nextfile = readdir($dirfh); > } > @@ -728,14 +742,30 @@ sub finish_job { > > sub get_dir_filelist_sorted { > my ($self, $dir) = @_; > + my $last_enqueued_file = $self->q_dir().SLASH."last_enqueued"; > + > + if ($self->{ordered} == 2 && $self->{filelist_sorted}{$dir}) { > + my $mtime = (stat($last_enqueued_file))[9]; > + return $self->{filelist_sorted}{$dir} > + if defined $mtime && defined $self->{filelist_sorted_mtime}{$dir} && > + $mtime <= $self->{filelist_sorted_mtime}{$dir} && > + @{$self->{filelist_sorted}{$dir}}; > + } > > if (!opendir (DIR, $dir)) { > return (); # no dir? nothing queued > } > # have to read the lot, to sort them. > - my @files = sort { $a cmp $b } grep { /^\d/ } readdir(DIR); > + my @files = sort grep { /^\d/ } readdir(DIR); > closedir DIR; > - return @files; > + > + if ($self->{ordered} == 2) { > + $self->{filelist_sorted_mtime}{$dir} = (stat($last_enqueued_file))[9]; > + $self->{filelist_sorted}{$dir} = \@files; > + return $self->{filelist_sorted}{$dir}; > + } else { > + return \@files; > + } > } > > ########################################################################### > @@ -998,7 +1028,7 @@ sub new_q_filename { > $job->{pri}, > $gmt[5]+1900, $gmt[4]+1, $gmt[3], $gmt[2], $gmt[1], $gmt[0], > $job->{time_submitted_msecs}, > - hash_string_to_filename ($self->gethostname().$$)); > + hash_string_to_filename ($self->gethostname().$$.$self->get_random_int())); > > # normally, this isn't used. but if there's a collision, > # all retries after that will do this; in this case, the > > \Anton. > -- > An undefined problem has an infinite number of solutions. > -- Robert A. Humphrey -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Exmh CVS iD8DBQFDvdyhMJF5cimLx9ARAgM4AJ0bCfFNHsx9SjdGYXLtzOjhPW9MeQCgguxc Ng8JAoBN7/thHp5CBgpSIIU= =l6su -----END PGP SIGNATURE----- http://www.nntp.perl.org/group/perl.ipc.dirqueue/2006/01/msg6.html Thu, 05 Jan 2006 18:59:39 +0000 CPAN Upload: J/JM/JMASON/IPC-DirQueue-0.05.tar.gz (fwd) by jm -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 IPC::DirQueue version 0.05 is released! Here's the key new features: - new API: added enqueue_sub() API, to enqueue data from a closure. - bug: more paranoid link_into_dir_no_retry implementation, hopefully will fix a bug that leaves tmp active files lying around. - bug: skip fork()-requiring tests on win32. should be showing up on http://www.cpan.org/modules/by-authors/id/J/JM/JMASON/ rsn... - --j. - ------- Forwarded Message > Date: Fri, 22 Apr 2005 22:05:22 +0200 > From: PAUSE <upload@pause.perl.org> > To: "Justin Mason" <jm-cpan@jmason.org> > Subject: CPAN Upload: J/JM/JMASON/IPC-DirQueue-0.05.tar.gz > > The uploaded file > > IPC-DirQueue-0.05.tar.gz > > has entered CPAN as > > file: $CPAN/authors/id/J/JM/JMASON/IPC-DirQueue-0.05.tar.gz > size: 16861 bytes > md5: 4ceb6ae1882c5412c3daed7e30f04f3a > > No action is required on your part > Request entered by: JMASON (Justin Mason) > Request entered on: Fri, 22 Apr 2005 20:03:17 GMT > Request completed: Fri, 22 Apr 2005 20:05:22 GMT -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Exmh CVS iD8DBQFCaVstMJF5cimLx9ARAu4hAKCVooT93bvn4JGmQW3qO93OaGTRhQCcCcjs +gvXBYoxsCcTHVaJdWNtvfI= =+2G+ -----END PGP SIGNATURE----- http://www.nntp.perl.org/group/perl.ipc.dirqueue/2005/04/msg5.html Fri, 22 Apr 2005 13:16:16 +0000 qpsmtpd queue plugin? by David Sparks Greetings, I've been investigating using this module with qpsmtpd. Does such a plugin already exist? I hacked together a quick proof of concept to test performance of the methods and found roughly: $dq->enqueue_file: 2600 msg/min $dq->enqueue_string: 2900 msg/min Note these are qpsmtpd #s and the queue dir lives on a NFS mounted volume so it isn't showing IPC::DirQueue performance. It also seems that when using the enqueue_file method the original file still exists and has to be rm'd separately? Is this expected? My assumption (love em) was that the original file would be re-linked into the queue. Cheers, ds http://www.nntp.perl.org/group/perl.ipc.dirqueue/2005/04/msg4.html Thu, 21 Apr 2005 00:21:55 +0000 Re: how to "kill" a job by jm -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Bruce Robertson writes: > Is there such an API point? > If not, do you have something under development? > Else, any pointers on how to do it? There isn't yet such a thing, no, and I haven't planned it. (yet) Kill an as-yet unprocessed job would be easy -- simply an unlink. But if the job is active, it'd be quite a bit harder; it's easy in the LPR-style of queueing, because that has (a) a central server and (b) each job is a separate process, so a kill(3) of the process can be used to kill the job, but in DQ, there's no separate process per job -- one process can deal with many jobs sequentially. I would say the cleanest way to do it would be to arrange that dq servers can provide a new constructor switch indicating a signal that should be delivered if a job kill is to be requested; if that switch isn't present, killing an active job isn't supported. The DQ server code would then write that to a flag file in the queue directory, so that DQ clients (ie. a new "dq-kill" script) can be aware that the server(s) support this. The server code should then implement a signal handler for that signal; that handler will check to see if it is currently processing a killed job (ie. if the data file has been unlinked before it's complete), and give up the remainder of the job's processing if it is. (alternatively, the server could just ignore the signal, if it's not an issue for that server's design, e.g. if the jobs are very lightweight, and completing processing of a killed job will not cause problems.) If that constructor switch was not specified, I think it would be unsafe to assume that DQ kill is supported by that queue; we can't go unlinking active jobs without some kind of "this is supported" negotiation, as a disappearing job file may cause a die() in server code that wasn't aware this was likely to happen. - --j. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Exmh CVS iD8DBQFCAZN6MJF5cimLx9ARAr5NAJ0XaiEfTu55tve/g2gmuylLfwrK7wCdF2yF dLOlbsvNhHl50TfJYtzwJfA= =jxvb -----END PGP SIGNATURE----- http://www.nntp.perl.org/group/perl.ipc.dirqueue/2005/02/msg3.html Wed, 02 Feb 2005 18:59:14 +0000 IPC::DirQueue 0.04 released by jm -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Jeffrey -- as the (so far) only known dq user, thought you might like to know about a few DQ bits and pieces. I've cc'd the all-new mailing list, for any other users out there ;) First off, there's a new release -- changes: - bug: dqserver should run forever when --njobs is 0, or unset; instead, it was exiting immediately. Patch from Jeffrey Wescott <jeff . binaryfeed . org>. - bug: tracked down a mysterious condition that resulted in unused files being left in the active dir. - added debugging (so far only enableable by setting a package global var) - updated documentation on the directory structure used - added note to POD about mailing list Download from CPAN, once the mirrors update. Next, the SVN repository lives here: http://svn.perl.org/modules/ipc-dirqueue/ and there's a mailing list -- mail ipc-dirqueue-subscribe .at. perl.org to subscribe. (thanks to Ask and Robert @perl.org who are now providing free SVN repos and list hosting for CPAN modules!) - --j. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Exmh CVS iD8DBQFB7bZAMJF5cimLx9ARAvKiAKCdnfTuBhqyjXcaalJ/arsRykMGeACgoyZu Cu+G9JasWZC6KzF9TrN9SVo= =pUoq -----END PGP SIGNATURE----- http://www.nntp.perl.org/group/perl.ipc.dirqueue/2005/01/msg2.html Tue, 18 Jan 2005 17:22:32 +0000 Re: bug in IPC-DirQueue 0.03? by jm -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Jeffrey -- thanks for that patch -- it's applied to svn now. I should really do a new release RSN... - --j. Jeffrey Wescott writes: > A patch for the bug described below ... I may send you another patch > for a -q option (quiet). > > -------------------------------->8 snip, snip > Index: dq-server > ==================================================================RCS file: /usr/local/cvsroot/IPC-DirQueue/dq-server,v > retrieving revision 1.1.1.1 > diff -u -r1.1.1.1 dq-server > --- dq-server 11 Jan 2005 21:55:01 -0000 1.1.1.1 > +++ dq-server 11 Jan 2005 23:56:21 -0000 > @@ -56,10 +56,11 @@ > ) or usage(); > $dir or usage(); > > +my $jobsleft = $njobs; > my $dq = IPC::DirQueue->new({ dir => $dir }); > > while (1) { > - last if ($njobs-- <= 0); > + $njobs && last if ($jobsleft-- <= 0); > > my $job = $dq->wait_for_queued_job(); > if (!$job) { next; } > -------------------------------->8 snip, snip > > ++jeffrey > > On Jan 11, 2005, at 15:48, Jeffrey Wescott wrote: > > > The dq-server command claims in the documentation that not specifying > > --njobs or specifying --njobs 0 should mean that the server will run > > forever (i.e. -- process an unlimited number of jobs). However, if > > you don't specify --njobs or specify --njobs 0, it just quits. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Exmh CVS iD8DBQFB5dxiMJF5cimLx9ARAskvAKCcDqmjF5fv4H7CccGoYk9zMUKK9QCeLDOt k3L2s9sElVjlOKyrO08l6Go= =gjXV -----END PGP SIGNATURE----- http://www.nntp.perl.org/group/perl.ipc.dirqueue/2005/01/msg1.html Thu, 13 Jan 2005 02:55:59 +0000