perl.poe

Re: poe and amqp/rabbitmq as a worker manager by Nick Perez

2013-03-15T07:29:37Z

From: Nick Perez None of this sounds particularly difficult, just a SMOP. The
interesting bits aren't getting the job server up and running. What
will be interesting is scaling the server. Am I correct to assume that
the workers now directly pull from the queue? And then you would be
changing that to a single queue puller that will then redistribute to
worker processes? You may end up killing your over all capacity with an
artificial bottleneck.

We ended up building a similar system (but using 0mq, which has been
both good and bad, but mostly bad, heh) at work with a very small/simple
broker written in C for speed distributing work to individual workers.
But instead of having complicated management, we just opted for a round
robin approach. And instead of querying the broker for stats, we setup
graphite and statsd and the worker processes themselves spit out
metrics in a non-blocking fashion (after sending off the reply) so we
can monitor things like request processing time, various counts on
request resolution, etc.

The workers themselves actually have a control channel to which we can
individually address to tell them to pause/resume/stop. This gives us
the flexibility to do rolling restarts of our workers when we are doing
a deployment of updated worker code.

And knowing that we can easily consume all of the cores on the machine,
our infrastructure is such that we have multiple of such machines and
do round-robin dns (with the ability to drop a host out for maintenance
if needed).

In the end, I would shy away from monolithic systems, especially in
POE. POE is fast, especially with the right event loop, but ultimately,
it is still a single process. Any time spent processing your management
requests is time that it is not spending assigning work or
gathering results from a worker.

Some things to consider (and watch out for) if you decide to move
forward with this:

Inter-session communication is slow. Really slow. And the number of
sessions that the kernel must keep track of also has an impact on
performance (POE does a lot of book keeping). You'll need to figure out
some sort of serialization mechanism for framing your requests to the
workers. By default, POE::Filter::Reference uses Storable which is
really slow and also produces pretty bloated output. You will need to
possibly consider a different Filter module (I know of someone building
a Filter using Sereal). Also be prepared to manage timeouts and your
worker processes going away and what your master process will do in
those cases. You will likely end up writing more error handling code
than actual get-work-done code.

For what it is worth, I actually wrote our worker code using Reflex (a
better abstraction layer on top of POE). It basically spins up a
singleton session and completes as many operations within a timeslice
as possible to avoid going through the kernel as often. If latency in
your worker processes is a concern you might consider using Reflex.
There are various POE adaptors as well so they interoperate.

On Thu, 14 Mar 2013 18:23:24 -0700
Kevin Goess wrote:

> We currently use rabbitmq for message between our web application and
> asynchronous workers. The worker management is somewhat ad hoc, and
> we're looking for a way to get a better handle on them.
>
> It looks like POE has the components that I want, so it's finally
> time for me to learn about POE. But there's an awful lot of POE
> material out there, and I'm afraid if I try to digest it all I'll
> have made a lot of false starts before I find the right path. Can
> anybody tell me if I'm going in the right direction, or if there's
> already something out there that does this?
>
> I think I want to use POE as a job server driven by the AMQP POE
> client, with workers in separate child processes handled by something
> like POE::Component::Daemon (which has a scoreboard) or
> POE::Wheel::Run.
>
> I'd like to be able to query the server on a management port with
> questions like
>
> - How many messages per queue are you receiving
> - What's the completion time for jobs on each queue
> - How idle/busy are your child workers?
>
> It should be able to take commands like "add or drop these queues",
> and it should automatically take care of tasks like making sure no
> queue is being starved in favor of another queue.
>
> Is this the right idea? Is there a general direction for this that
> would be obvious to sketch out that would save me having to
> understand every example in the poe cookbook?
>
> Any pointers would be appreciated. Thanks!

--

Nicholas Perez
XMPP/Email: nick@nickandperla.net
https://metacpan.org/author/NPEREZ
http://github.com/nperez

Re: poe and amqp/rabbitmq as a worker manager by S Chaitanya

2013-03-15T05:07:31Z

From: S Chaitanya Great question, Kevin. I'm looking for an answer to the same question!

On Fri, Mar 15, 2013 at 6:53 AM, Kevin Goess wrote:

> We currently use rabbitmq for message between our web application and
> asynchronous workers. The worker management is somewhat ad hoc, and we're
> looking for a way to get a better handle on them.
>
> It looks like POE has the components that I want, so it's finally time for
> me to learn about POE. But there's an awful lot of POE material out there,
> and I'm afraid if I try to digest it all I'll have made a lot of false
> starts before I find the right path. Can anybody tell me if I'm going in
> the right direction, or if there's already something out there that does
> this?
>
> I think I want to use POE as a job server driven by the AMQP POE client,
> with workers in separate child processes handled by something like
> POE::Component::Daemon (which has a scoreboard) or POE::Wheel::Run.
>
> I'd like to be able to query the server on a management port with questions
> like
>
> - How many messages per queue are you receiving
> - What's the completion time for jobs on each queue
> - How idle/busy are your child workers?
>
> It should be able to take commands like "add or drop these queues", and it
> should automatically take care of tasks like making sure no queue is being
> starved in favor of another queue.
>
> Is this the right idea? Is there a general direction for this that would be
> obvious to sketch out that would save me having to understand every example
> in the poe cookbook?
>
> Any pointers would be appreciated. Thanks!
>

poe and amqp/rabbitmq as a worker manager by Kevin Goess

2013-03-15T01:23:37Z

From: Kevin Goess We currently use rabbitmq for message between our web application and
asynchronous workers. The worker management is somewhat ad hoc, and we're
looking for a way to get a better handle on them.

It looks like POE has the components that I want, so it's finally time for
me to learn about POE. But there's an awful lot of POE material out there,
and I'm afraid if I try to digest it all I'll have made a lot of false
starts before I find the right path. Can anybody tell me if I'm going in
the right direction, or if there's already something out there that does
this?

I think I want to use POE as a job server driven by the AMQP POE client,
with workers in separate child processes handled by something like
POE::Component::Daemon (which has a scoreboard) or POE::Wheel::Run.

I'd like to be able to query the server on a management port with questions
like

- How many messages per queue are you receiving
- What's the completion time for jobs on each queue
- How idle/busy are your child workers?

It should be able to take commands like "add or drop these queues", and it
should automatically take care of tasks like making sure no queue is being
starved in favor of another queue.

Is this the right idea? Is there a general direction for this that would be
obvious to sketch out that would save me having to understand every example
in the poe cookbook?

Any pointers would be appreciated. Thanks!

Re: Wait for first recv. by Rizwan Hisham

2013-02-28T10:01:22Z

From: Rizwan Hisham Just a thought.

You can devise a ping mechanism. Ping the other client, recieve PONG then
transmit the data. You will have to do it every time you transfer the data.

On Thu, Feb 28, 2013 at 2:46 PM, yakudzo wrote:

> Hello POE guys,
>
> I used example poe.perl.org/?POE_Cookbook/**TCP_Forwardingto create application that will connect two clients and will send/recv info
> between them. I need to send information between them only when both of
> them are connected so I need first listen on both ports and only when both
> are accepted I need to start recv info from them and send it to another.
> How can I do this?
>
> With best regards,
> Konstantin
>

--
Best Ragards
Rizwan H Qureshi

V: +92 (0) 3333 6767 26
linkedin.com/in/rhqureshi

Wait for first recv. by yakudzo

2013-02-28T09:46:32Z

From: yakudzo Hello POE guys,

I used example poe.perl.org/?POE_Cookbook/TCP_Forwarding to create
application that will connect two clients and will send/recv info
between them. I need to send information between them only when both of
them are connected so I need first listen on both ports and only when
both are accepted I need to start recv info from them and send it to
another. How can I do this?

With best regards,
Konstantin

Re: Combining services. by Antti Linno

2013-02-02T15:18:29Z

From: Antti Linno Hm, seems we need something in the lines of "Even monkey can program POE"
( http://en.wikipedia.org/wiki/Even_a_Monkey_Can_Draw_Manga )

Server:
#!/usr/bin/env perl

use Modern::Perl '2012';
use POE qw(Component::Server::TCP Filter::Reference);

POE::Component::Server::TCP->new(
Alias => "sum_server",
Port => 11211,
ClientFilter => "POE::Filter::Reference",
ClientInput => sub {
my ( $sender, $heap, $input ) = @_[ SESSION, HEAP, ARG0 ];
say $input;
},
);
$poe_kernel->run();
exit 0;

Client:
#!/usr/bin/env perl

use Modern::Perl '2012';
use IO::Socket::INET;
use POE::Filter::Reference;
use Data::Dumper;

my $socket = new IO::Socket::INET(
PeerHost => '127.0.0.1',
PeerPort => '11211',
Proto => 'tcp',
) or die "ERROR in Socket Creation : $!\n";

my $filter = POE::Filter::Reference->new();

my %result = (
task => 'Fire and forget',
status => " seems ok to me ",
);

my $output = $filter->put( [ \%result ] );
$socket->send($output);

$socket->close();

say "Theoretically sent";

==

My problem is, the server's ClientInput is never fired. So how do I send
data from point A to point B?

Greetings,
Antti

On Sat, Feb 2, 2013 at 12:43 AM, Rocco Caputo wrote:

> Don't use an asynchronous client if you don't need one. The cookbook
> includes examples using POE::Filter::Reference by itself.
>
> --
> Rocco Caputo
>
> On Jan 31, 2013, at 09:41, Antti Linno wrote:
>
> > Hm, as my needs for application server are very simple, I feel that POE
> TCP
> > client is too heavy, compared to IO::Socket for example.
> >
> > Slim client=>
> >
> > use IO::Socket;
> > use JSON qw( encode_json );
> >
> > my $sock = new IO::Socket::INET (
> > PeerAddr => 'localhost',
> > PeerPort => '11211',
> > Proto => 'tcp'
> > );
> > print $sock ( encode_json( { event => "position", phone => "12345" } ) .
> > '[EOM]' );
> > close $sock;
> > exit;
> >
> > In server I parse messages with=>
> >
> > POE::Component::Server::TCP->new(
> > Alias => "emt_server",
> > Address => "localhost",
> > Port => 11211,
> > ClientFilter => [ "POE::Filter::Line", Literal => "[EOM]" ],
> >
> > ClientConnected => sub {
> > say "Client connected";
> > },
> > # Handle client requests here.
> > ClientInput => sub {
> > my ( $heap, $params_json ) = @_[ HEAP, ARG0 ];
> > print $params_json;
> > Wrapper::emt_event( decode_json $params_json );
> > },
> >
> > This works. But I somehow feel that this is not elegant solution. How do
> I
> > use POE::Filter::Reference instead of Line? Tried to freeze and send, but
> > the server did not enter ClientInput event. Is there a slim and elegant
> > solution? :) Or should I use cookbook client and just kill client after
> > every request with $kernel->yield("shutdown")? Cookbook client works
> fine.
> >
> > If I use ClientFilter => [ "POE::Filter::Reference", "YAML" ], I cannot
> > trigger ClientConnected subroutine with IO::Socket :) Tried to use
> > standalone Filter::Reference->put and send it over socket, but still no
> > luck.
> >
> > Thank you all.
> >
> >
> > On Wed, Jan 30, 2013 at 7:29 PM, Antti Linno
> wrote:
> >
> >> Thank you. You saved me from fourth standalone daemon :D
> >>
> >>
> >> On Wed, Jan 30, 2013 at 5:09 PM, Rocco Caputo
> wrote:
> >>
> >>> On Jan 30, 2013, at 09:49, Antti Linno wrote:
> >>>>
> >>>> Or I need advice, how to merge several sungo's(
> >>>> http://poe.perl.org/?POE_Cookbook/TCP_Servers) daemons into one
> >>> package, if
> >>>> someone would be so kind. Adding UDP should be fairly similar then.
> >>>
> >>>> Any hints, second opinion(maybe not to merge TCP and UDP), or code
> >>> examples
> >>>> are welcome :)
> >>>
> >>>
> >>> I've attached a working version of your sample code. It starts two TCP
> >>> servers and a UDP service, and lets them all run at once in the same
> >>> process.
> >>>
> >>> --
> >>> Rocco Caputo
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
>
>

Re: Combining services. by Rocco Caputo

2013-02-01T22:44:01Z

From: Rocco Caputo Don't use an asynchronous client if you don't need one. The cookbook includes examples using POE::Filter::Reference by itself.

--
Rocco Caputo

On Jan 31, 2013, at 09:41, Antti Linno wrote:

> Hm, as my needs for application server are very simple, I feel that POE TCP
> client is too heavy, compared to IO::Socket for example.
>
> Slim client=>
>
> use IO::Socket;
> use JSON qw( encode_json );
>
> my $sock = new IO::Socket::INET (
> PeerAddr => 'localhost',
> PeerPort => '11211',
> Proto => 'tcp'
> );
> print $sock ( encode_json( { event => "position", phone => "12345" } ) .
> '[EOM]' );
> close $sock;
> exit;
>
> In server I parse messages with=>
>
> POE::Component::Server::TCP->new(
> Alias => "emt_server",
> Address => "localhost",
> Port => 11211,
> ClientFilter => [ "POE::Filter::Line", Literal => "[EOM]" ],
>
> ClientConnected => sub {
> say "Client connected";
> },
> # Handle client requests here.
> ClientInput => sub {
> my ( $heap, $params_json ) = @_[ HEAP, ARG0 ];
> print $params_json;
> Wrapper::emt_event( decode_json $params_json );
> },
>
> This works. But I somehow feel that this is not elegant solution. How do I
> use POE::Filter::Reference instead of Line? Tried to freeze and send, but
> the server did not enter ClientInput event. Is there a slim and elegant
> solution? :) Or should I use cookbook client and just kill client after
> every request with $kernel->yield("shutdown")? Cookbook client works fine.
>
> If I use ClientFilter => [ "POE::Filter::Reference", "YAML" ], I cannot
> trigger ClientConnected subroutine with IO::Socket :) Tried to use
> standalone Filter::Reference->put and send it over socket, but still no
> luck.
>
> Thank you all.
>
>
> On Wed, Jan 30, 2013 at 7:29 PM, Antti Linno wrote:
>
>> Thank you. You saved me from fourth standalone daemon :D
>>
>>
>> On Wed, Jan 30, 2013 at 5:09 PM, Rocco Caputo wrote:
>>
>>> On Jan 30, 2013, at 09:49, Antti Linno wrote:
>>>>
>>>> Or I need advice, how to merge several sungo's(
>>>> http://poe.perl.org/?POE_Cookbook/TCP_Servers) daemons into one
>>> package, if
>>>> someone would be so kind. Adding UDP should be fairly similar then.
>>>
>>>> Any hints, second opinion(maybe not to merge TCP and UDP), or code
>>> examples
>>>> are welcome :)
>>>
>>>
>>> I've attached a working version of your sample code. It starts two TCP
>>> servers and a UDP service, and lets them all run at once in the same
>>> process.
>>>
>>> --
>>> Rocco Caputo
>>>
>>>
>>>
>>>
>>>
>>>
>>

Re: Combining services. by Antti Linno

2013-01-31T14:41:13Z

From: Antti Linno Hm, as my needs for application server are very simple, I feel that POE TCP
client is too heavy, compared to IO::Socket for example.

Slim client=>

use IO::Socket;
use JSON qw( encode_json );

my $sock = new IO::Socket::INET (
PeerAddr => 'localhost',
PeerPort => '11211',
Proto => 'tcp'
);
print $sock ( encode_json( { event => "position", phone => "12345" } ) .
'[EOM]' );
close $sock;
exit;

In server I parse messages with=>

POE::Component::Server::TCP->new(
Alias => "emt_server",
Address => "localhost",
Port => 11211,
ClientFilter => [ "POE::Filter::Line", Literal => "[EOM]" ],

ClientConnected => sub {
say "Client connected";
},
# Handle client requests here.
ClientInput => sub {
my ( $heap, $params_json ) = @_[ HEAP, ARG0 ];
print $params_json;
Wrapper::emt_event( decode_json $params_json );
},

This works. But I somehow feel that this is not elegant solution. How do I
use POE::Filter::Reference instead of Line? Tried to freeze and send, but
the server did not enter ClientInput event. Is there a slim and elegant
solution? :) Or should I use cookbook client and just kill client after
every request with $kernel->yield("shutdown")? Cookbook client works fine.

If I use ClientFilter => [ "POE::Filter::Reference", "YAML" ], I cannot
trigger ClientConnected subroutine with IO::Socket :) Tried to use
standalone Filter::Reference->put and send it over socket, but still no
luck.

Thank you all.

On Wed, Jan 30, 2013 at 7:29 PM, Antti Linno wrote:

> Thank you. You saved me from fourth standalone daemon :D
>
>
> On Wed, Jan 30, 2013 at 5:09 PM, Rocco Caputo wrote:
>
>> On Jan 30, 2013, at 09:49, Antti Linno wrote:
>> >
>> > Or I need advice, how to merge several sungo's(
>> > http://poe.perl.org/?POE_Cookbook/TCP_Servers) daemons into one
>> package, if
>> > someone would be so kind. Adding UDP should be fairly similar then.
>>
>> > Any hints, second opinion(maybe not to merge TCP and UDP), or code
>> examples
>> > are welcome :)
>>
>>
>> I've attached a working version of your sample code. It starts two TCP
>> servers and a UDP service, and lets them all run at once in the same
>> process.
>>
>> --
>> Rocco Caputo
>>
>>
>>
>>
>>
>>
>

Re: Combining services. by Antti Linno

2013-01-30T17:29:18Z

From: Antti Linno Thank you. You saved me from fourth standalone daemon :D

On Wed, Jan 30, 2013 at 5:09 PM, Rocco Caputo wrote:

> On Jan 30, 2013, at 09:49, Antti Linno wrote:
> >
> > Or I need advice, how to merge several sungo's(
> > http://poe.perl.org/?POE_Cookbook/TCP_Servers) daemons into one
> package, if
> > someone would be so kind. Adding UDP should be fairly similar then.
>
> > Any hints, second opinion(maybe not to merge TCP and UDP), or code
> examples
> > are welcome :)
>
>
> I've attached a working version of your sample code. It starts two TCP
> servers and a UDP service, and lets them all run at once in the same
> process.
>
> --
> Rocco Caputo
>
>
>
>
>
>

Re: Combining services. by Rocco Caputo

2013-01-30T15:10:13Z

From: Rocco Caputo On Jan 30, 2013, at 09:49, Antti Linno wrote:
>
> Or I need advice, how to merge several sungo's(
> http://poe.perl.org/?POE_Cookbook/TCP_Servers) daemons into one package, if
> someone would be so kind. Adding UDP should be fairly similar then.

> Any hints, second opinion(maybe not to merge TCP and UDP), or code examples
> are welcome :)

I've attached a working version of your sample code. It starts two TCP servers and a UDP service, and lets them all run at once in the same process.

--
Rocco Caputo

Combining services. by Antti Linno

2013-01-30T14:49:37Z

From: Antti Linno Hello.

I need help in consolidating services. Tried to read documentation, but
didn't understand much. Checked cookbook and found a couple of suitable
examples and as the time was pressing, just started services.

Went with sungo's TCP server and UDP socket recipe. Now I have 3 tcp
programs running and 1 udp. All are standalone daemons.It's time to migrate
them to one executable file.
I can migrate sungo's recipe to the POE::Component::Server::TCP easily(I
think), hence I can create 3 servers in one file, but how do I add UDP
server?

TCP test snippet(skipped some parts):

use POE qw(Component::Server::TCP);

POE::Component::Server::TCP->new(
Alias => "echo_server",
Port => 11211,
ClientInput => sub {
my ($session, $heap, $input) = @_[SESSION, HEAP, ARG0];
print "Server [1] Session ", $session->ID(), " got input: $input\n";
$heap->{client}->put($input);
}
);

POE::Component::Server::TCP->new(
Alias => "echo_server",
Port => 11212,
ClientInput => sub {
my ($session, $heap, $input) = @_[SESSION, HEAP, ARG0];
print "Server [2] Session ", $session->ID(), " got input: $input\n";
$heap->{client}->put($input);
}
);

$poe_kernel->run();

Or I need advice, how to merge several sungo's(
http://poe.perl.org/?POE_Cookbook/TCP_Servers) daemons into one package, if
someone would be so kind. Adding UDP should be fairly similar then.

With UDP part I tried some rookie "throw it together, maybe it works"
stuff, but it didn't :) Tried to use Session->create twice, but the second
session didn't work.

Any hints, second opinion(maybe not to merge TCP and UDP), or code examples
are welcome :)

ATM the barebones of the working UDP service are(took from cookbook,
cleaner and less code, but the gist is same):

POE::Session->create(
inline_states => {
_start => \&server_start,
get_datagram => \&server_read,
});POE::Kernel->run();exit;
sub server_start {
my $kernel = $_[KERNEL];
my $socket = IO::Socket::INET->new(
Proto => 'udp',
LocalPort => '8675',
);
die "Couldn't create server socket: $!" unless $socket;
$kernel->select_read($socket, "get_datagram");}
sub server_read {
my ($kernel, $socket) = @_[KERNEL, ARG0];
my $remote_address = recv($socket, my $message = "", DATAGRAM_MAXLEN, 0);
return unless defined $remote_address;
my ($peer_port, $peer_addr) = unpack_sockaddr_in($remote_address);
my $human_addr = inet_ntoa($peer_addr);
print "(server) $human_addr : $peer_port sent us $message\n";
$message =~ tr[a-zA-Z][n-za-mN-ZA-M];
send($socket, $message, 0, $remote_address) == length($message)
or warn "Trouble sending response: $!";}

Thank you.

Re: POE and the time by Jeremy Leader

2012-12-17T19:55:44Z

From: Jeremy Leader This discussion reminded me of:

http://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time

--
Jeremy Leader
jleader@oversee.net

On 12/14/2012 09:52 PM, Chris Fedde wrote:
> A few comments about the wiki notes.
>
> First ntpd is careful about ensuring that time always moves forward. It
> does this by adjusting the clock slew rather than jumping time forwards or
> backwards. When ntpd first starts up and the system clock is not set
> within 5 seconds relative to its ntp servers then ntpd will die with a log
> message rather than jump the time. Within about an hour of starting ntpd
> can have the system synchronized within a few milliseconds of its best
> server.
>
> Some admins like to run ntpdate out of cron and force a time reset at
> regular intervals. It's never a good idea to force time jumps. It can case
> all kinds of trouble for databases, NFS, cryptography, process scheduler
> and even event systems written in perl.
>
> Most systems have a separate hardware clocks that is pretty accurate. OS
> time is typically synchronized with it at boot then runs independently of
> it. Some admins like to synch with hwclock out of cron. This is also a
> bad idea. On "wintel" type systems the hardware clock typically reports
> only down to seconds. This is much too rough for modern time keeping.,
>
> Virtual machines add another whole layer of complexity to the picture. A
> vm will frequently have time jumps between time slices when it gets to run.
> Things that require real time behaviors probably should not be run in a
> vm. VM guests have to trust their host for an accurate representation of
> time. And all the issues that crop up above add another layer of
> complexity to accurate time keeping here
>
> There are very few good reasons for administrative clock adjustments.
> These days there are no good reasons based on simple calendrical causes.
> Soon even the dreaded leap second will be dealt with through math and
> lookup tables rather than clock adjustments. Nearly all other reasons to
> adjust the clock are political and organizational rather than technical.
>
> If we still want to deal with time sifts we have to detect them. When time
> jumps occur it is hard for a program to detect them. The only way that the
> program would know that time has changed is if it has two sources to
> compare. If such an event has been detected then it has to decide if there
> was a good administrative reason for the change or not.
>
> If the program decides to take action because it has detected a time change
> then it must have an array of logic to decide what special behaviors to
> apply for all of the various ways that the change could have occurred.
> Should it fire missed events? Should it recompute the offset for future
> events? what if the scale of the shift is large? What if the scale of the
> shift is tiny? What if time shifts occur frequently? What if they are
> noisy, crossing and recrossing the same point in time? We quickly get into
> pathological behaviors that either need to be accounted or ignored.
>
> Most developers who have cracked open this nut have eventually decided to
> leave well enough alone and have their system trust that the OS knows what
> time it is and that the admin had a "good reason" for mucking with the
> system time and leave it at that. Trying to correct for bad admin
> behavior is better done at the social layer than at the technical one.
>
> Just my thoughts.
> chris

Re: POE and the time by Philip Gwyn

2012-12-17T17:39:46Z

From: Philip Gwyn
Your post is an excellent counterpoint. I agree in theory with all your
comments. But in practice, things are broken. POE should not depend on a
stable, monotonic time(2) because there is zero guarantee of such.

On 15-Dec-2012 Chris Fedde wrote:

> The only way that the
> program would know that time has changed is if it has two sources to
> compare.

Modern systems do have 2 sources to compare. See timer_gettime(2)
CLOCK_REALTIME vs CLOCK_MONOTONIC (for Linux). Win32 has SetTimer et al

> If the program decides to take action because it has detected a time change
> then it must have an array of logic to decide what special behaviors to
> apply for all of the various ways that the change could have occurred.

This is why I propose the 'clock-skew' psuedo-signal.

> Trying to correct for bad admin behavior is better done at the social layer
> than at the technical one.

POE does not currently act usefully during clock skew. If time jumps back, then
the queue blocks until time(2) gets back to the previous value. This precludes
any events being delivered unless there is new FD activity, and user code can
not react usefully. Saying "but that shouldn't happen" is not very useful. I
agree it shouldn't happen, but it does happen and must be able to handle the
situation gracefully.

-Philip

Re: POE and the time by Chris Fedde

2012-12-15T05:52:29Z

From: Chris Fedde A few comments about the wiki notes.

First ntpd is careful about ensuring that time always moves forward. It
does this by adjusting the clock slew rather than jumping time forwards or
backwards. When ntpd first starts up and the system clock is not set
within 5 seconds relative to its ntp servers then ntpd will die with a log
message rather than jump the time. Within about an hour of starting ntpd
can have the system synchronized within a few milliseconds of its best
server.

Some admins like to run ntpdate out of cron and force a time reset at
regular intervals. It's never a good idea to force time jumps. It can case
all kinds of trouble for databases, NFS, cryptography, process scheduler
and even event systems written in perl.

Most systems have a separate hardware clocks that is pretty accurate. OS
time is typically synchronized with it at boot then runs independently of
it. Some admins like to synch with hwclock out of cron. This is also a
bad idea. On "wintel" type systems the hardware clock typically reports
only down to seconds. This is much too rough for modern time keeping.,

Virtual machines add another whole layer of complexity to the picture. A
vm will frequently have time jumps between time slices when it gets to run.
Things that require real time behaviors probably should not be run in a
vm. VM guests have to trust their host for an accurate representation of
time. And all the issues that crop up above add another layer of
complexity to accurate time keeping here

There are very few good reasons for administrative clock adjustments.
These days there are no good reasons based on simple calendrical causes.
Soon even the dreaded leap second will be dealt with through math and
lookup tables rather than clock adjustments. Nearly all other reasons to
adjust the clock are political and organizational rather than technical.

If we still want to deal with time sifts we have to detect them. When time
jumps occur it is hard for a program to detect them. The only way that the
program would know that time has changed is if it has two sources to
compare. If such an event has been detected then it has to decide if there
was a good administrative reason for the change or not.

If the program decides to take action because it has detected a time change
then it must have an array of logic to decide what special behaviors to
apply for all of the various ways that the change could have occurred.
Should it fire missed events? Should it recompute the offset for future
events? what if the scale of the shift is large? What if the scale of the
shift is tiny? What if time shifts occur frequently? What if they are
noisy, crossing and recrossing the same point in time? We quickly get into
pathological behaviors that either need to be accounted or ignored.

Most developers who have cracked open this nut have eventually decided to
leave well enough alone and have their system trust that the OS knows what
time it is and that the admin had a "good reason" for mucking with the
system time and leave it at that. Trying to correct for bad admin
behavior is better done at the social layer than at the technical one.

Just my thoughts.
chris

RE: POE and the time by Philip Gwyn

2012-12-15T01:25:07Z

From: Philip Gwyn
The previous patch does not solve the problem properly. I've been thinking
this over for a week now. I've collected my thoughts on the wiki, in my usual
rambling style of writing.

If anyone has any ideas or thoughts about POE and timing, please read and
comment here. In particular, I'd like maintainers of POE::Loop to comment on
the specifics of their implementation.

http://poe.perl.org/?POE_Documentation/Problem_with_the_clock

-Philip

Re: Custom server by Douglas W. Stevenson

2012-12-10T01:34:09Z

From: Douglas W. Stevenson Hry Mark,

While I'm significantly less qualified than many on the POE list, I will give it a shot.

Part of what POE has taught me is to organize my code into more distinct steps where I can stack them up and execute them asynchronously.

Part of the thing with Fork is that when you look under the covers, it clones the process. So, every time you fork, you create a new instance.

With POE, you organize the things you need to do in subroutines accessed via Call backs. When you need to something that blocks, you use a Run Wheel to keep your process working.

Everything starts off with a _start. In this sub, start up your SocketFactory stuff, your Run Wheels, etc. If these need to be done is specific order, use a yield yo callback to the stack a separate subroutine to start the other things.

As things start running and you accept connections, each accept sets up callbacks for each thing you need to do.

Look at delay_add to setup a delayed timer. Wait and sleep block... And when they do, so does everything else.

I built an event management system capable of handling and processing > 1 million events a minute.

HTH,

Dougie!!!

-----Original Message-----
From: Mark Richards
To: poe
Sent: Sun, Dec 9, 2012 3:15 pm
Subject: Custom server

I think POE would be a good choice for what I am about to describe, but
there are a few bits I am not certain about (how to implement; whether
it's possible to implement).

I already have a working model of a server that listens on a designated
port (built in php!). It forks each connection request and handles these
independently. There may be one or 100 clients connecting at any time.
However there is the problem of zombies, and the lack of centralized
control and reporting. I want to build this in perl, and feel POE would
be a good choice.

My server waits for a connection. The client, once connected, sends a
header that contains its ID and WAN IP address. The server reads this,
and responds by either continuing some communications with the client,
or dropping the connection if it is not validated.

A valid connection provides a MODBUS ASCII session with a MODBUS network
attached to the client. So the server instance sends some MODBUS
commands and gets data in response. In the present implementation the
server keeps the connection alive for 10 minutes (this is a design
requirement - don't ask!), and then drops the connection (this is where
we get the zombies from).

What I need POE to do is the same thing, but for this I think I need a
manager of some kind that will, based on the state of the exchange,
dispatch the proper request. Each remote may require a different set of
requests, which further complicates things.

I've thrown together a simple POE version of my server. It simply
accepts the connection and reads in the first data from the client (a
header). What's missing is a way to dispatch requests dynamically and
also to wait without making the whole POE instance wait.

Hoping I have described this adequately, and that the experienced out
there might offer some suggestions as to how to approach this - and
whether POE is the right move.

Thanks!

Custom server by Mark Richards

2012-12-09T21:15:16Z

From: Mark Richards I think POE would be a good choice for what I am about to describe, but
there are a few bits I am not certain about (how to implement; whether
it's possible to implement).

I already have a working model of a server that listens on a designated
port (built in php!). It forks each connection request and handles these
independently. There may be one or 100 clients connecting at any time.
However there is the problem of zombies, and the lack of centralized
control and reporting. I want to build this in perl, and feel POE would
be a good choice.

My server waits for a connection. The client, once connected, sends a
header that contains its ID and WAN IP address. The server reads this,
and responds by either continuing some communications with the client,
or dropping the connection if it is not validated.

A valid connection provides a MODBUS ASCII session with a MODBUS network
attached to the client. So the server instance sends some MODBUS
commands and gets data in response. In the present implementation the
server keeps the connection alive for 10 minutes (this is a design
requirement - don't ask!), and then drops the connection (this is where
we get the zombies from).

What I need POE to do is the same thing, but for this I think I need a
manager of some kind that will, based on the state of the exchange,
dispatch the proper request. Each remote may require a different set of
requests, which further complicates things.

I've thrown together a simple POE version of my server. It simply
accepts the connection and reads in the first data from the client (a
header). What's missing is a way to dispatch requests dynamically and
also to wait without making the whole POE instance wait.

Hoping I have described this adequately, and that the experienced out
there might offer some suggestions as to how to approach this - and
whether POE is the right move.

Thanks!

POE and the time by Philip Gwyn

2012-12-04T21:15:07Z

From: Philip Gwyn A summary for those who don't follow IRC:

- POE keeps a priority queue of pending events
- The current system time is used as the priority
- If the system time is skewed (sysadmin, ntpd), events could happen at the
"wrong time"

An example of this problem would be calling

$poe_kernel->delay( something => 10 );

Then the sysadmin or ntpd moves the time 5 seconds back. The 'something' would
be posted after 15 seconds, not 10 as expected. A pathelogical case is calling

$poe_kernel->yield( 'something' );

And then the sytem time is moved 1 day back. The 'something' would only be
delivered tomorrow!

The included patch adds POE::Resource::Clock which uses POSIX::RT::Clock's
'monotonic' time source as the queue priority and the 'realtime' time source as
the system time.

However, some cheating has to be done; if converting from system time to
monotonic time is not static, then the order following events are delivered is
non-defined.

$poe_kernel->alarm( one => $time );
$poe_kernel->alarm( two => $time );

This patch is not finale; it still needs to detect clock skew and adjust the
queue priorities of alarms (but not delays or other events) when it happens.

-Philip

Re: alarm function under while loop by Nick Perez

2012-11-02T00:50:20Z

From: Nick Perez On Fri, 2 Nov 2012 13:12:58 +0530
Gokul Prasad wrote:

> Hi,

Can you nopaste a complete (non)working test case? Trying to help you
debug this piece meal isn't working. Also, explain what it is you are
trying to achieve? Perhaps you do not need to do something so
complicated.

--

Nicholas Perez
XMPP/Email: nick@nickandperla.net
https://metacpan.org/author/NPEREZ
http://github.com/nperez

Re: alarm function under while loop by Gokul Prasad

2012-11-02T00:43:28Z

From: Gokul Prasad Hi,

To give more context on the issue, the following lines are not
working(alarm function is being called ) but later runs fine

NOT WORKING!!!!! (recursive call thro alarm)
------------------------
delete_sub(){
.
.
.
.

if($heap->{RESULT}->{delete_sub_status} ne 'deleted'){
$heap->{next_delete_alarm_time} = int(time())+30;
# $_[KERNEL]->alarm(get_del_status =>
$heap->{next_delete_alarm_time}) if($end_time >= time);

$_[KERNEL]->alarm(delete_sub =>
$heap->{next_delete_alarm_time}) if($end_time >= time);
$heap->{RESULT}->{delete_sub_status_counter}++;
}

}

WORKING!!!(recursive call thro alarm)
------------------
get_poll_status(){
.
.
.
if($heap->{RESULT}->{poll_success} != 1){
# while($end_time >= time){
$heap->{next_poll_alarm_time} = int(time())+30;
$_[KERNEL]->alarm(get_poll_status =>
$heap->{next_poll_alarm_time}) if($end_time >= time);
$heap->{RESULT}->{poll_status_error_counter}++;
# }
}else{
$_[KERNEL]->yield("get_delivery_status",$pollquerystring);
}

}
Please let me know if am missing something in it!!, it works at one
stage not at other, as i told, i couldnt get it running with
run_while() function, googling didnt gave me much info nor poe
cookbook.

On Fri, Nov 2, 2012 at 12:45 PM, Gokul Prasad wrote:
> If i run using run_whie() it gives me error, will the kernel->alarm()
> work fine wthin if loop?
>
> Even with TRACE_REFCNT am not getting any clue!!!
>
> -gokul
>
>
> On Sat, Aug 25, 2012 at 1:45 PM, Rocco Caputo wrote:
>> Hi, Gokul.
>>
>> POE::Kernel->alarm() enqueues an alarm, but it is not like the built-in alarm(). POE's alarms are dispatched by POE::Kernel in between callbacks. Your while() loop pre-empts the dispatcher, so the enqueued alarms never occur.
>>
>> If your program must delay there, you can use POE::Kernel->run_while() to run POE's dispatcher while a variable is true. Set the variable false from the callback you are waiting for.
>>
>> --
>> Rocco Caputo
>>
>> On Aug 22, 2012, at 05:54, Gokul Prasad wrote:
>>
>>> Hi,
>>>
>>> Am setting "alarm" in calling a function to check if particular task
>>> has executed properly or what.
>>>
>>> when i place the below code under while loop this alarm function is
>>> not being called, but when i comment while loop, it is just works fine
>>> correctly.
>>>
>>> while(end_time >= current_time){
>>> $heap->{next_alarm_time} = int(time())+30;
>>> print STDERR "$heap->{next_alarm_time} why ti is printing\n";
>>> $kernel->alarm(next_task =>
>>> $heap->{next_alarm_time});#$heap->{RESULT}->{event});
>>> }
>>> Not sure why it is happening? any suggestion.please
>>>
>>>
>>> Regards,
>>> gokul
>>

Re: POE, log4perl and sqlite by Antti Linno

2012-10-19T05:59:23Z

From: Antti Linno Ok, thanks for suggestion. No select is taking place, unless I do it
manually and via sqlite client. The crash occurred in the middle of
the night, so my interference can be ruled out :D
Maybe I should move to a file based logging as this seems more robust.

Thank you all for your time.

On Fri, Oct 19, 2012 at 3:46 PM, Rocco Caputo wrote:
> Every module seems to be re-initializing your logger. I don't know if it's harmful in your situation, but it's not recommended. Wherever you call Log::Log4perl::init($log_conf) try Log::Log4perl->init_once($log_conf) instead. See: http://search.cpan.org/~mschilli/Log-Log4perl-1.38/lib/Log/Log4perl.pm#Initialize_once_and_only_once
>
> Offhand I don't see anything else that would lock your log database. Maybe another program is locking it in SELECT? If that's true, the queries are taking too long for the bot to wait. Maybe you could optimize the table indexes so the queries complete in time.
>
> --
> Rocco Caputo
>
> On Oct 19, 2012, at 07:04, Antti Linno wrote:
>
>> Thank you for your kind interest.
>>
>> The pastebins:
>>
>> log4perl.conf http://pastebin.com/nY4twgjS
>> main file http://pastebin.com/ruYsu2qt
>> application logic http://pastebin.com/hMyyBJ0U
>> database interaction http://pastebin.com/Cq7d4geK
>>
>> As you can see, all the perl files are logging, main and application
>> import database functions. I have snipped the application and database
>> somewhat, but you could see the main structure and how I am doing the
>> logging.
>>
>> Greetings.
>>
>> On Thu, Oct 18, 2012 at 4:19 PM, Rocco Caputo wrote:
>>> On Oct 18, 2012, at 07:58, Antti Linno wrote:
>>>
>>>> First of all, I have to apologise. Still no pastebin.
>>>
>>> We have to apologize, too. You're asking questions about code we cannot see. This impairs our ability to provide satisfying answers.
>>>
>>>> But a general tree would be something like that.
>>>>
>>>> Main POE module, imported submodule1, imported submodule2. All 3
>>>> modules open the same logger conf and use the same sqlite file to
>>>> append the log. Theoretically the log file could be locked, if all 3
>>>> or even 2 modules try to write at the same time.
>>>
>>> No. As others have said, POE doesn't pre-empt. Under ordinary circumstances, we expect all three modules to call Log::Log4perl sequentially rather than concurrently. We don't expect Log::Log4perl suffer from lock contention.
>>>
>>> It's very likely that something extraordinary is happening as a result of code we cannot see. Others have already outlined the most likely reasons for the symptoms you described. I hope you understand that's the best we can do without your cooperation.
>>>
>>>> What would be the nice and tidy thing to do? Initialize logger in main
>>>> POE module and pass it to submodules?
>>>
>>> That's such a good question that Log::Log4perl already answers it:
>>>
>>> "Why Log::Log4perl->get_logger and not Log::Log4perl->new? We don't want to create a new object every time. Usually in OO-Programming, you create an object once and use the reference to it to call its methods. However, this requires that you pass around the object to all functions and the last thing we want is pollute each and every function/method we're using with a handle to the Logger:
>>>
>>> [example omitted]
>>>
>>> "Instead, if a function/method wants a reference to the logger, it just calls the Logger's static get_logger($category) method to obtain a reference to the one and only possible logger object of a certain category. That's called a singleton if you're a Gamma fan."
>>>
>>> That quote is from http://search.cpan.org/~mschilli/Log-Log4perl-1.38/lib/Log/Log4perl.pm#Initialize_via_a_configuration_file
>>>
>>> --
>>> Rocco Caputo
>

Re: POE, log4perl and sqlite by Rocco Caputo

2012-10-19T05:46:24Z

From: Rocco Caputo Every module seems to be re-initializing your logger. I don't know if it's harmful in your situation, but it's not recommended. Wherever you call Log::Log4perl::init($log_conf) try Log::Log4perl->init_once($log_conf) instead. See: http://search.cpan.org/~mschilli/Log-Log4perl-1.38/lib/Log/Log4perl.pm#Initialize_once_and_only_once

Offhand I don't see anything else that would lock your log database. Maybe another program is locking it in SELECT? If that's true, the queries are taking too long for the bot to wait. Maybe you could optimize the table indexes so the queries complete in time.

--
Rocco Caputo

On Oct 19, 2012, at 07:04, Antti Linno wrote:

> Thank you for your kind interest.
>
> The pastebins:
>
> log4perl.conf http://pastebin.com/nY4twgjS
> main file http://pastebin.com/ruYsu2qt
> application logic http://pastebin.com/hMyyBJ0U
> database interaction http://pastebin.com/Cq7d4geK
>
> As you can see, all the perl files are logging, main and application
> import database functions. I have snipped the application and database
> somewhat, but you could see the main structure and how I am doing the
> logging.
>
> Greetings.
>
> On Thu, Oct 18, 2012 at 4:19 PM, Rocco Caputo wrote:
>> On Oct 18, 2012, at 07:58, Antti Linno wrote:
>>
>>> First of all, I have to apologise. Still no pastebin.
>>
>> We have to apologize, too. You're asking questions about code we cannot see. This impairs our ability to provide satisfying answers.
>>
>>> But a general tree would be something like that.
>>>
>>> Main POE module, imported submodule1, imported submodule2. All 3
>>> modules open the same logger conf and use the same sqlite file to
>>> append the log. Theoretically the log file could be locked, if all 3
>>> or even 2 modules try to write at the same time.
>>
>> No. As others have said, POE doesn't pre-empt. Under ordinary circumstances, we expect all three modules to call Log::Log4perl sequentially rather than concurrently. We don't expect Log::Log4perl suffer from lock contention.
>>
>> It's very likely that something extraordinary is happening as a result of code we cannot see. Others have already outlined the most likely reasons for the symptoms you described. I hope you understand that's the best we can do without your cooperation.
>>
>>> What would be the nice and tidy thing to do? Initialize logger in main
>>> POE module and pass it to submodules?
>>
>> That's such a good question that Log::Log4perl already answers it:
>>
>> "Why Log::Log4perl->get_logger and not Log::Log4perl->new? We don't want to create a new object every time. Usually in OO-Programming, you create an object once and use the reference to it to call its methods. However, this requires that you pass around the object to all functions and the last thing we want is pollute each and every function/method we're using with a handle to the Logger:
>>
>> [example omitted]
>>
>> "Instead, if a function/method wants a reference to the logger, it just calls the Logger's static get_logger($category) method to obtain a reference to the one and only possible logger object of a certain category. That's called a singleton if you're a Gamma fan."
>>
>> That quote is from http://search.cpan.org/~mschilli/Log-Log4perl-1.38/lib/Log/Log4perl.pm#Initialize_via_a_configuration_file
>>
>> --
>> Rocco Caputo

Re: POE, log4perl and sqlite by Antti Linno

2012-10-19T04:04:39Z

From: Antti Linno Thank you for your kind interest.

The pastebins:

log4perl.conf http://pastebin.com/nY4twgjS
main file http://pastebin.com/ruYsu2qt
application logic http://pastebin.com/hMyyBJ0U
database interaction http://pastebin.com/Cq7d4geK

As you can see, all the perl files are logging, main and application
import database functions. I have snipped the application and database
somewhat, but you could see the main structure and how I am doing the
logging.

Greetings.

On Thu, Oct 18, 2012 at 4:19 PM, Rocco Caputo wrote:
> On Oct 18, 2012, at 07:58, Antti Linno wrote:
>
>> First of all, I have to apologise. Still no pastebin.
>
> We have to apologize, too. You're asking questions about code we cannot see. This impairs our ability to provide satisfying answers.
>
>> But a general tree would be something like that.
>>
>> Main POE module, imported submodule1, imported submodule2. All 3
>> modules open the same logger conf and use the same sqlite file to
>> append the log. Theoretically the log file could be locked, if all 3
>> or even 2 modules try to write at the same time.
>
> No. As others have said, POE doesn't pre-empt. Under ordinary circumstances, we expect all three modules to call Log::Log4perl sequentially rather than concurrently. We don't expect Log::Log4perl suffer from lock contention.
>
> It's very likely that something extraordinary is happening as a result of code we cannot see. Others have already outlined the most likely reasons for the symptoms you described. I hope you understand that's the best we can do without your cooperation.
>
>> What would be the nice and tidy thing to do? Initialize logger in main
>> POE module and pass it to submodules?
>
> That's such a good question that Log::Log4perl already answers it:
>
> "Why Log::Log4perl->get_logger and not Log::Log4perl->new? We don't want to create a new object every time. Usually in OO-Programming, you create an object once and use the reference to it to call its methods. However, this requires that you pass around the object to all functions and the last thing we want is pollute each and every function/method we're using with a handle to the Logger:
>
> [example omitted]
>
> "Instead, if a function/method wants a reference to the logger, it just calls the Logger's static get_logger($category) method to obtain a reference to the one and only possible logger object of a certain category. That's called a singleton if you're a Gamma fan."
>
> That quote is from http://search.cpan.org/~mschilli/Log-Log4perl-1.38/lib/Log/Log4perl.pm#Initialize_via_a_configuration_file
>
> --
> Rocco Caputo

Re: POE, log4perl and sqlite by Rocco Caputo

2012-10-18T06:19:59Z

From: Rocco Caputo On Oct 18, 2012, at 07:58, Antti Linno wrote:

> First of all, I have to apologise. Still no pastebin.

We have to apologize, too. You're asking questions about code we cannot see. This impairs our ability to provide satisfying answers.

> But a general tree would be something like that.
>
> Main POE module, imported submodule1, imported submodule2. All 3
> modules open the same logger conf and use the same sqlite file to
> append the log. Theoretically the log file could be locked, if all 3
> or even 2 modules try to write at the same time.

No. As others have said, POE doesn't pre-empt. Under ordinary circumstances, we expect all three modules to call Log::Log4perl sequentially rather than concurrently. We don't expect Log::Log4perl suffer from lock contention.

It's very likely that something extraordinary is happening as a result of code we cannot see. Others have already outlined the most likely reasons for the symptoms you described. I hope you understand that's the best we can do without your cooperation.

> What would be the nice and tidy thing to do? Initialize logger in main
> POE module and pass it to submodules?

That's such a good question that Log::Log4perl already answers it:

"Why Log::Log4perl->get_logger and not Log::Log4perl->new? We don't want to create a new object every time. Usually in OO-Programming, you create an object once and use the reference to it to call its methods. However, this requires that you pass around the object to all functions and the last thing we want is pollute each and every function/method we're using with a handle to the Logger:

[example omitted]

"Instead, if a function/method wants a reference to the logger, it just calls the Logger's static get_logger($category) method to obtain a reference to the one and only possible logger object of a certain category. That's called a singleton if you're a Gamma fan."

That quote is from http://search.cpan.org/~mschilli/Log-Log4perl-1.38/lib/Log/Log4perl.pm#Initialize_via_a_configuration_file

--
Rocco Caputo

Re: POE, log4perl and sqlite by Antti Linno

2012-10-18T04:58:20Z

From: Antti Linno First of all, I have to apologise. Still no pastebin.

But a general tree would be something like that.

Main POE module, imported submodule1, imported submodule2. All 3
modules open the same logger conf and use the same sqlite file to
append the log. Theoretically the log file could be locked, if all 3
or even 2 modules try to write at the same time.
What would be the nice and tidy thing to do? Initialize logger in main
POE module and pass it to submodules?

I'm amazed, that lockups happens once in a blue moon. Maybe load is
still not that big, that events per second are low.

Thank you for your time.

On Wed, Oct 17, 2012 at 6:38 PM, wrote:
> On Mon, 15 Oct 2012, Antti Linno wrote:
>
>> 1) Do POE's sessions run in parallel? (ok, this might be obvious, but
>> I have to be sure :) This might explain the database lock, as parallel
>> session could write at the same time. Oddly this crash
>> happens rarely.
>
>
> Is using the database spawning multiple processes within POE? POE by
> itself is single-process-single-threaded, but some components spawn
> multiple processes, like this one:
>
>
> http://cpansearch.perl.org/src/XANTUS/POE-Component-EasyDBI-1.23/lib/POE/Component/EasyDBI/SubProcess.pm
>
> If you have multiple processes within POE, then I guess getting a SQLite
> locked situation is possible.
>
>
>> 2) Does POE::Component::Log4perl cure this problem?
>
>
> This component is just a thin wrapper around Log4perl, and doesn't do
> much except provide an easy to use POE-like interface.
>
> If you're using a single-process-single-threaded POE application,
> Log4perl won't have issues with multiple writes to a file. If you have
> multiple processes, there's easy ways to avoid overlapping writes, as
> outlined in the Log4perl FAQ:
>
>
> http://search.cpan.org/dist/Log-Log4perl/lib/Log/Log4perl/FAQ.pm#How_can_I_synchronize_access_to_an_appender?
>
> --
> -- Mike
>
> Mike Schilli
> poe@perlmeister.com
>
>
>>
>> If there is no cure, will there be a problem with file based logging?
>> I can switch from SQL to grep based search.
>>
>> Thanks,
>> Antti
>>
>

Re: POE, log4perl and sqlite by poe

2012-10-17T08:39:18Z

From: poe On Mon, 15 Oct 2012, Antti Linno wrote:

> 1) Do POE's sessions run in parallel? (ok, this might be obvious, but
> I have to be sure :) This might explain the database lock, as parallel
> session could write at the same time. Oddly this crash
> happens rarely.

Is using the database spawning multiple processes within POE? POE by
itself is single-process-single-threaded, but some components spawn
multiple processes, like this one:

http://cpansearch.perl.org/src/XANTUS/POE-Component-EasyDBI-1.23/lib/POE/Component/EasyDBI/SubProcess.pm

If you have multiple processes within POE, then I guess getting a SQLite
locked situation is possible.

> 2) Does POE::Component::Log4perl cure this problem?

This component is just a thin wrapper around Log4perl, and doesn't do
much except provide an easy to use POE-like interface.

If you're using a single-process-single-threaded POE application,
Log4perl won't have issues with multiple writes to a file. If you have
multiple processes, there's easy ways to avoid overlapping writes, as
outlined in the Log4perl FAQ:

http://search.cpan.org/dist/Log-Log4perl/lib/Log/Log4perl/FAQ.pm#How_can_I_synchronize_access_to_an_appender?

--
-- Mike

Mike Schilli
poe@perlmeister.com

>
> If there is no cure, will there be a problem with file based logging?
> I can switch from SQL to grep based search.
>
> Thanks,
> Antti
>

Re: SocketFactory Server communication between sessions by Rocco Caputo

2012-10-17T07:19:01Z

From: Rocco Caputo On Oct 12, 2012, at 09:16, Ty Roden wrote:

>
> The server code, once modified seems to be doing most of what I need, as
> clients can connect, and I can properly respond to events coming down the
> line from clients. In place of the 'echo back anything the client sends
> you' I am examining the data, and responding to it with what I need and all
> seems to be well. The problem is that I have one client that connects that
> sends me data that I need to make sure reaches the other clients so that
> they can process the data arriving. I can accept and process the data from
> this master client, but I cannot find how I might send the data to each
> running process to in turn have their sessions react to the data arriving.
> Since it is a client session just like any other that is receiving the
> data, even saving down the session IDs into a hash I am having an issue
> sending data across to the other sessions. Perhaps this just means that I
> am going about this wrong.

http://poe.perl.org/?POE_Cookbook/Chat_Server seems to do what you want. Whenever one client sends data, it's broadcast to all others. It uses a shared hash to register clients and a global helper function to iterate the hash and broadcast messages.

Of course your version of broadcast() can do more complex calculations and determine which other clients will receive the results. Your shared hash can also contain a lot more information for each client. And since it's shared, any client can inspect it at will... without pesky message passing among clients.

If a shared hash and accessor functions doesn't feel right, consider abstracting it into a singleton class or object, and give every client the package name or instance to the object.

I think this covers all your other questions. Please let us know if it doesn't.

Thanks!

--
Rocco Caputo

Re: How to send a quit message with POE::Component::IRC? by Andy Smith

2012-10-15T10:55:29Z

From: Andy Smith Hello,

On Mon, Oct 15, 2012 at 11:50:34AM +0000, Andy Smith wrote:
> So, is this a bug in POE::Component::IRC, or is Hinrik's code sample
> there incorrect?

..and the answer is neither.

I also sent this to the RT for POE::Component::IRC and Hinrik
replied to that asking if I was doing this on Freenode, as
Freenode's ircd ignores your quit message if you have only been
connected for a short time.

I'm not doing this on Freenode, but it turns out that the ircd of
the network I was doing it on also does this:

https://github.com/atheme/charybdis/blob/master/modules/core/m_quit.c#L75

So, uh, yeah! That was it.

Apologies for the noise.

Cheers,
Andy

How to send a quit message with POE::Component::IRC? by Andy Smith

2012-10-15T04:50:45Z

From: Andy Smith Hi,

I've got a simple IRC bot implemented with POE::Component::IRC and I
decided to add a SIGINT handler to it that gives a useful quit
message.

I couldn't get it to actually send a quit message from the handler,
so I then tried to make it issue a quit message anywhere at all. I
was also unsuccessful in this.

Looking around I found:

http://stackoverflow.com/a/2652671/1394607

Which has some sample code from Hinrik showing how to do such a
handler. Unfortunately this code also does not send a quit message.
It always quits with either no message ("Client Quit") or else
"Remote host closed the connection". That's the exact same problem I
was having.

I'm using the package libpoe-component-irc-perl version 6.78+dfsg-1
on Debian 6.0, but I also got the latest POE::Component::IRC from
CPAN and have the same issue.

So, is this a bug in POE::Component::IRC, or is Hinrik's code sample
there incorrect?

I did also find:

http://search.cpan.org/~bingos/POE-Component-IRC-6.79/lib/POE/Component/IRC/Cookbook/Disconnecting.pod

but was similarly unsuccessful with "shutdown".

Cheers,
Andy

Re: POE, log4perl and sqlite by Nick Perez

2012-10-15T01:19:20Z

From: Nick Perez On Mon, 15 Oct 2012 10:11:32 +0300
Antti Linno wrote:

> 1) Do POE's sessions run in parallel?

Not unless you are forking.

> 2) Does POE::Component::Log4perl cure this problem?

Unknown. We need to see some code.

I could speculate on a number of things potentially causing this issue.
Are you spinning up more than one process? Do you have any children
processes that are running? Do you use any other mechanism to read from
the sqlite file? What does your database code look like? Are you using
alarm() (perl's alarm, not POE::Kernel's).

Anyhow without any code it is difficult to debug. Please nopaste some
code so we can help you.

--

Nicholas Perez
XMPP/Email: nick@nickandperla.net
https://metacpan.org/author/NPEREZ
http://github.com/nperez

POE, log4perl and sqlite by Antti Linno

2012-10-15T00:11:40Z

From: Antti Linno Good day.

I'm using a bit modified Sungo's TCP server (Using SocketFactory and
ReadWrite). I'm using a plain Log4perl, not POE::Log4perl. Yesterday I
got a crash with sqlite database lock.

So basically I have 2 questions:

1) Do POE's sessions run in parallel? (ok, this might be obvious, but
I have to be sure :) This might explain the database lock, as parallel
session could write at the same time. Oddly this crash
happens rarely.
2) Does POE::Component::Log4perl cure this problem?

If there is no cure, will there be a problem with file based logging?
I can switch from SQL to grep based search.

Thanks,
Antti

SocketFactory Server communication between sessions by Ty Roden

2012-10-12T06:16:25Z

From: Ty Roden I am still rather new to POE, but I am charging ahead and attempting to
write a tool I need using it, but I am having an issue with communication
that I am sure someone can assist with.

I started with the code here:
http://poe.perl.org/?POE_Cookbook/TCP_Servers- the "Using Socket
Factory and ReadWrite" example at the bottom.

The server code, once modified seems to be doing most of what I need, as
clients can connect, and I can properly respond to events coming down the
line from clients. In place of the 'echo back anything the client sends
you' I am examining the data, and responding to it with what I need and all
seems to be well. The problem is that I have one client that connects that
sends me data that I need to make sure reaches the other clients so that
they can process the data arriving. I can accept and process the data from
this master client, but I cannot find how I might send the data to each
running process to in turn have their sessions react to the data arriving.
Since it is a client session just like any other that is receiving the
data, even saving down the session IDs into a hash I am having an issue
sending data across to the other sessions. Perhaps this just means that I
am going about this wrong.

Using the example code above, here is an example of how I am trying to
implement this:

sub socket_input {
my ($heap, $buf) = @_[HEAP, ARG0];
$buf =~ s/[\r\n]//gs;

#Pseudocode here to explain - not actual code.
if ($buf =~ m/matchcriteria/g) { send_to_all_clients($buf); };
}

This really just means that I need to not just process what arrives, but be
able to make other sessions process what arrives from the master client. Do
I have to build a session that is not a normal client, and bind that port
manually to make sure that it is high enough in the hierarchy to send to
all of the users using their session ID? No matter how much I have read on
the topic, I am having difficulty understanding how I might accomplish this.

The only other issue, is that I would like to be able to call a function
for each client to occasionally check local variables to make decisions on
what to do. As it sits now, the clients are purely driven by the data
arriving in the port - they examine arriving data, then do something. I
would like to add a function to have the clients regularly check a variable
for data to determine if they need to initiate a send to the connected
remote user.

Thank you all for any information you can provide.

-Ty Roden

Re: patch for SIG{DIE} behaviour by Rocco Caputo

2012-09-27T12:01:54Z

From: Rocco Caputo I redirected it to POE's request queue at bug-poe@rt.cpan.org. It's tracked at https://rt.cpan.org/Public/Bug/Display.html?id=79886 now.

--
Rocco Caputo

On Sep 26, 2012, at 10:46, Philip Gwyn wrote:

> Hello,
>
> This is a patch that fixes $SIG{__DIE__} handling for perl 5.8.8. While
> __DIE__ starts out as undef, if you assign undef to __DIE__ in 5.8.8, it
> becomes an empty string ("").
>
> This patch includes a regression test.
>
> -Philip
>
>

patch for SIG{DIE} behaviour by Philip Gwyn

2012-09-26T07:46:42Z

From: Philip Gwyn Hello,

This is a patch that fixes $SIG{__DIE__} handling for perl 5.8.8. While
__DIE__ starts out as undef, if you assign undef to __DIE__ in 5.8.8, it
becomes an empty string ("").

This patch includes a regression test.

-Philip

POE::Component::Client::HTTP blocking by Farow

2012-08-30T01:27:13Z

From: Farow Hello,

I've been trying to write an irc bot using POE::Component::IRC. Some
functions would require getting information from the web. I was able to
implement a simple way to do http get requests however, they seem to be
blocking. Here's the code: http://pastebin.com/E41fQDcb

A few notes on the code: It is based on
https://metacpan.org/module/POE::Component::IRC::Plugin::Proxy although
that shouldn't matter a lot. It waits for someone to say .test in an irc
channel and when someone does, it sends a message processes an http
request. The problem is that it is supposed to send the message instantly
but it actually sends it after the request is completed and I can't figure
out why.

I'm not very experienced with POE. Am I missing something obvious (like
the request being processed in the irc session for some reason) or is it a
Windows specific bug? Or perhaps is it POE::Component::IRC's fault?

I'm using ActiveState Perl 5.14.2 on Windows XP.

Thanks in advance,
Farow

RE: POE process getting too busy by Markus Jansen

2012-08-26T15:27:28Z

From: Markus Jansen Hi Rocco,

thanks for your hints - would it help to start a debuggable run, e.g. with enbugger and trepan,
to find out where the problem happens?

Best regards,
Markus

-----Original Message-----
From: Rocco Caputo [mailto:rcaputo@pobox.com]
Sent: den 25 augusti 2012 10:37
To: POE Mailing List
Subject: Re: POE process getting too busy

Since this problem is platform-specific, the best I can manage without access to the machine is advice.

Web searches imply that Solaris may be broken, the program may have run out of file descriptors, an OS upgrade may have broken compatibility with an older build of Perl, or some other issue:

https://community.emc.com/thread/111914
http://dbaspot.com/solaris/242660-high-cpu-utilization-solaris-10-a.html
http://www.mail-archive.com/dovecot@dovecot.org/msg04283.html
http://developerweb.net/viewtopic.php?id=5481
http://www.symantec.com/business/support/index?page=content&pmv=print&impressions=&viewlocale=&id=TECH163245
http://wesunsolve.net/bugid/id/6404383

--
Rocco Caputo

On Aug 21, 2012, at 06:09, Markus Jansen wrote:

> Hi,
>
> thanks a lot for the immediate answer, I also changed the code immediately, and started another long-term test run.
> Just found the process 99% busy again - the good news is that the signal pipe does not seem to be the culprit :-).
>
> Any other suggestions are highly welcome ...
>
> Thanks & best regards,
> Markus
>
> -----Original Message-----
> From: Tod McQuillin [mailto:devin@spamcop.net]
> Sent: den 16 augusti 2012 02:46
> To: Markus Jansen
> Cc: poe@perl.org
> Subject: Re: POE process getting too busy
>
> On Wed, 15 Aug 2012, Markus Jansen wrote:
>
>> I have discovered an annoying phenomenon, where after some while
>> (days, weeks) a process (which is part of a multi-server POE network) all of a sudden goes berzerk by using all resources of its CPU.
>> The good news is that the process itself remains fully functional, but from a system usage/load perspective, the situation is quite intolerable.
>
> This sounds very familiar to me.
>
> I encountered a similar problem on solaris and was able to make it stop by putting the following line of code at the beginning of my POE program:
>
> BEGIN { eval "sub POE::Kernel::USE_SIGNAL_PIPE () { 0 }" }
>
> There seems to be a problem with the signal pipe on some platforms, but I haven't yet done more research to track down what's going wrong.
> --
> Tod

Re: POE process getting too busy by Rocco Caputo

2012-08-25T01:36:50Z

From: Rocco Caputo Since this problem is platform-specific, the best I can manage without access to the machine is advice.

Web searches imply that Solaris may be broken, the program may have run out of file descriptors, an OS upgrade may have broken compatibility with an older build of Perl, or some other issue:

https://community.emc.com/thread/111914
http://dbaspot.com/solaris/242660-high-cpu-utilization-solaris-10-a.html
http://www.mail-archive.com/dovecot@dovecot.org/msg04283.html
http://developerweb.net/viewtopic.php?id=5481
http://www.symantec.com/business/support/index?page=content&pmv=print&impressions=&viewlocale=&id=TECH163245
http://wesunsolve.net/bugid/id/6404383

--
Rocco Caputo

On Aug 21, 2012, at 06:09, Markus Jansen wrote:

> Hi,
>
> thanks a lot for the immediate answer, I also changed the code immediately, and started another long-term test run.
> Just found the process 99% busy again - the good news is that the signal pipe does not seem to be the culprit :-).
>
> Any other suggestions are highly welcome ...
>
> Thanks & best regards,
> Markus
>
> -----Original Message-----
> From: Tod McQuillin [mailto:devin@spamcop.net]
> Sent: den 16 augusti 2012 02:46
> To: Markus Jansen
> Cc: poe@perl.org
> Subject: Re: POE process getting too busy
>
> On Wed, 15 Aug 2012, Markus Jansen wrote:
>
>> I have discovered an annoying phenomenon, where after some while
>> (days, weeks) a process (which is part of a multi-server POE network) all of a sudden goes berzerk by using all resources of its CPU.
>> The good news is that the process itself remains fully functional, but from a system usage/load perspective, the situation is quite intolerable.
>
> This sounds very familiar to me.
>
> I encountered a similar problem on solaris and was able to make it stop by putting the following line of code at the beginning of my POE program:
>
> BEGIN { eval "sub POE::Kernel::USE_SIGNAL_PIPE () { 0 }" }
>
> There seems to be a problem with the signal pipe on some platforms, but I haven't yet done more research to track down what's going wrong.
> --
> Tod

Re: alarm function under while loop by Rocco Caputo

2012-08-25T01:15:58Z

From: Rocco Caputo Hi, Gokul.

POE::Kernel->alarm() enqueues an alarm, but it is not like the built-in alarm(). POE's alarms are dispatched by POE::Kernel in between callbacks. Your while() loop pre-empts the dispatcher, so the enqueued alarms never occur.

If your program must delay there, you can use POE::Kernel->run_while() to run POE's dispatcher while a variable is true. Set the variable false from the callback you are waiting for.

--
Rocco Caputo

On Aug 22, 2012, at 05:54, Gokul Prasad wrote:

> Hi,
>
> Am setting "alarm" in calling a function to check if particular task
> has executed properly or what.
>
> when i place the below code under while loop this alarm function is
> not being called, but when i comment while loop, it is just works fine
> correctly.
>
> while(end_time >= current_time){
> $heap->{next_alarm_time} = int(time())+30;
> print STDERR "$heap->{next_alarm_time} why ti is printing\n";
> $kernel->alarm(next_task =>
> $heap->{next_alarm_time});#$heap->{RESULT}->{event});
> }
> Not sure why it is happening? any suggestion.please
>
>
> Regards,
> gokul

alarm function under while loop by Gokul Prasad

2012-08-22T02:54:52Z

From: Gokul Prasad Hi,

Am setting "alarm" in calling a function to check if particular task
has executed properly or what.

when i place the below code under while loop this alarm function is
not being called, but when i comment while loop, it is just works fine
correctly.

while(end_time >= current_time){
$heap->{next_alarm_time} = int(time())+30;
print STDERR "$heap->{next_alarm_time} why ti is printing\n";
$kernel->alarm(next_task =>
$heap->{next_alarm_time});#$heap->{RESULT}->{event});
}
Not sure why it is happening? any suggestion.please

Regards,
gokul

RE: POE process getting too busy by Markus Jansen

2012-08-21T03:10:08Z

From: Markus Jansen Hi,

thanks a lot for the immediate answer, I also changed the code immediately, and started another long-term test run.
Just found the process 99% busy again - the good news is that the signal pipe does not seem to be the culprit :-).

Any other suggestions are highly welcome ...

Thanks & best regards,
Markus

-----Original Message-----
From: Tod McQuillin [mailto:devin@spamcop.net]
Sent: den 16 augusti 2012 02:46
To: Markus Jansen
Cc: poe@perl.org
Subject: Re: POE process getting too busy

On Wed, 15 Aug 2012, Markus Jansen wrote:

> I have discovered an annoying phenomenon, where after some while
> (days, weeks) a process (which is part of a multi-server POE network) all of a sudden goes berzerk by using all resources of its CPU.
> The good news is that the process itself remains fully functional, but from a system usage/load perspective, the situation is quite intolerable.

This sounds very familiar to me.

I encountered a similar problem on solaris and was able to make it stop by putting the following line of code at the beginning of my POE program:

BEGIN { eval "sub POE::Kernel::USE_SIGNAL_PIPE () { 0 }" }

There seems to be a problem with the signal pipe on some platforms, but I haven't yet done more research to track down what's going wrong.
--
Tod

perl.poe

Re: poe and amqp/rabbitmq as a worker manager by Nick Perez

Re: poe and amqp/rabbitmq as a worker manager by S Chaitanya

poe and amqp/rabbitmq as a worker manager by Kevin Goess

Re: Wait for first recv. by Rizwan Hisham

Wait for first recv. by yakudzo

Re: Combining services. by Antti Linno

Re: Combining services. by Rocco Caputo

Re: Combining services. by Antti Linno

Re: Combining services. by Antti Linno

Re: Combining services. by Rocco Caputo

Combining services. by Antti Linno

Re: POE and the time by Jeremy Leader

Re: POE and the time by Philip Gwyn

Re: POE and the time by Chris Fedde

RE: POE and the time by Philip Gwyn

Re: Custom server by Douglas W. Stevenson

Custom server by Mark Richards

POE and the time by Philip Gwyn

Re: alarm function under while loop by Nick Perez

Re: alarm function under while loop by Gokul Prasad

Re: POE, log4perl and sqlite by Antti Linno

Re: POE, log4perl and sqlite by Rocco Caputo

Re: POE, log4perl and sqlite by Antti Linno

Re: POE, log4perl and sqlite by Rocco Caputo

Re: POE, log4perl and sqlite by Antti Linno

Re: POE, log4perl and sqlite by poe

Re: SocketFactory Server communication between sessions by Rocco Caputo

Re: How to send a quit message with POE::Component::IRC? by Andy Smith

How to send a quit message with POE::Component::IRC? by Andy Smith

Re: POE, log4perl and sqlite by Nick Perez

POE, log4perl and sqlite by Antti Linno

SocketFactory Server communication between sessions by Ty Roden

Re: patch for SIG{__DIE__} behaviour by Rocco Caputo

patch for SIG{__DIE__} behaviour by Philip Gwyn

POE::Component::Client::HTTP blocking by Farow

RE: POE process getting too busy by Markus Jansen

Re: POE process getting too busy by Rocco Caputo

Re: alarm function under while loop by Rocco Caputo

alarm function under while loop by Gokul Prasad

RE: POE process getting too busy by Markus Jansen

Re: patch for SIG{DIE} behaviour by Rocco Caputo

patch for SIG{DIE} behaviour by Philip Gwyn