develooper Front page | perl.poe | Postings from April 2012

Multiple LWP::UserAgents at the same time

Thread Next
From:
Gabor Szabo
Date:
April 3, 2012 06:58
Subject:
Multiple LWP::UserAgents at the same time
Message ID:
CABe4FJDd2yEV9ShcLqC-TjEsuajDKDuPhsfDHh1RJh1C042r3Q@mail.gmail.com
Hi,

I am trying to run multiple LWP::UserAgents at the same time but I
don't seem to be able to get that work.
It seems the 1st session waited for the others and even the others
seemed to be stuck
(11 sec seem to be too much getting those pages)

See code and output below.

regards
   Gabor


use strict ;
use warnings ;

use POE;
use LWP::UserAgent::POE;


my @urls = qw(
	http://perl.org/
	http://jobs.perl.org/
	http://www.perl.org/
	http://blogs.perl.org/
	http://moose.perl.org/
	http://dbi.perl.org/
	http://debugger.perl.org/
);

for (1 .. 3) {
  POE::Session->create(
    inline_states => {
      _start  => \&handler_start,
      get     => \&handler_get,
      count   => \&handler_count,
      _stop   => \&handler_stop,
    }
  );
}

POE::Kernel->run();
exit;

sub handler_start {
	my ($kernel, $heap, $session) = @_[KERNEL, HEAP, SESSION];
	print "Session ", $session->ID, " has started.\n";
	$heap->{cnt} = 0;
	$kernel->yield('get');
}

sub handler_get {
	my ($kernel, $heap, $session) = @_[KERNEL, HEAP, SESSION];
	my $url = shift @urls;
	if ($url) {
		$heap->{url} = $url;
		$kernel->yield('count');
		print "Session ", $session->ID, " $url\n";
		my $t0 = time;
		my $ua = LWP::UserAgent::POE->new(timeout => 10);
		my $response = $ua->get($url) ;
		print "Session ", $session->ID, " $url DONE ", (time - $t0) , "\n";
		delete $heap->{url};
		$kernel->yield('get');
	} else {
		print "We ran out of urls " . $session->ID, "\n";
	}
}

sub handler_count {
	my ($kernel, $heap, $session) = @_[KERNEL, HEAP, SESSION];
	$heap->{cnt}++;
	print "Session " . $session->ID . " count $heap->{cnt}\n";
	if ($heap->{url}) {
		$kernel->delay( count => 1 );
	}
}

sub handler_stop {
	print "Session ", $_[SESSION]->ID, " has stopped.\n";
}


==========================
The result is


Session 1 has started.
Session 2 has started.
Session 3 has started.
Session 1 http://perl.org/
Session 2 http://jobs.perl.org/
Session 3 http://www.perl.org/
Session 1 count 1
Session 2 count 1
Session 3 count 1
Session 1 count 2
Session 2 count 2
Session 3 count 2
Session 1 count 3
Session 2 count 3
Session 3 count 3
Session 1 count 4
Session 2 count 4
Session 3 count 4
Session 1 count 5
Session 2 count 5
Session 3 count 5
Session 1 count 6
Session 2 count 6
Session 3 count 6
Session 1 count 7
Session 2 count 7
Session 3 count 7
Session 1 count 8
Session 2 count 8
Session 3 count 8
Session 1 count 9
Session 2 count 9
Session 3 count 9
Session 1 count 10
Session 2 count 10
Session 3 count 10
Session 1 count 11
Session 2 count 11
Session 3 count 11
Session 1 count 12
Session 2 count 12
Session 3 count 12
Session 3 http://www.perl.org/ DONE 11
Session 2 http://jobs.perl.org/ DONE 11
Session 3 http://blogs.perl.org/
Session 2 http://moose.perl.org/
Session 3 count 13
Session 2 count 13
Session 1 count 13
Session 3 count 14
Session 2 count 14
Session 2 http://moose.perl.org/ DONE 1
Session 3 http://blogs.perl.org/ DONE 1
Session 2 http://dbi.perl.org/
Session 3 http://debugger.perl.org/
Session 2 count 15
Session 3 count 15
Session 1 count 14
Session 2 count 16
Session 3 count 16
Session 3 http://debugger.perl.org/ DONE 1
We ran out of urls 3
Session 1 count 15
Session 2 count 17
Session 3 count 17
Session 1 count 16
Session 2 count 18
Session 1 count 17
Session 2 count 19
Session 1 count 18
Session 2 count 20
Session 1 count 19
Session 2 count 21
Session 1 count 20
Session 2 count 22
Session 1 count 21
Session 2 count 23
Session 1 count 22
Session 2 count 24
Session 1 count 23
Session 2 count 25
Session 1 count 24
Session 2 count 26
Session 2 http://dbi.perl.org/ DONE 12
Session 1 http://perl.org/ DONE 24
We ran out of urls 2
We ran out of urls 1
Session 1 count 25
Session 2 count 27
Session 2 has stopped.
Session 3 has stopped.
Session 1 has stopped.

Thread Next


nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About