develooper Front page | perl.libwww | Postings from April 2001

How do I use webchat to do webscrapping

From:
ben goswami
Date:
April 24, 2001 07:14
Subject:
How do I use webchat to do webscrapping
Message ID:
004b01c0ccc8$d43766e0$2b0b0c18@smateo1.sfba.home.com
Hi,
I'm trying to use webchat to download content from web and write them to file.  I could not make much progress. 
my questions are :
1. how do I print the $res (the content) from the site to std out or to a file 
2. do you have more docs on how to write the 'chat scripts' for advance usage
3. How do I print the variables if 
4. how do I see what script has been generated before it is fed to perl
5. Should I modify the webchat.pl if I want to extend the functionality
here is what I'm tryin to do:

1. go to a ur:(ex: www.epaynews.com)
2. get the links for head lines (they start with pattern "<A HREF="/index.cgi?survey=&ref=browse&f=view&id=")
3. go to each of those links and download content and parse the content from <head line title> to <Related links> and write to a file. 
4. do it for all the headline links and write the contents to a single file.
thx ben




nntp.perl.org: Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at ask@perl.org | Group listing | About