develooper Front page | perl.beginners | Postings from November 2019

Re: reading data from a web site

Thread Previous | Thread Next
November 20, 2019 02:37
Re: reading data from a web site
Message ID:
hw <> writes:

> Hi,
> how can I read data from a web site which is using multiple frames and some 
> javascript?

Provided that the web site does not change too often and that they don't
implement stupid "security" features, this should not be too complicate.

Each frame is a web page, with it own URL. So you can examine the source
code of the web page to find the URL of the first frame and second frame

Them you can use any Perl library you like to load that URLand pars it
for what you are looking for.

Then use that data to load the second frame with a URL modified to
include the type of data you have selected.

Being frames makes it much easier, you hould not have to care about the
javascript too much.


> When using a web browser, I need to log in into the web site and follow a 
> couple links to finally get to the page I want.  This page I need to get data 
> from has a frame with a select list and another frame displaying a table.  By 
> default, the table contains data corresponding to the first entry in the 
> select list.  Selecting an entry from the select list reloads the table in the 
> other frame once I click on the entry.
> I need to automatically pick all the entries from the select list one after 
> another so that the table is being updated.  Once I've read the new table, the 
> next entry in the select list needs to be picked to get the table updated, 
> until there are no more entries.  The order in which the entries are being 
> picked can be random.
> Once all the available tables are read, only the first entry in the select 
> needs to be read once per day.  "Reading the table" means that I need to put 
> the data in the table into a database.  It would help if could save all the 
> tables to files and convert the files later; the problem is getting the 
> tables.
> Is this even possible?  There doesn't seem be any useful support for 
> javascript with WWW::Mechanize, and even frames seem to be an issue.
> I can only see that some java (or javascript or whatever it is) function is 
> being called when clicking on an entry in the select list the only purpose of 
> which seems to be to display a banner showing "Loading ..." with the dots 
> moving while the table is being loaded.  Yet the table is being updated.  I 
> can't see how and if there are GET or POST requests being sent by the web 
> browser.  The only way to update seems to be to somehow fake a click on an 
> entry in the select list.
> Any ideas?


Thread Previous | Thread Next Perl Programming lists via nntp and http.
Comments to Ask Bjørn Hansen at | Group listing | About