How to automate opening Webpage and copying the contents into a file in the text format using perl

[sowm18]

Hi all,

Here i wanted to automate opening the web page and copy the contents into a file in txt format.Please let me know how to do in linux.

Regards,
BS

[Paul Cross_1 · Accepted solution]

If you want to retain the html tags, wget would do the trick, otherwise you could pipe it to html2txt or something to strip out the html tags. If you need to do this inside perl, wget could be called inside a perl script with HTML::strip to remove the tags.

[Yogeeraj_1]

hi BS,

try the solution as proposed by Paul above - wget

followed by one "simple-minded" approach, that works for most files:

#!/usr/bin/perl -p0777
s/<(?:[^>'"]*|(['"]).*?\1)*>//gs


hope this helps!
kind regards
yogeeraj

[Court Campbell]

http://search.cpan.org/~gaas/libwww-perl-5.812/lib/LWP/Simple.pm

[sowm18]

Hi again,

if the web page needs authentication with username and password then how to takecare of that.

Regards,
BS

[Steven Schweda]

> [...] how to takecare of that.

wget -h

Look for options like:

[...]
HTTP options:
--http-user=USER set http user to USER.
--http-password=PASS set http password to PASS.
[...]

[James R. Ferguson]

Hi:

If your objective is only to snapshot a webpage and copy it to a file in text format, 'wget' is probably the simplest way.

http://hpux.cs.utah.edu/hppd/hpux/Gnu/wget-1.11.1/

The LWP::Simple module from CPAN provides the 'getstore' function to do the same:

http://search.cpan.org/~gaas/libwww-perl-5.812/lib/LWP/Simple.pm

Have a look, too, at:

http://search.cpan.org/~gaas/libwww-perl-5.812/lwpcook.pod

If you are serious about parsing HTML, though, I suggest you look at the HTML::TreeBuilder module beginning with:

http://search.cpan.org/~petek/HTML-Tree-3.23/lib/HTML/Tree.pm

and:

http://search.cpan.org/~petek/HTML-Tree-3.23/lib/HTML/TreeBuilder.pm

Regards!

...JRF...