topic Download web page from cron in Operating System - Linux

Download web page from cron

Vladimir Fabecic — Wed, 13 Sep 2006 07:40:43 GMT

Hello
I need to download a web page (to a text file) from a cron script. Web page is also a text file.
Any good idea from where to start?

Re: Download web page from cron

spex — Wed, 13 Sep 2006 07:44:19 GMT

Hi,

I would use curl or wget.

http://hpux.connect.org.uk/hppd/hpux/Networking/WWW/curl-7.15.5/
http://hpux.connect.org.uk/hppd/hpux/Gnu/wget-1.10.2/

PCS

Re: Download web page from cron

spex — Wed, 13 Sep 2006 07:46:55 GMT

A textual web browser, such as lynx or links would also do the trick.

http://hpux.connect.org.uk/hppd/hpux/Networking/WWW/lynx-2.8.5/
http://hpux.connect.org.uk/hppd/hpux/Networking/WWW/links-0.99/

Re: Download web page from cron

Murat SULUHAN — Wed, 13 Sep 2006 07:52:58 GMT

Hi Vladimir,

wget http:/www.hp.com/index.html -O myTextFile.txt

Best Regards

Re: Download web page from cron

Peter Godron — Wed, 13 Sep 2006 07:56:12 GMT

Vladimir,
I installed wget, but ran into problems as our company does not allow direct access to external Web (proxy setup).
Check this before you start !

Re: Download web page from cron

spex — Wed, 13 Sep 2006 08:02:28 GMT

If you don't want to install new software, telnet will also work.

#!/usr/bin/sh

{
telnet www.google.com 80 << _EOF_
GET / HTTP/1.1

exit
_EOF_
} > index.html

exit

Re: Download web page from cron

Murat SULUHAN — Wed, 13 Sep 2006 08:13:25 GMT

Hi Spex

thanks for excellent tip

Best Regards

Re: Download web page from cron

Vladimir Fabecic — Wed, 13 Sep 2006 09:51:01 GMT

Thanks guys
I forgot wget. Very good idea.
Solution with telnet is also good.

Re: Download web page from cron

Arunvijai_4 — Wed, 13 Sep 2006 13:56:59 GMT

Hi Vladimir,

I would also consider HTTrack if it going to be a linux machine.

-Arun

Re: Download web page from cron

Vladimir Fabecic — Thu, 14 Sep 2006 01:00:17 GMT

No, it is a HP-UX machine. Wget idea is perfect.
HTTrack? Is there something like that for HP-UX?

Re: Download web page from cron

Yogeeraj_1 — Thu, 14 Sep 2006 06:13:23 GMT

hi vladmir,

for httrack, you may wish to download the source and compile it for HP-UX. It may require some modifications to the makefile.

see: http://www.httrack.com/html/faq.html#QG6

kind regards
yogeeraj