topic Re: Retrieve a webpage from a shell script in Operating System - HP-UX

Retrieve a webpage from a shell script

John_880 — Wed, 10 Nov 2004 05:12:29 GMT

O wise ones,

I'm looking for a way to retrieve a webpage from a shellscript or command line. The webpage contains basic system info like output from 'bdf' but I do not have a prompt login on that machine as it is fully serviced by others.

How do i get hold of that system info page on a daily basis best? I tried perl but we don't have the LWP module to do something like:

#!/usr/contrib/bin/perl
use LWP::Simple;
$AS=get('http://somelanhost/cgi-bin/solaris_cmd?df');
print $AS;

I'm not allowed to install tools like 'lynx' et al to retrieve the info.

Has hp-ux 11.11 any standard tools to retrieve a webpage from the command line?

Thanks in advance, John

Re: Retrieve a webpage from a shell script

Sridhar Bhaskarla — Wed, 10 Nov 2004 05:15:34 GMT

Hi John,

Try 'wget' from here.

http://hpux.cs.utah.edu/hppd/hpux/Gnu/wget-1.9.1/

There are few more tools available in the same web site. Go to home and search for web.

-Sri

Re: Retrieve a webpage from a shell script

Fred Ruffet — Wed, 10 Nov 2004 05:47:21 GMT

AS said by Sridhar, wget is probably your solution. It has very interresting options such as background download, recursive download, continue after error, bandwidth limitation and so on.

Regards,

Fred

Re: Retrieve a webpage from a shell script

H.Merijn Brand (procura — Wed, 10 Nov 2004 06:15:48 GMT

If you cannot use lynx, you can also probably not use wget, how phenomenal and right it would be.

You can however use Perl, as your post shows, but as you do not have LWP::Simple, just fake it using CORE modules:

--8<---
#!/pro/bin/perl -w

use strict;
use IO::Socket::INET;

my $itrcweb = "http://forums1.itrc.hp.com/service/forums";
my $server = "forums1.itrc.hp.com";
my $ux_url = "service/forums";

sub webget ($;$)
{
my ($url, $F) = @_;
local $/;

my $peer = $url =~ s{^(\w+)://}{} ? $1 : "http(80)";
my ($server, $page) = ($url =~ m{^([^/]+)(?:/(.*)});

(my $f = $page) =~ s:.*/::;
$F ||= $f;
unlink $F;

printf STDERR "get %-38s\t", $F;
my $socket = IO::Socket::INET->new (
PeerHost => $server,
PeerPort => $peer,
Proto => "tcp",
timeout => 90) or die;
print $socket
"GET $page\n",
"Host: $server\n",
"User-Agent: Internet Explorer\n",
"\n";

my $t = 0;
my $data = <$socket>;
if (ref $F) {
$$F = $data;
return $t;
}
if ($data) {
$t = length $data;
undef $p;
open $p, "> $F" or die "$F: $!\n";
print $p $data;
close $p;
}
$t;
} # webget

webget ("http://somehost.lan/cgi-bin/solaris_cmd?df", "solaris.df");

my $sol_df;
webget ("http://somehost.lan/cgi-bin/solaris_cmd?df", \$sol_df);

-->8---

(Not tested)

Enjoy, Have FUN! H.Merijn

Re: Retrieve a webpage from a shell script

Fred Ruffet — Wed, 10 Nov 2004 06:18:49 GMT

You can't install, but you can compile... just get sources (either for wget or perl modules) compile, and run from your directory.

Regards,

Fred

Re: Retrieve a webpage from a shell script

Baz_2 — Wed, 10 Nov 2004 07:56:44 GMT

I've used wget and cURL which work fine.

http://www.gnu.org/software/wget/wget.html
http://curl.haxx.se/

Re: Retrieve a webpage from a shell script

Michael Roberts_3 — Wed, 10 Nov 2004 10:27:40 GMT

you can also use /usr/bin/sh and telnet:

#!/sbin/sh
if [[ $# -lt 1 ]] ;then
print "supply stock symbol on command line"
exit 1
fi
for sym in $@
{
(stelnet quote.yahoo.com 80 << eof
GET http://quote.yahoo.com/download/quotes.csv?symbols=${sym}&format=sl1d1t1

eof
) 2>/dev/null |awk '$0 ~ /\"/ {print}'
}

Re: Retrieve a webpage from a shell script

rmueller58 — Wed, 10 Nov 2004 15:35:14 GMT

I'd just use "wget"

wget http://hostname.ext/page.html

if you need to set it up where it's a timed thing then set up a cron job to do it.

You may want to Google "hcat" and "hgrepurl."

these are little perl scripts that OReilly press has on their FTP server. If you can't find them there I am attaching a tarball.

Re: Retrieve a webpage from a shell script

John_880 — Wed, 10 Nov 2004 20:46:23 GMT

Hi all, thanks for the replies. I changed to a machine with the proper perl modules and all is fine. I will definately try to get wget installed.

Cheers! John