Retrieve a webpage from a shell script

[John_880]

O wise ones,

I'm looking for a way to retrieve a webpage from a shellscript or command line. The webpage contains basic system info like output from 'bdf' but I do not have a prompt login on that machine as it is fully serviced by others.

How do i get hold of that system info page on a daily basis best? I tried perl but we don't have the LWP module to do something like:

#!/usr/contrib/bin/perl
use LWP::Simple;
$AS=get('http://somelanhost/cgi-bin/solaris_cmd?df');
print $AS;

I'm not allowed to install tools like 'lynx' et al to retrieve the info.

Has hp-ux 11.11 any standard tools to retrieve a webpage from the command line?

Thanks in advance, John

[Sridhar Bhaskarla]

Hi John,

Try 'wget' from here.

http://hpux.cs.utah.edu/hppd/hpux/Gnu/wget-1.9.1/

There are few more tools available in the same web site. Go to home and search for web.

-Sri

[Fred Ruffet]

AS said by Sridhar, wget is probably your solution. It has very interresting options such as background download, recursive download, continue after error, bandwidth limitation and so on.

Regards,

Fred

[H.Merijn Brand (procura]

If you cannot use lynx, you can also probably not use wget, how phenomenal and right it would be.

You can however use Perl, as your post shows, but as you do not have LWP::Simple, just fake it using CORE modules:

--8<---
#!/pro/bin/perl -w

use strict;
use IO::Socket::INET;

my $itrcweb = "http://forums1.itrc.hp.com/service/forums";
my $server = "forums1.itrc.hp.com";
my $ux_url = "service/forums";

sub webget ($;$)
{
my ($url, $F) = @_;
local $/;

my $peer = $url =~ s{^(\w+)://}{} ? $1 : "http(80)";
my ($server, $page) = ($url =~ m{^([^/]+)(?:/(.*)});

(my $f = $page) =~ s:.*/::;
$F ||= $f;
unlink $F;

printf STDERR "get %-38s\t", $F;
my $socket = IO::Socket::INET->new (
PeerHost => $server,
PeerPort => $peer,
Proto => "tcp",
timeout => 90) or die;
print $socket
"GET $page\n",
"Host: $server\n",
"User-Agent: Internet Explorer\n",
"\n";

my $t = 0;
my $data = <$socket>;
if (ref $F) {
$$F = $data;
return $t;
}
if ($data) {
$t = length $data;
undef $p;
open $p, "> $F" or die "$F: $!\n";
print $p $data;
close $p;
}
$t;
} # webget

webget ("http://somehost.lan/cgi-bin/solaris_cmd?df", "solaris.df");

my $sol_df;
webget ("http://somehost.lan/cgi-bin/solaris_cmd?df", \$sol_df);

-->8---

(Not tested)

Enjoy, Have FUN! H.Merijn

[Fred Ruffet]

You can't install, but you can compile... just get sources (either for wget or perl modules) compile, and run from your directory.

Regards,

Fred

[Baz_2]

I've used wget and cURL which work fine.

http://www.gnu.org/software/wget/wget.html
http://curl.haxx.se/

[Michael Roberts_3 · Accepted solution]

you can also use /usr/bin/sh and telnet:

#!/sbin/sh
if [[ $# -lt 1 ]] ;then
print "supply stock symbol on command line"
exit 1
fi
for sym in $@
{
(stelnet quote.yahoo.com 80 << eof
GET http://quote.yahoo.com/download/quotes.csv?symbols=${sym}&format=sl1d1t1

eof
) 2>/dev/null |awk '$0 ~ /\"/ {print}'
}

[rmueller58]

I'd just use "wget"

wget http://hostname.ext/page.html

if you need to set it up where it's a timed thing then set up a cron job to do it.

You may want to Google "hcat" and "hgrepurl."

these are little perl scripts that OReilly press has on their FTP server. If you can't find them there I am attaching a tarball.

[John_880]

Hi all, thanks for the replies. I changed to a machine with the proper perl modules and all is fine. I will definately try to get wget installed.

Cheers! John