1748204 Members
3395 Online
108759 Solutions
New Discussion юеВ

wget fail with HTTP 500

 
Henry Chua
Super Advisor

wget fail with HTTP 500

Hi Guys,

I been trying to download data from a website using wget but it does not seems to work. It only seems to work when i download via a browser. Is there anyway i can force wget to get the data?

example:
$ wget --debug http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/$File/SESprice.zip
DEBUG output created by Wget 1.9+cvs-stable (Red Hat modified) on linux-gnu.

--00:09:32-- http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/SESprice.zip
=> `SESprice.zip'
Resolving info.sgx.com... 203.127.29.6
Caching info.sgx.com => 203.127.29.6
Connecting to info.sgx.com[203.127.29.6]:80... connected.
Created socket 3.
Releasing 0x9dc26b0 (new refcount 1).
---request begin---
GET /webprices.nsf/stkprices/2007-09-05/SESprice.zip HTTP/1.0
User-Agent: Wget/1.9+cvs-stable (Red Hat modified)
Host: info.sgx.com
Accept: */*
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... HTTP/1.1 500 Internal Server Error
Server: Lotus-Domino/0
Date: Thu, 06 Sep 2007 15:58:18 GMT
Content-Type: text/html; charset=US-ASCII
Content-Length: 139
Expires: Tue, 01 Jan 1980 06:00:00 GMT
Pragma: no-cache
Connection: Close


Closing fd 3
00:09:32 ERROR 500: Internal Server Error.


Thanks.

Henry
8 REPLIES 8
Steven E. Protter
Exalted Contributor

Re: wget fail with HTTP 500

Shalom Henry,

You may be getting a proxy error. Are there proxy settings in your browser not present in wget .wgetrc ??

Could also be security on the site itself designed to thwart wholesale download of the entire site.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Rob Leadbeater
Honored Contributor

Re: wget fail with HTTP 500

Hi Henry,

Looks like a problem with the site to me.
I just tried dropping the URL:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/SESprice.zip

into Internet Explorer, and also got a 500 error.

Cheers,

Rob
Heironimus
Honored Contributor

Re: wget fail with HTTP 500

If you look at the wget output the URL is different from the one you supplied. If the string "$File" is a literal part of the URL and not something you're supposed to get from your environment you'll need to either escape the $ or enclose the URL in single quotes.
James R. Ferguson
Acclaimed Contributor

Re: wget fail with HTTP 500

Hi Henry:

To add to Heironimus' comments, you would be best advised to enclose any interpolated variables (parameter substitution) in curly braces to avoid ambiguity. Thus, instead of:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/$File/SESprice.zip

...use:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/${File}/SESprice.zip

..,if ${file} is intended to be interpolated into its value.

Regards!

...JRF...



Henry Chua
Super Advisor

Re: wget fail with HTTP 500

Hi all,

Thanks for the reply... I have it appears that i can only download the file from the website only.. which is 'http://info.sgx.com/webprices.nsf/new+historical+stock+prices?openview'. Is there any reason y this is so??

Best regards
Henry
Steven E. Protter
Exalted Contributor

Re: wget fail with HTTP 500

Shalom,

I've operated web servers for a few years right now. I've read in the trade press that its possible to block wget from grabbing the web content.

I assume it's either a firewall or httpd.conf configuration fix. I believe the operator of the website wishes to prevent this type of access.

I can't say I know the details but I can see why an operator would want to do this.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Heironimus
Honored Contributor

Re: wget fail with HTTP 500

I can download the file with wget.
Ivan Krastev
Honored Contributor

Re: wget fail with HTTP 500

Just escaped $ (\$) and everything was OK:

wget --debug "http://info.sgx.com/webprices.nsf/stkprices/2007-09-10/\$File/SESprice.zip"
DEBUG output created by Wget 1.10.2 on linux-gnu.

--21:34:04-- http://info.sgx.com/webprices.nsf/stkprices/2007-09-10/$File/SESprice.zip
=> `SESprice.zip'
Resolving info.sgx.com... 203.127.29.6
Caching info.sgx.com => 203.127.29.6
Connecting to info.sgx.com|203.127.29.6|:80... connected.
Created socket 3.
Releasing 0x0808e618 (new refcount 1).

---request begin---
GET /webprices.nsf/stkprices/2007-09-10/$File/SESprice.zip HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: info.sgx.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server: Lotus-Domino/0
Date: Mon, 10 Sep 2007 19:28:11 GMT
Connection: close
Content-Type: application/x-zip
Content-Length: 48860
Last-Modified: Mon, 10 Sep 2007 15:28:21 GMT
Accept-Ranges: bytes

---response end---
200 OK
Length: 48,860 (48K) [application/x-zip]

100%[=================================================================================================>] 48,860 29.23K/s

Closed fd 3
21:34:06 (29.21 KB/s) - `SESprice.zip' saved [48860/48860]

regards,
ivan