topic Re: wget fail with HTTP 500 in Operating System - Linux

wget fail with HTTP 500

Henry Chua — Thu, 06 Sep 2007 11:01:30 GMT

Hi Guys,

I been trying to download data from a website using wget but it does not seems to work. It only seems to work when i download via a browser. Is there anyway i can force wget to get the data?

example:
$ wget --debug http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/$File/SESprice.zip
DEBUG output created by Wget 1.9+cvs-stable (Red Hat modified) on linux-gnu.

--00:09:32-- http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/SESprice.zip
=> `SESprice.zip'
Resolving info.sgx.com... 203.127.29.6
Caching info.sgx.com => 203.127.29.6
Connecting to info.sgx.com[203.127.29.6]:80... connected.
Created socket 3.
Releasing 0x9dc26b0 (new refcount 1).
---request begin---
GET /webprices.nsf/stkprices/2007-09-05/SESprice.zip HTTP/1.0
User-Agent: Wget/1.9+cvs-stable (Red Hat modified)
Host: info.sgx.com
Accept: */*
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... HTTP/1.1 500 Internal Server Error
Server: Lotus-Domino/0
Date: Thu, 06 Sep 2007 15:58:18 GMT
Content-Type: text/html; charset=US-ASCII
Content-Length: 139
Expires: Tue, 01 Jan 1980 06:00:00 GMT
Pragma: no-cache
Connection: Close

Closing fd 3
00:09:32 ERROR 500: Internal Server Error.

Thanks.

Henry

Re: wget fail with HTTP 500

Steven E. Protter — Thu, 06 Sep 2007 11:28:28 GMT

Shalom Henry,

You may be getting a proxy error. Are there proxy settings in your browser not present in wget .wgetrc ??

Could also be security on the site itself designed to thwart wholesale download of the entire site.

SEP

Re: wget fail with HTTP 500

Rob Leadbeater — Thu, 06 Sep 2007 14:07:04 GMT

Hi Henry,

Looks like a problem with the site to me.
I just tried dropping the URL:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/SESprice.zip

into Internet Explorer, and also got a 500 error.

Cheers,

Rob

Re: wget fail with HTTP 500

Heironimus — Fri, 07 Sep 2007 10:02:20 GMT

If you look at the wget output the URL is different from the one you supplied. If the string "$File" is a literal part of the URL and not something you're supposed to get from your environment you'll need to either escape the $ or enclose the URL in single quotes.

Re: wget fail with HTTP 500

James R. Ferguson — Fri, 07 Sep 2007 16:50:52 GMT

Hi Henry:

To add to Heironimus' comments, you would be best advised to enclose any interpolated variables (parameter substitution) in curly braces to avoid ambiguity. Thus, instead of:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/$File/SESprice.zip

...use:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/${File}/SESprice.zip

..,if ${file} is intended to be interpolated into its value.

Regards!

...JRF...

Re: wget fail with HTTP 500

Henry Chua — Sat, 08 Sep 2007 10:06:04 GMT

Hi all,

Thanks for the reply... I have it appears that i can only download the file from the website only.. which is 'http://info.sgx.com/webprices.nsf/new+historical+stock+prices?openview'. Is there any reason y this is so??

Best regards
Henry

Re: wget fail with HTTP 500

Steven E. Protter — Sat, 08 Sep 2007 19:33:04 GMT

Shalom,

I've operated web servers for a few years right now. I've read in the trade press that its possible to block wget from grabbing the web content.

I assume it's either a firewall or httpd.conf configuration fix. I believe the operator of the website wishes to prevent this type of access.

I can't say I know the details but I can see why an operator would want to do this.

SEP

Re: wget fail with HTTP 500

Heironimus — Mon, 10 Sep 2007 11:18:38 GMT

I can download the file with wget.

Re: wget fail with HTTP 500

Ivan Krastev — Mon, 10 Sep 2007 14:29:14 GMT

Just escaped $ (\$) and everything was OK:

wget --debug "http://info.sgx.com/webprices.nsf/stkprices/2007-09-10/\$File/SESprice.zip"
DEBUG output created by Wget 1.10.2 on linux-gnu.

--21:34:04-- http://info.sgx.com/webprices.nsf/stkprices/2007-09-10/$File/SESprice.zip
=> `SESprice.zip'
Resolving info.sgx.com... 203.127.29.6
Caching info.sgx.com => 203.127.29.6
Connecting to info.sgx.com|203.127.29.6|:80... connected.
Created socket 3.
Releasing 0x0808e618 (new refcount 1).

---request begin---
GET /webprices.nsf/stkprices/2007-09-10/$File/SESprice.zip HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: info.sgx.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server: Lotus-Domino/0
Date: Mon, 10 Sep 2007 19:28:11 GMT
Connection: close
Content-Type: application/x-zip
Content-Length: 48860
Last-Modified: Mon, 10 Sep 2007 15:28:21 GMT
Accept-Ranges: bytes

---response end---
200 OK
Length: 48,860 (48K) [application/x-zip]

100%[=================================================================================================>] 48,860 29.23K/s

Closed fd 3
21:34:06 (29.21 KB/s) - `SESprice.zip' saved [48860/48860]

regards,
ivan