wget fail with HTTP 500

[Henry Chua]

Hi Guys,

I been trying to download data from a website using wget but it does not seems to work. It only seems to work when i download via a browser. Is there anyway i can force wget to get the data?

example:
$ wget --debug http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/$File/SESprice.zip
DEBUG output created by Wget 1.9+cvs-stable (Red Hat modified) on linux-gnu.

--00:09:32-- http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/SESprice.zip
=> `SESprice.zip'
Resolving info.sgx.com... 203.127.29.6
Caching info.sgx.com => 203.127.29.6
Connecting to info.sgx.com[203.127.29.6]:80... connected.
Created socket 3.
Releasing 0x9dc26b0 (new refcount 1).
---request begin---
GET /webprices.nsf/stkprices/2007-09-05/SESprice.zip HTTP/1.0
User-Agent: Wget/1.9+cvs-stable (Red Hat modified)
Host: info.sgx.com
Accept: */*
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... HTTP/1.1 500 Internal Server Error
Server: Lotus-Domino/0
Date: Thu, 06 Sep 2007 15:58:18 GMT
Content-Type: text/html; charset=US-ASCII
Content-Length: 139
Expires: Tue, 01 Jan 1980 06:00:00 GMT
Pragma: no-cache
Connection: Close


Closing fd 3
00:09:32 ERROR 500: Internal Server Error.


Thanks.

Henry

[Steven E. Protter]

Shalom Henry,

You may be getting a proxy error. Are there proxy settings in your browser not present in wget .wgetrc ??

Could also be security on the site itself designed to thwart wholesale download of the entire site.

SEP

[Rob Leadbeater]

Hi Henry,

Looks like a problem with the site to me.
I just tried dropping the URL:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/SESprice.zip

into Internet Explorer, and also got a 500 error.

Cheers,

Rob

[Heironimus]

If you look at the wget output the URL is different from the one you supplied. If the string "$File" is a literal part of the URL and not something you're supposed to get from your environment you'll need to either escape the $ or enclose the URL in single quotes.

[James R. Ferguson]

Hi Henry:

To add to Heironimus' comments, you would be best advised to enclose any interpolated variables (parameter substitution) in curly braces to avoid ambiguity. Thus, instead of:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/$File/SESprice.zip

...use:

http://info.sgx.com/webprices.nsf/stkprices/2007-09-05/${File}/SESprice.zip

..,if ${file} is intended to be interpolated into its value.

Regards!

...JRF...

[Henry Chua]

Hi all,

Thanks for the reply... I have it appears that i can only download the file from the website only.. which is 'http://info.sgx.com/webprices.nsf/new+historical+stock+prices?openview'. Is there any reason y this is so??

Best regards
Henry

[Steven E. Protter]

Shalom,

I've operated web servers for a few years right now. I've read in the trade press that its possible to block wget from grabbing the web content.

I assume it's either a firewall or httpd.conf configuration fix. I believe the operator of the website wishes to prevent this type of access.

I can't say I know the details but I can see why an operator would want to do this.

SEP

[Heironimus]

I can download the file with wget.

[Ivan Krastev]

Just escaped $ (\$) and everything was OK:

wget --debug "http://info.sgx.com/webprices.nsf/stkprices/2007-09-10/\$File/SESprice.zip"
DEBUG output created by Wget 1.10.2 on linux-gnu.

--21:34:04-- http://info.sgx.com/webprices.nsf/stkprices/2007-09-10/$File/SESprice.zip
=> `SESprice.zip'
Resolving info.sgx.com... 203.127.29.6
Caching info.sgx.com => 203.127.29.6
Connecting to info.sgx.com|203.127.29.6|:80... connected.
Created socket 3.
Releasing 0x0808e618 (new refcount 1).

---request begin---
GET /webprices.nsf/stkprices/2007-09-10/$File/SESprice.zip HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: info.sgx.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server: Lotus-Domino/0
Date: Mon, 10 Sep 2007 19:28:11 GMT
Connection: close
Content-Type: application/x-zip
Content-Length: 48860
Last-Modified: Mon, 10 Sep 2007 15:28:21 GMT
Accept-Ranges: bytes

---response end---
200 OK
Length: 48,860 (48K) [application/x-zip]

100%[=================================================================================================>] 48,860 29.23K/s

Closed fd 3
21:34:06 (29.21 KB/s) - `SESprice.zip' saved [48860/48860]

regards,
ivan