CPAN issues with proxy server

[chindi]

Hi ,

We need to install perl modules in our setup.

We dont have direct internet connection.

11iv3 host

when we are trying through our proxy we are getting the fllowing issues ;

cpan[2]> install Rex::Task
Warning: Cannot install Rex::Task, don't know what it is.
Try the command

i /Rex::Task/

to find objects with matching identifiers.

cpan[3]> i /Rex::Task/
No objects found of any type for argument /Rex::Task/

cpan[4]> install IO::All::FTP
Warning: Cannot install IO::All::FTP, don't know what it is.
Try the command

i /IO::All::FTP/

to find objects with matching identifiers.

cpan[5]> o conf
$CPAN::Config options from '/opt/perl_32/lib/5.8.8/CPAN/Config.pm':
commit [Commit changes to disk]
defaults [Reload defaults from disk]
help [Short help about 'o conf' usage]
init [Interactive setting of all options]

applypatch []
auto_commit [0]
build_cache [100]
build_dir [/.cpan/build]
build_dir_reuse [0]
build_requires_install_policy [ask/yes]
bzip2 [/usr/local/bin/bzip2]
cache_metadata [1]
check_sigs [0]
colorize_debug undef
colorize_output [0]
colorize_print undef
colorize_warn undef
commandnumber_in_prompt [1]
commands_quote undef
connect_to_internet_ok [0]
cpan_home [/.cpan]
curl [/usr/bin/curl]
dontload_hash undef
dontload_list undef
ftp []
ftp_passive [1]
ftp_proxy [33.xx.xx.xx:8080]
ftpstats_period undef
ftpstats_size undef
getcwd [cwd]
gpg []
gzip [/usr/contrib/bin/gzip]
halt_on_failure [0]
histfile [/.cpan/histfile]
histsize [500]
http_proxy [3.xx.xx.xx:8080]
inactivity_timeout [0]
index_expire [1]
inhibit_startup_message [0]
keep_source_where [/.cpan/sources]
load_module_verbosity [v]
lynx []
make [/usr/bin/make]
make_arg []
make_install_arg []
make_install_make_command [/usr/bin/make]
makepl_arg []
mbuild_arg []
mbuild_install_arg []
mbuild_install_build_command [./Build]
mbuildpl_arg []
ncftp []
ncftpget []
no_proxy []
pager [/usr/bin/more]
password undef
patch [/usr/bin/patch]
perl5lib_verbosity [v]
prefer_installer [MB]
prefs_dir [/.cpan/prefs]
prerequisites_policy [ask]
proxy_pass undef
proxy_user []
randomize_urllist undef
scan_cache [atstart]
shell [/sbin/sh]
show_unparsable_versions [0]
show_upload_date [0]
show_zero_versions [0]
tar [/usr/bin/tar]
tar_verbosity [v]
term_is_latin [1]
term_ornaments [1]
test_report [0]
trust_test_report_history [0]
unzip [/oracle/app/oracle/product/10.2.0/bin/unzip]
urllist
0 [http://perlmirror.indialinks.com/]
use_sqlite [0]
username undef
wait_list undef
wget [/usr/local/bin/wget]
yaml_load_code [0]
yaml_module [YAML]

[Matti_Kurkela]

In another thread, we saw that your proxy at 33.xx.xx.xx:8080 is really a HTTP proxy.

You should specify the "ftp_proxy" setting as "http://33.xx.xx.xx:8080".

The "http://" prefix tells the download utilities that the proxy is a HTTP proxy and FTP-over-HTTP technique should be used.

[chindi]

Hi Matti ,

Even after putting ftp_proxy its the same ;

cpan[4]> o conf
$CPAN::Config options from '/opt/perl_32/lib/5.8.8/CPAN/Config.pm':
commit [Commit changes to disk]
defaults [Reload defaults from disk]
help [Short help about 'o conf' usage]
init [Interactive setting of all options]

applypatch []
auto_commit [0]
build_cache [100]
build_dir [/.cpan/build]
build_dir_reuse [0]
build_requires_install_policy [ask/yes]
bzip2 [/usr/local/bin/bzip2]
cache_metadata [1]
check_sigs [0]
colorize_debug undef
colorize_output [0]
colorize_print undef
colorize_warn undef
commandnumber_in_prompt [1]
commands_quote undef
connect_to_internet_ok [0]
cpan_home [/.cpan]
curl [/usr/bin/curl]
dontload_hash undef
dontload_list undef
ftp []
ftp_passive [1]
ftp_proxy http://33.xx.xx.xx:8080
ftpstats_period undef
ftpstats_size undef
getcwd [cwd]
gpg []
gzip [/usr/contrib/bin/gzip]
halt_on_failure [0]
histfile [/.cpan/histfile]
histsize [500]
http_proxy http://33.xx.xx.xx:8080
inactivity_timeout [0]
index_expire [1]
inhibit_startup_message [0]
keep_source_where [/.cpan/sources]
load_module_verbosity [v]
lynx []
make [/usr/bin/make]
make_arg []
make_install_arg []
make_install_make_command [/usr/bin/make]
makepl_arg []
mbuild_arg []
mbuild_install_arg []
mbuild_install_build_command [./Build]
mbuildpl_arg []
ncftp []
ncftpget []
no_proxy []
pager [/usr/bin/more]
password undef
patch [/usr/bin/patch]
perl5lib_verbosity [v]
prefer_installer [MB]
prefs_dir [/.cpan/prefs]
prerequisites_policy [ask]
proxy_pass undef
proxy_user []
randomize_urllist undef
scan_cache [atstart]
shell [/sbin/sh]
show_unparsable_versions [0]
show_upload_date [0]
show_zero_versions [0]
tar [/usr/bin/tar]
tar_verbosity [v]
term_is_latin [1]
term_ornaments [1]
test_report [0]
trust_test_report_history [0]
unzip [/oracle/app/oracle/product/10.2.0/bin/unzip]
urllist
0 [http://perlmirror.indialinks.com/]
use_sqlite [0]
username undef
wait_list undef
wget [/usr/local/bin/wget]
yaml_load_code [0]
yaml_module [YAML]

=========================================

cpan[5]> install CAM::PDF
CPAN: Storable loaded ok (v2.15)
Going to read /.cpan/Metadata
Warning: Cannot install CAM::PDF, don't know what it is.
Try the command

i /CAM::PDF/

to find objects with matching identifiers.
CPAN: Time::HiRes loaded ok (v1.86)

cpan[6]> install Tk::Text
Warning: Cannot install Tk::Text, don't know what it is.
Try the command

i /Tk::Text/

to find objects with matching identifiers.

cpan[7]> install Tk::Text
Warning: Cannot install Tk::Text, don't know what it is.
Try the command

i /Tk::Text/

to find objects with matching identifiers.

cpan[8]> install Text::Reform
Warning: Cannot install Text::Reform, don't know what it is.
Try the command

i /Text::Reform/

to find objects with matching identifiers.

why am not able to download any module ???

[chindi]

Hi Matti ,

We ned to change Config.pm right ?

[chindi]

Hi Matti ,

From the proxy configuration support we got below reply ;

Force users to use FTP via browser (ftp over http) only, that way you’ll force the content to be scanned by the gateway security server.

Using the FW You should block all ftp outgoing traffic except for the Gateway Security – with such users will be forced using the Gateway Security proxy for FTP using a browser over port 8080

[Matti_Kurkela]

> cpan[5]> install CAM::PDF
> CPAN: Storable loaded ok (v2.15)
> Going to read /.cpan/Metadata
> Warning: Cannot install CAM::PDF, don't know what it is.

I guess that your CPAN shell previously attempted to download the metadata (= the list of available modules), and got an empty file. Now it thinks it has valid metadata, but actually it has nothing at all. Assuming that your proxy configuration is now correct, you would need to make the CPAN shell retry loading the metadata before you can do anything else.

Hmm... Looking at the output of the "help" command of the CPAN shell, I think you should try the "reload index" command first. If that does not help, exit the CPAN shell and remove /.cpan/Metadata; the CPAN shell should reload it automatically when it does not exist.

Edit:

Your situation seems to be exactly like the "http firewall" case in the CPAN shell documentation:

http://perldoc.perl.org/CPAN.html#WORKING-WITH-CPAN.pm-BEHIND-FIREWALLS

 Your proxy at port 8080 is apparently some sort of a security/anti-virus filter.

The answer from the proxy configuration support looks like the support person was thinking about "workstation-like" use only. "FTP using a browser" means ftp-over-http, exactly what we are trying to do by setting the ftp_proxy to a http:// prefixed value.

[chindi]

Hi Matti ,

root #/ >rm -rf /.cpan/Metadata
root #/ >cpan
Terminal does not support AddHistory.

cpan shell -- CPAN exploration and modules installation (v1.9301)
ReadLine support available (maybe install Bundle::CPAN or Bundle::CPANxxl?)

cpan[1]> reload index
CPAN: Storable loaded ok (v2.15)
CPAN: LWP::UserAgent loaded ok (v5.819)
CPAN: Time::HiRes loaded ok (v1.86)
LWP not available

Trying with "/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.:8080"" to get
ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz
curl: (52) Empty reply from server

System call "/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.:8080" "ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz" > /.cpan/sources/authors/01mailrc.txt.tmp23723"
returned status 52 (wstat 13312)
Warning: expected file [/.cpan/sources/authors/01mailrc.txt.gz.tmp23723] doesn't exist

Trying with "/usr/local/bin/wget -O /.cpan/sources/authors/01mailrc.txt.tmp23723" to get
ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz
ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz: Unsupported scheme.

System call "/usr/local/bin/wget -O /.cpan/sources/authors/01mailrc.txt.tmp23723 "ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz" "
returned status 1 (wstat 256)
Warning: expected file [/.cpan/sources/authors/01mailrc.txt.gz.tmp23723] doesn't exist
LWP not available

Trying with "/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.xx.xx8080"" to get
http://mirror.is.co.za/authors/01mailrc.txt.gz
curl: (52) Empty reply from server

System call "/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.xx.:8080" "http://mirror.is.co.za/authors/01mailrc.txt.gz" > /.cpan/sources/authors/01mailrc.txt.tmp23723"
returned status 52 (wstat 13312)
Warning: expected file [/.cpan/sources/authors/01mailrc.txt.gz.tmp23723] doesn't exist

Trying with "/usr/local/bin/wget -O /.cpan/sources/authors/01mailrc.txt.tmp23723" to get
http://mirror.is.co.za/authors/01mailrc.txt.gz
http://mirror.is.co.za/authors/01mailrc.txt.gz: Unsupported scheme.

System call "/usr/local/bin/wget -O /.cpan/sources/authors/01mailrc.txt.tmp23723 "http://mirror.is.co.za/authors/01mailrc.txt.gz" "
returned status 1 (wstat 256)
Warning: expected file [/.cpan/sources/authors/01mailrc.txt.gz.tmp23723] doesn't exist
LWP not available

Trying with "/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.xx.:8080"" to get
http://ftp.rediris.es/authors/01mailrc.txt.gz
curl: (22) The requested URL returned error: 404 Not Found

System call "/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.xx.xx:8080" "http://ftp.rediris.es/authors/01mailrc.txt.gz" > /.cpan/sources/authors/01mailrc.txt.tmp23723"
returned status 22 (wstat 5632)
Warning: expected file [/.cpan/sources/authors/01mailrc.txt.gz.tmp23723] doesn't exist

Trying with "/usr/local/bin/wget -O /.cpan/sources/authors/01mailrc.txt.tmp23723" to get
http://ftp.rediris.es/authors/01mailrc.txt.gz
--10:34:38-- http://ftp.rediris.es/authors/01mailrc.txt.gz
=> `/.cpan/sources/authors/01mailrc.txt.tmp23723'
Connecting to 33.xx.xx.xx:8080... connected.
Proxy request sent, awaiting response... 404 Not Found
10:34:39 ERROR 404: Not Found.

[chindi]

Hi Matti ,

cpan[1]> reload index
CPAN: Storable loaded ok (v2.15)
Going to read /.cpan/Metadata
CPAN: LWP::UserAgent loaded ok (v5.819)
CPAN: Time::HiRes loaded ok (v1.86)
LWP not available
LWP not available

Trying with "/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.x.x:8080"" to get
ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz
CPAN: Compress::Zlib loaded ok (v2.015)
Going to read /.cpan/sources/authors/01mailrc.txt.gz
............................................................................DONE
LWP not available
LWP not available

why am getting LWP not available ??????

[Matti_Kurkela]

I tried to access the mirror.is.co.za and ftp.rediris.es CPAN mirrors you used in your previous message, and neither of them seemed to be good. Apparently you noticed that on your own, and switched to cpan.mirror.ac.za instead. If that mirror works for you, that's OK.

LWP (also known as libwww-perl) is a perl module that can be used to access HTTP and FTP servers without using an external utility like wget or curl.

The "LWP not available" message means that the LWP module is not working in your system for some reason.

In that case, the CPAN shell uses other methods instead. In this case, it has apparently used the curl utility, successfully.

If the CPAN shell is now working with your proxy, you might want to try CPAN commands "install LWP" or "upgrade LWP".

The first one should install LWP if it is not installed; the second one will try to upgrade LWP, which might help if you have a version of LWP installed but it does not work.

[chindi]

Hi Matti ,

cpan[2]> install LWP
CPAN: Storable loaded ok (v2.15)
Going to read /.cpan/Metadata
LWP is up to date (5.820).

cpan[3]> upgrade LWP
All modules are up to date for LWP

[chindi]

Hi Matti ,

root #/opt/perl-5.18.0 >wget ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz
--15:39:43-- ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz
=> `01mailrc.txt.gz'
Connecting to 33.xx.xx.xx:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 211,352 (206K) [application/octet-stream]

100%[============================================================================================================================>] 211,352 460.44K/s

15:39:52 (458.67 KB/s) - `01mailrc.txt.gz' saved [211352/211352]

For me wget is working fine .

Can we ask cpan.pm to use wget and not curl ???????

how do i check whether curl is working or not ??

 PFA Config.pm herewith .

[Matti_Kurkela]

Trying with "/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.x.x:8080"" to get
ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz
CPAN: Compress::Zlib loaded ok (v2.015) 
Going to read /.cpan/sources/authors/01mailrc.txt.gz ............................................................................DONE

 This looks like your curl might be working.

But if you want to check, take the command line from within the quotes, add the ftp:// URL and see if you can get the file:

/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.x.x:8080" ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz

 (all on a single command line)

If you receive a file named 01mailrc.txt.gz and it is not empty, curl should be working.

If you want to disable curl in CPAN shell, set the path of curl in the configuration to an empty string, i.e.

'curl' => q[],

 just like the other possible download commands that don't exist on your system (lynx, ncftp, ncftpget).

[chindi]

Hi Matti ,

For that curl , am getting some garbage on my screen.

[Matti_Kurkela]

Oops, my mistake. By default, curl outputs the file it downloads to standard output. The ".gz" suffix in the filename indicates that "01mailrc.txt.gz" is compressed, so it would look like garbage.

If you still are not convinced that curl works, you can test it this way:

/usr/bin/curl -L -f -s -S --netrc-optional -U ":" -x "http://33.xx.x.x:8080" ftp://cpan.mirror.ac.za/authors/01mailrc.txt.gz >01mailrc.txt.gz

 (all in one command line)

Then use "gunzip 01mailrc.txt.gz" to uncompress it.

After that, you should have a readable text file you can view with "more 01mailrc.txt".

The first 2 lines in the file should be:

alias AADLER     "Andy Adler <adler@ncf.ca>"
alias AAKD       "Aaron K Dancygier <aaron@dancygier.com>"

 and the last 2 lines should be:

alias ZZCGUMK    "Mark Mc Keown <mark.mckeown@manchester.ac.uk>"
alias ZZZ        "Alexandr Gomoliako <zzz@zzz.org.ua>"

 If you can see that, then curl has successfully loaded the complete file through your proxy, and so it can be used with the CPAN shell.

[chindi]

Hi Matti ,

am able to download perl modules now.

But everytime its asking for username and password  and we dont have any .

How to set the cpn such that it doesnt ask for the same .

please check o conf settings below ;

cpan[1]> o conf
$CPAN::Config options from /opt/perl_32/lib/5.8.8/CPAN/Config.pm:
commit [Commit changes to disk]
defaults [Reload defaults from disk]
help [Short help about 'o conf' usage]
init [Interactive setting of all options]

applypatch []
auto_commit [0]
build_cache [100]
build_dir [/.cpan/build]
build_dir_reuse [0]
build_requires_install_policy [ask/yes]
bzip2 [/usr/local/bin/bzip2]
cache_metadata [1]
check_sigs [0]
colorize_debug undef
colorize_output [0]
colorize_print undef
colorize_warn undef
commandnumber_in_prompt [1]
commands_quote undef
connect_to_internet_ok [0]
cpan_home [/.cpan]
curl []
dontload_hash
[Net::FTP] => []
[LWP] => []
dontload_list undef
ftp [/usr/bin/ftp -p]
ftp_passive [1]
ftp_proxy [http://33.xx.xx.xx:8080]
ftpstats_period undef
ftpstats_size undef
getcwd [cwd]
gpg []
gzip [/usr/contrib/bin/gzip]
halt_on_failure [0]
histfile [/.cpan/histfile]
histsize [500]
http_proxy [http://33.xx.xx.xx:8080]
inactivity_timeout [0]
index_expire [1]
inhibit_startup_message [0]
keep_source_where [/.cpan/sources]
load_module_verbosity [v]
lynx []
make [/usr/bin/make]
make_arg []
make_install_arg []
make_install_make_command [/usr/bin/make]
makepl_arg []
mbuild_arg []
mbuild_install_arg []
mbuild_install_build_command [./Build]
mbuildpl_arg []
ncftp []
ncftpget []
no_proxy []
pager [/usr/bin/more]
password undef
patch [/usr/bin/patch]
patches_dir undef
perl5lib_verbosity [v]
prefer_external_tar undef
prefer_installer [MB]
prefs_dir [/.cpan/prefs]
prerequisites_policy [ask]
proxy_pass undef
proxy_user []
randomize_urllist undef
scan_cache [atstart]
shell [/sbin/sh]
show_unparsable_versions [1]
show_upload_date [0]
show_zero_versions [0]
tar [/usr/bin/tar]
tar_verbosity [v]
term_is_latin [1]
term_ornaments [1]
test_report [0]
trust_test_report_history [0]
unzip [/oracle/app/oracle/product/10.2.0/bin/unzip]
urllist
0 [ftp://cpan.mirror.ac.za/]
1 [http://mirror.is.co.za/]
2 [http://cpan.nctu.edu.tw/]
3 [http://perlmirror.indialinks.com/]
4 [ftp://ftp.nara.wide.ad.jp/pub/CPAN/]
5 [http://ftp.nara.wide.ad.jp/pub/CPAN/]
use_sqlite [0]
username undef
version_timeout undef
wait_list undef
wget [/usr/local/bin/wget]
yaml_load_code [0]
yaml_module [YAML]

Also i have updated cpan version to 2.00 and ran reload CPAN after it .

[Matti_Kurkela · Accepted solution]

Looks like your proxy_user setting is defined as an empty string.

For Perl, there is a difference between "undefined" and "defined as an empty string". In this case, "undefined" apparently means "the proxy won't need authentication", while "defined as an empty string" means "the proxy will need authentication, let the user supply the username & password manually".

I'm not using a proxy, and my CPAN shell configuration says:

    proxy_pass         undef
    proxy_user         undef

 You probably should completely remove the proxy_user line from your CPAN shell configuration file. After that, the proxy_user setting should be undefined.

[chindi]

Thanks Matti you were spot on :)