IPSec between HP - SUN servers

[Fredric Vådegård]

Hi,
have any one successfully configured IPSec for a HP-UX 11.00 and a Solaris 9 server? Will use this for backup thru a firewall. The test do not include the FW, that is the next step.

The problem seems to be on the Sun box since I have successfully configured and connected to a W2K server from the HP box. There are no GUI for the IPSec on Solaris. I have followed an example from the "IPsec and IKE Administration Guide" from Sun (http://docs-pdf.sun.com/817-2694/817-2694.pdf).
When starting up the in.iked process in debug mode, I see that there are problem in Phase 1 (Phase 1 negotiation error: No proposal chosen)

Thanks!

//Fredric

[Peter Godron]

Hi,
could you double-check your IKE policies.
That message would appear if policies have a security or encryption mismatch.
Are there any other messages before or after?
Regards

[Fredric Vådegård]

Hi,

now I havn't the same problem, but it still don´t work.

I can see "ESP SPI=0x323 Replay=1" when snooping (Replay is counting up by one every) and running a telnet against the remote IP-address.

Currently I use the following files
/etc/inet/ipsecinit.conf
/etc/inet/ike/config
/etc/inet/ipseckey
removed the /etc/inet/secret/ike.preshared, because it was not in the new example of setup I saw on http://builder.com.com/5100-6372-1044095.html

//Fredri

[Slawomir Gora]

Hi,

I'am using IPSEC between Solaris 9 and HPUX 11.0 and 11.11 but only in transport mode.
Maby you have problem with your preshared password - on Solaris it must be hex.

Can you attach yours configs ?
HPUX: /var/adm/ipsec/polices.txt
Sol9: /etc/inet/ike/config
/etc/inet/ipsecinit.conf

[Fredric Vådegård]

I have attached the requested files.

HP:
server name - hpbup
Below I have listed what is used for the setup of HP/SUN in the policies.txt
begin filter hpbup_setest02
begin oakley isakmp_hpbup2
begin transform ESP-3DES-HMAC-SHA1




Thanks!

//Fredric

[Slawomir Gora · Accepted solution]

Hi,

I think that your problem is in oakley group for in.iked daemon.
On HPUX you have oakley group 2
On Solaris you have oakley group 5
Your solaris polices are in old style try use my config:

#########################
/etc/inet/ipsecinit.conf

{ saddr 192.36.176.142
daddr 192.36.176.216
} ipsec {
encr_auth_algs md5
encr_algs 3des
sa shared
}

#########################
/etc/inet/iked/config
p1_lifetime_secs 28800
p2_lifetime_secs 28800

{
label "hpbup_setest02"
local_id_type ip
local_addr 192.36.176.142
remote_addr 192.36.176.216

p1_xform {
auth_method preshared
oakley_group 2
auth_alg md5
# encr_alg 3des-cbc - YOU CAN TRY THIS
encr_alg 3des
p1_lifetime_secs 28800
}
p2_lifetime_secs 28800
}

#####################
If you want convert key from text to hex use my script:

#!/bin/sh

if [ "$1" = "" ]
then
echo "usage `basename $0` keyfile"
exit
fi

OUTFILE="$1.hex"

if [ -f $1 ]
then
cat $1 | od -X | \
awk '{printf "%s%s%s%s",$2,$3,$4,$5} END {printf "\n"}' |\
sed -e 's/0a000000//g' > ${OUTFILE}

echo "Key is stored in ${OUTFILE}"
else
echo "No such file $1"
fi


#end of file

[Slawomir Gora]

Hi,

if you will have problems try to debug iked daemon:

ikeadm -n set debug all /tmp/iked.log

I know you can have problem with your communication when you will restart or flush keys on solaris site - no information is send to HP and HP will send encrypted packages to solaris and no negotiation until KEY TIMEOUT (in your example 28800) will happen.

[Andrew Cowan]

IPSec is also very fussy about the declarion of system hostnames, and IP-Addresses. Ensure that both machines really know each other by the exact canonical or alias hostnames, and that if you have amulti-homed hosts, data is arriving through the correct interfaces.

[Fredric Vådegård]

Thank you for your help so far!

I have changed the configuration files as you suggested.
I don´t get any error msg when starting up the in.iked process, it even says "added rule hpbup_setest02", "config_update succeeded!".
But, as you wrote, I don´t get any answear from hpbup when trying to telnet hpbup from setest02(SUN).
Should I wait 8h (28800sec) and see if works after the time_out?

//Fre

[Andrew Cowan]

This may sound really stupid but there isn't any IP netmask issues here are there? Re my earlier point: If one of the machines has the wrong netmask, or can be reached on a couple of interfaces, then IP-SEC will fail during the negotiation stage. The other thing you can try is to go to the lowest level of link first e.g. start with AH and no ESP, and see if that works with a really basic secret/key e.g. "test". If successful keep upping the authentication levels until it fails.

Another idea. There aren't any hidden control characters or trailing spaces or tabs in your config files are there?

[Fredric Vådegård]

Hi,

I have rebooted the SUN box (setest02).
I have attached the iked.log file, hopefully it is useful.

I will check/test a more basic connection.

//Fredric

[Slawomir Gora]

Hi,

I have one question do you need to configure
all trafic between HP and SUN ?
I always configure only selected services (telnet, ssh etc..).

[Fredric Vådegård]

I will use the connection for backup (Data Protector, port 5555).

[Andrew Cowan]

From this error message it looks like IP-SEC is unsure of what kind of IKE key to expect. Are you sure that you are not missing a parameter in the config file that explicity mentions a static key, and its format e.g. ASCII or HEX.

I have seen this problem when connecting between two AIX boxes when using the GUI and entering exactly the same data on both. The only solution was to do all the configuration on ONE node and then export it. The export reverses the data. You could see if either Solaris or HP-UX provides such a utility and examine the XML files for strange characters or extra parameters?

[Fredric Vådegård]

Hi,

I had renamed the /etc/inet/secret/ike.preshared file. The file is now renamed. I assume that I must have the ike.preshared file, it holds the key. But still it don't work!


I would like to double check with you to see if I doing right after I have changed some of the config files.

1) ipsecconf -f
2) ipsecconf -a /etc/inet/ipsecinit.conf
3) pkill in.iked
4) /usr/lib/inet/in.iked -d
5) ikeadm -n set debug all /tmp/iked.log

I had a config file earlier called /etc/inet/ipseckey, that removed. Should I have that file?

I appreciate your help!

//Fredric

[Fredric Vådegård]

YES - IT WORKS !!!!!!!!!!!

I tried between 2 SUN boxes and get it working at once. Then I found out how to audit the trafic on the HP box and I got more information that lead me on the right target.

Thanks!

//Fredric