ipsec between linux and hp-ux

[ATIL VOLKAN YILDIRIM_1]

hi experts,
has anybody come to make ipsec work between linux and hp-ux.
I used openswan on the linux server, but whatever I try,
the tunnel is not being up...Instead it passes phase 1 but stops in
phase 2.

My hp-ux(11.31) ipsec(A.02.01.01 )conf is:

ipsec_config show all
startup
-autoboot OFF
-auditlvl ERROR
-auditdir /var/adm/ipsec
-maxsize 100
-spi_min 0x12c
-spi_max 0x2625a0
-spd_soft 25
-spd_hard 50

auth aspendos
-remote 10.1.121.169/32
-preshared volkan
-exchange MM

ike aspendos
-remote 10.1.121.169/32
-priority 20
-authentication PSK
-group 2
-hash SHA1
-encryption 3DES
-life 28800
-maxqm 100

gateway default
-action FORWARD

host aspendos_dene
-source 0.0.0.0/0/0
-destination 10.1.121.169/32/0
-protocol 0
-priority 30
-action ESP_AES128_HMAC_SHA1/28800/0
-flags NONE

host default
-action PASS


and my openswan(2.6.14) conf is:

config setup
# Debug-logging controls: "none" for (almost) none, "all" for lots.
# klipsdebug=none
# plutodebug="control parsing"
# For Red Hat Enterprise Linux and Fedora, leave protostack=netkey
protostack=netkey
nat_traversal=no
interfaces="ipsec0=bond0"

#include /etc/ipsec.d/*.conf

conn %default
auth=esp
disablearrivalcheck=no
keyingtries=1
keylife=1800s
ikelifetime=28800s
pfs=no
#keyexchange=ikev2

conn deneme
authby=secret
left=10.1.121.169
leftnexthop=10.1.121.254
right=10.1.121.162
rightnexthop=10.1.121.254
auto=add
compress=no
#esp=aes128-sha1
esp=3des-sha1-96
ike=3des-sha1-96
type=transport



When I try to bring up the conn, openswan says:
[root@aspendos etc]# ipsec auto --up deneme
104 "deneme" #5: STATE_MAIN_I1: initiate
003 "deneme" #5: ignoring unknown Vendor ID payload [e4e14cf3b8a3fb199581535b94b0d73c]
106 "deneme" #5: STATE_MAIN_I2: sent MI2, expecting MR2
108 "deneme" #5: STATE_MAIN_I3: sent MI3, expecting MR3
004 "deneme" #5: STATE_MAIN_I4: ISAKMP SA established {auth=OAKLEY_PRESHARED_KEY cipher=oakley_3des_cbc_192 prf=oakley_sha group=modp1024}
117 "deneme" #6: STATE_QUICK_I1: initiate
010 "deneme" #6: STATE_QUICK_I1: retransmission; will wait 20s for response
010 "deneme" #6: STATE_QUICK_I1: retransmission; will wait 40s for response
031 "deneme" #6: max number of retransmissions (2) reached STATE_QUICK_I1. No acceptable response to our first Quick Mode message: perhaps peer likes no proposal


and in the hp-ux ipsec log file, it writes:
ipsec_report -audit auditMon-Jun--1-07-38-42-2009.log

----------------------------- Audit Log -------------------------------
Audit File: /var/adm/ipsec/auditMon-Jun--1-07-38-42-2009.log

Msg: 1 From: IPSEC_ADMIN Lvl: ALERT Date: Mon Jun 1 07:38:42 2009
Event: Starting up IPSec/9000.

Msg: 2 From: IKMPD Lvl: ALERT Date: Mon Jun 1 07:38:42 2009
Event: mip6mod is not running (instance=0).
Msg: 3 From: SECPOLICYD Lvl: ALERT Date: Mon Jun 1 07:38:42 2009
Event: Found interface: family=2 name = lan901 addr = 10.1.121.162 flag=0x2
Msg: 4 From: IKMPD Lvl: ALERT Date: Mon Jun 1 07:38:42 2009
Event: Bind address 10.1.121.162 with INET socket 16.
Msg: 5 From: IKMPD Lvl: ERROR Date: Mon Jun 1 07:38:48 2009
Event: atts GROUP_DESC:Alternate 1024-bit MODP group is not acceptable
Msg: 6 From: IKMPD Lvl: ERROR Date: Mon Jun 1 07:38:48 2009
Event: Rejected Transform ID: KEY_IKE
Msg: 7 From: IKMPD Lvl: ERROR Date: Mon Jun 1 07:38:48 2009
Event: Responder cannot get the ID payload for QM negotiation.
Msg: 8 From: IKMPD Lvl: ERROR Date: Mon Jun 1 07:38:48 2009
Event: Quick Mode verify failed, mess ID 0x9b864ce
Msg: 9 From: IKMPD Lvl: ERROR Date: Mon Jun 1 07:38:58 2009
Event: Responder cannot get the ID payload for QM negotiation.
Msg: 10 From: IKMPD Lvl: ERROR Date: Mon Jun 1 07:38:58 2009
Event: Quick Mode verify failed, mess ID 0x9b864ce
Msg: 11 From: IKMPD Lvl: ERROR Date: Mon Jun 1 07:39:18 2009
Event: Responder cannot get the ID payload for QM negotiation.
Msg: 12 From: IKMPD Lvl: ERROR Date: Mon Jun 1 07:39:18 2009
Event: Quick Mode verify failed, mess ID 0x9b864ce

Message Summary:
Alerts: 4 Errors: 8 Warnings: 0 Informative: 0 Debug: 0 Unknown: 0

--------------------------- End Audit Log -----------------------------


Any helps would be appreciated...

Thanks...

[Wenxiao He]

Hi,

First, it appears the ESP does not match. On HP-UX side it has

host aspendos_dene
-action ESP_AES128_HMAC_SHA1/28800/0

On Openswan side it has:

conn deneme
esp=3des-sha1-96

Secondly, it would be helpful if the "informative" level logging on hpux side is posted.

And finally for IPsec support you can follow the support channel and log a support call, the support people can collect more detail info for further investigation.

Regards,

[ATIL VOLKAN YILDIRIM_1]

Hi,
thanks for your reply...aes128 has come from one of my tries I guess, I tried so many things that I should've posted a wrong combination...Anyway, even when using the right parameters the result is the same...

I guess I'll be calling hp...

Thanks...

[Steven E. Protter]

Shalom,

Please provide information on the distribution of Linux and the version of IPSEC in use.


SEP

[ATIL VOLKAN YILDIRIM_1]

Hi,
the ipsec/linux version is:
Linux Openswan U2.6.14/K2.6.18-92.1.18.el5 (netkey)