[CentOS] Openswan <-> VyOS

Wed Feb 17 14:38:45 UTC 2016
John Cenile <jcenile1983 at gmail.com>

Hello,


I'm having a bit of trouble connecting our current CentOS Openswan server
with a Vyos server via IPSec.

I've posted this on the VyOS forums, but haven't had many helpful
responses, so I thought I would ask here.

http://forum.vyos.net/showthread.php?tid=26504&pid=29703#pid29703

Basically our Openswan configuration is as follows:

conn VYOS
        keyingtries=0
        keylife=20m
        ikelifetime=2h
        left=<VYOS IP>
        right=<OPENSWAN IP>
        leftsubnets={
10.1.1.0/24,10.1.2.0/24,10.1.3.0/24,10.1.4.0/24,10.1.5.0/24}
        rightsubnets={10.2.1.0/24,10.2.2.0/24,10.2.3.0/24,10.2.4.0/24}
        auto=start
        authby=secret
        dpddelay=30
        dpdtimeout=120
        dpdaction=hold
        phase2alg=aes256-sha1;modp1536
        phase2=esp
        ike=aes256-sha1;modp1536

Our VyOS configuration is posted in the above forum post, except now I have
followed their advice and created 20 tunnels (each subnet to each subnet,
if that makes sense).

However, when I enabled this, I got the following errors on the Openswan
server:


Feb 18 01:24:27 OPENSWAN pluto[8010]: "VYOS/3x3" #70: next payload type of
ISAKMP Hash Payload has an unknown value: 243
Feb 18 01:24:27 OPENSWAN pluto[8010]: "VYOS/3x3" #70: malformed payload in
packet
Feb 18 01:24:27 OPENSWAN pluto[8010]: "VYOS/3x3" #70: sending notification
PAYLOAD_MALFORMED to <VYOS IP>:500
Feb 18 01:24:27 OPENSWAN pluto[8010]: "VYOS/4x4" #69: next payload type of
ISAKMP Hash Payload has an unknown value: 170
Feb 18 01:24:27 OPENSWAN pluto[8010]: "VYOS/4x4" #69: malformed payload in
packet
Feb 18 01:24:27 OPENSWAN pluto[8010]: "VYOS/5x4" #68: next payload type of
ISAKMP Hash Payload has an unknown value: 63
Feb 18 01:24:27 OPENSWAN pluto[8010]: "VYOS/5x4" #68: malformed payload in
packet


And on our VyOS server we got the following errors:

Feb 18 01:17:19 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-20" #381:
sending encrypted notification INVALID_ID_INFORMATION to <OPENSWAN IP>:500
Feb 18 01:17:19 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-20" #381:
cannot respond to IPsec SA request because no connection is known for
10.1.1.0/24===<VYOS IP>[<VYOS IP>]...<OPENSWAN IP>[<OPENSWAN IP>]===
10.2.3.0/24
Feb 18 01:17:19 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-20" #381:
sending encrypted notification INVALID_ID_INFORMATION to <OPENSWAN IP>:500
Feb 18 01:17:23 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-11" #422:
cannot install eroute -- it is in use for "peer-<OPENSWAN IP>-tunnel-3" #403
Feb 18 01:17:23 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-16" #421:
cannot install eroute -- it is in use for "peer-<OPENSWAN IP>-tunnel-4" #395
Feb 18 01:17:23 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-20" #420:
cannot install eroute -- it is in use for "peer-<OPENSWAN IP>-tunnel-5" #417
Feb 18 01:17:23 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-20" #381:
Informational Exchange message must be encrypted
Feb 18 01:17:24 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-20" #381:
Quick Mode I1 message is unacceptable because it uses a previously used
Message ID 0x14702d90 (perhaps this is a duplicated packet)
Feb 18 01:17:24 VYOS pluto[20807]: "peer-<OPENSWAN IP>-tunnel-20" #381:
sending encrypted notification INVALID_MESSAGE_ID to <OPENSWAN IP>:500

Does anyone have any idea what I might be doing wrong? I've tried doing
only 5 tunnels, however then some subnets couldn't reach certain subnets
(as I said in the VyOS forum thread), and now I've tried each subnet to
each subnet.

I can't find much (any) information on it, but does Openswan support VTI
interfaces? Would that solve my problem?

Thanks in advance.