FreeBSD (racoon) - ipsec tunnel mode - Mikrotik

Hi, has anyone managed to make an “ipsec tunnel” between Freebsd(Linux) using ipsec-tools with racoon on one side and Mikrotik on the other in “tunnel” mode when each side has its own local LAN? Can someone publish the material how to do this? Each side has a “white” static ip. Maybe there are at least some ideas how to create a similar FreeBSD on Mikrotik?

https://www.freebsd.org/doc/ru_RU.KOI8-R/books/handbook/ipsec.html
FreeBSD side:
ifconfig gif0 A. B. C. D W. X. Y. Z
ifconfig gif0 inet 192.168.1.1 192.168.2.1 netmask 0xffffffff

Mikrotik ?

The settings on the link you gave describe a setup which uses a virtual interface as an entry point to the IPsec processing of the packet, whereas RouterOS currently only supports the method specified in the IPsec RFC, which is selection of traffic for sending via the IPsec SA using matching on IP protocol (udp, tcp , …), source and destination addresses, and source and destination ports where applicable. That RFC also requires that received packets matching any IPsec traffic selection criteria that were not received via a corresponding IPsec SA must be dropped.

The traffic selector of the SA is not only used locally but also negotiated between the peers. The “virtual interface” approach uses an “any to any” traffic selector, so you cannot use a traffic selector on Mikrotik side and virtual interface on the remote side just like that. A workaround is possible, which consists in defining a list of exceptions from “any to any” and assigning “action=none” to these exceptions, but it can be used only for a single remote peer.

So first, check whether FreeBSD can support also the traffic selection method, and if yes, go that way. If it cannot, the simplified idea is the following:

/ip ipsec policy
add action=none src-address=0.0.0.0/0 dst-address=0.0.0.0/1
add action=none src-address=0.0.0.0/0 dst-address=192.0.0.0/2
add action=encrypt src-address=0.0.0.0/0 dst-address=0.0.0.0/0 peer=FreeBSD

This way, packets to 0.0.0.0-127.255.255.255 are matched by the first policy, packets to 192.0.0.0-255.255.255.255 by the second one, and only packets to 128.0.0.0-191.255.255.255 make it to the last policy which sends them down the SA.

By extending this approach, you can narrow the address range which makes it to the last policy to the destinatinon subnet you want to be sent to the FreeBSD peer while the traffic selector used to negotiate the SA will be “any to any”.

It only makes sense to dive into the details of how to set the encryption and integrity protection algorithms and peer authentication methods once you resolve this key aspect.

Thanks for the information, let’s assume the FreeBSD side settings will be in accordance with the schema https://www.freebsd.org/doc/ru_RU.KOI8-R/books/handbook/security/ipsec-network.png
from https://www.freebsd.org/doc/ru_RU.KOI8-R/books/handbook/ipsec.html

ifconfig gif0

gif0: flags=8011<UP,POINTTOPOINT,MULTICAST> mtu 1280
inet 192.168.1.1 → 192.168.2.1 netmask 0xffffffff
physical address inet A.B.C.D → W.X.Y.Z

netstat -rn

Routing tables Internet:
Destination Gateway Flags Refs Use Netif Expire
192.168.2.1 192.168.1.1 UH 0 0 gif0

ipsec sa policy

spdadd A.B.C.D/32 W.X.Y.Z/32 ipencap -P out ipsec esp/tunnel/A.B.C.D-W.X.Y.Z/require;
spdadd W.X.Y.Z/32 A.B.C.D/32 ipencap -P in ipsec esp/tunnel/W.X.Y.Z-A.B.C.D/require;

racoon.conf

remote W.X.Y.Z {
nat_traversal off;
exchange_mode main;
proposal {
encryption_algorithm des;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group 2;
}
}

sainfo anonymous {
pfs_group 2;
lifetime time 1 hour;
encryption_algorithm des;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}

Mikrotik is side 2 - W.X.Y.Z 192.168.2.1 (192.168.2.0/24)

Which Mikrotik settings should correspond to this scheme. Let’s assume that encryption methods and key information exchange policies have been achieved.

Could you or Mikrotik specialists provide the basic interface, routing, and ipsec sa policy settings so that the FreeBSD-Mikrotik ipsec tunnel works? Perhaps needed to change some settings on the FreeBSD side to match Mikrotik.

This would significantly contribute to the promotion of Mikrotik products as this problem makes it difficult to use them in conjunction with existing OS such as Linux and FreeBSD.

Tell, why the this example doesn’t work https://serverfault.com/questions/779376/ipsec-passes-only-icmp

Please be patient. In my previous post, I’ve tried to explain

  • what is the essential incompatibility between the apparently only possible setup on the FreeBSD (with a virtual interface) and the definitely only possible setup on the RouterOS (with policy-based packet selection),
  • what is the possible workaround,
  • what are the limitations of that workaround.

In parallel, I was remotely debugging an almost identical setup where the FreeBSD is substituted by Google Cloud Platform, but the debugging has come to a dead end as what perfectly works for me, on the same CPU architecture (x64/CHR) and RouterOS release (6.45.9) like the OP of that topic uses, doesn’t work for the OP.

As in the meantime you have provided all the other settings necessary, the only thing which is not elementary is to compose the exception policies for the Mikrotik end. As soon as I find enough time for that, I’ll post the complete IPsec setup which should work against the FreeBSD one set the way you’ve posted originally. But it will not be in just hours from now.

The setup you ask about now, which is based on use of GRE instead of a virtual interface as an entry point of the IPsec SA itself, is definitely an alternative, but it’s a different approach, so it deserves a separate topic. It also has its drawbacks, namely, it may not work if one of the peers is behind NAT, but at least it removes the limitation to a single connection of the same type on the RouterOS side (you can have several GRE over IPsec tunnels, but only one tunnel with a virtual interface on the remote peer).

Thank you for your participation, I will wait for information

OK, so let’s start by a good news - you can forget all what I wrote about the incompatibility in my first response, as I have confused a gif tunnel with a vti one, due to the title of the topic title which mentions IPsec tunnel mode alone.

According to this article, in its IPv4 over IPv4 mode, the FreeBSD’s gif (referring to RFC 2893, which is an evolution of RFC 1933, as an inspiration) is compatible with the ipip tunnel, standardized by RFC 2003. No cross-reference between these two RFCs can be found in either of them, however the packet format differs in just the payload protocol type (4 for IPv4/RFC 2003 and 41 for IPv6/RFC 2893); I didn’t dive into all those ICMP and TTL handling details and loop prevention mechanisms.

So given that both A.B.C.D and W.X.Y.Z are directly routable (no NAT between them), it would be best to first check this layer of compatibility alone, before diving into IPsec.

At Mikrotik side, you add the tunnel interface using
/interface ipip add name=freebsd-gif0 local-address=W.X.Y.Z remote-address=A.B.C.D !keepalive

The “logical addresses” as FreeBSD refers to them may be added using
/ip address add address=192.168.2.1/32 network=192.168.1.1/32 interface=freebsd-gif0


For GRE, you would just use /interface gre instead of /interface ipip in the step above (at RouterOS side). Except the IP protocol to be matched by the policy (the tunneling protocol itself), the IPsec layer settings will be the same regardless whether the inner tunnel used will be GIF/IPIP or GRE.

In RouterOS configuration, a Phase 1 proposal mostly matches an /ip ipsec profile. Hence

remote W.X.Y.Z {
        nat_traversal off;
        exchange_mode main;
        proposal {
                encryption_algorithm des;
                hash_algorithm sha1;
                authentication_method pre_shared_key;
                dh_group 2;
        }
}

translates into
/ip ipsec profile add name=FreeBSD enc-algorithm=des hash-algorithm=sha1 nat-traversal=no
/ip ipsec peer add name=FreeBSD address=A.B.C.D local-address=W.X.Y.Z exchange-mode=main profile=FreeBSD
/ip ipsec identity add peer=FreeBSD auth-method=pre-shared-key secret=verycomplexsecret

Phase 2 proposal is called /ip ipsec proposal, so

sainfo anonymous {
      pfs_group 2;
      lifetime time 1 hour;
      encryption_algorithm des;
      authentication_algorithm hmac_sha1;
      compression_algorithm deflate;
}

translates into
/ip ipsec proposal add name=FreeBSD pfs-group=modp1024 lifetime=1h enc-algorithms=des auth-algorithms=sha1
There is no way to control use of compression algorithm in RouterOS so I suspect it is not used at all. It seems that it must be specified it in Racoon configuration but it is used only as an proposed option, not a mandatory requirement.

Lastly, in RouterOS, the policy for both SAs is configured using a single line, where the src-address represents the local IP or subnet and the dst-address represents the remote one. So

spdadd A.B.C.D/32 W.X.Y.Z/32 ipencap -P out ipsec esp/tunnel/A.B.C.D-W.X.Y.Z/require;
spdadd W.X.Y.Z/32 A.B.C.D/32 ipencap -P in ipsec esp/tunnel/W.X.Y.Z-A.B.C.D/require;

translates into
/ip ipsec policy add src-address=W.X.Y.Z dst-address=A.B.C.D ipsec-protocols=esp tunnel=yes proposal=FreeBSD protocol=ipencap level=require peer=FreeBSD

However, as you have public IP addresses at both machines and the gif/ipip/gre tunnels are linked to them (A.B.C.D, W.X.Y.Z), you can save some packet space by using a transport mode of IPsec - to do that, you need to change the above to
spdadd A.B.C.D/32 W.X.Y.Z/32 ipencap -P out ipsec esp/tunneltransport/A.B.C.D-W.X.Y.Z/require;
spdadd W.X.Y.Z/32 A.B.C.D/32 ipencap -P in ipsec esp/tunneltransport/W.X.Y.Z-A.B.C.D/require;

and
/ip ipsec policy add src-address=W.X.Y.Z dst-address=A.B.C.D ipsec-protocols=esp tunnel=yesno proposal=FreeBSD protocol=ipencap level=require peer=FreeBSD


For GRE, the protocol field would be set to gre rather than ipencap.


This part is a bit confusing. In Mikrotik IP address configuration, the network parameter of an /ip address row must either be the subnet into which the IP address itself fits, or a /32 address representing the remote end of the tunnel to the routing (so that you could specify it as a gateway of a route rather than the tunnel interface itself). So to reach the whole 192.168.1.0/24 via the tunnel, you have to manually add a route:
/ip route add dst-address=192.168.1.0/24 gateway=192.168.1.1

Thank you for the information.

Which route rule should be on the FreeBSD side for Mikrotik side remote LAN 192.168.2.0/24
(similarly Mikrotik side “/ip route add dst-address=192.168.1.0/24 gateway=192.168.1.1”) ?

static_routes=“vpn”
route_vpn=“192.168.2.0/24 192.168.2.1”

or

static_routes=“vpn”
route_vpn=“192.168.2.0/24 -interface gif0”

I assume the gif configuration is as follows:

ifconfig gif0 A.B.C.D W.X.Y.Z
ifconfig gif0 inet 192.168.1.1 192.168.2.1 netmask 0xffffffff

and which version RouterOS do you recommend for this: Long-Term 6.45.9 or Stable 6.47 ?

I’m not familiar with the structure of FreeBSD configuration, so since the tunnel configuration somewhere else indicated a shorter than a 32 bit netmask, I’ve assumed they somehow add the destination subnet to the tunnel parameters.

If it is not the case, there is in fact no need to assign any IP addresses to the tunnel endpoint interfaces at all, unless you need to use recursive next-hop search or dynamic routing protocols. So if you do assign them, both the route with the IP of the remote end of the tunnel as gateway and the route with the local interface name will work, at both the FreeBSD end and the RouterOS end. But if you want to assign the IP addresses to the tunnel endpoint interfaces, I’d recommend not to use ones that fit into larger subnets. Technically there is nothing wrong about doing so, but some behaviour may not be intuitive for the administrator.


From the point of view of IPsec and IPIP/GRE tunnel configuration there is no difference between these two. But the general rule is to use long-term wherever its feature set is sufficient for the purpose.

Today I created a tunnel that works.
Thank you, sindy, for the useful information - no wonder you have the status of “Forum guru” :wink:

Here is the configuration of both sides of the tunnel:

FreeBSD side

#rc.conf
gifconfig_gif0=“A.B.C.D W.X.Y.Z”
ifconfig_gif0=“inet 192.168.1.1 netmask 255.255.255.0 192.168.2.1 netmask 255.255.255.0 mtu 1480”
static_routes=“Net2”
route_Net2=“192.168.2.0/24 -interface gif0”

#ipsec.conf
flush;
spdflush;
spdadd A.B.C.D/32 W.X.Y.Z/32 ipencap -P out ipsec esp/transport/A.B.C.D-W.X.Y.Z/require;
spdadd W.X.Y.Z/32 A.B.C.D/32 ipencap -P in ipsec esp/transport/W.X.Y.Z-A.B.C.D/require;

#racoon.conf

racoon.conf

remote W.X.Y.Z {
nat_traversal off;
exchange_mode main;
proposal {
encryption_algorithm des;
hash_algorithm sha1;
authentication_method pre_shared_key;
dh_group 2;
}
}

sainfo anonymous {
pfs_group 2;
lifetime time 1 hour;
encryption_algorithm des;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}

#psk.txt
W.X.Y.Z verycomplexsecret

Mikrotik side

/interface ipip add name=freebsd-gif0 local-address=W.X.Y.Z remote-address=A.B.C.D !keepalive

Due to an error “invalid value for the network argument” when entering the command (as the address and network must belong to the same subnet)
“/ip address add address=192.168.2.1/32 network=192.168.1.1/32 interface=freebsd-gif0” was changed to “/ip address add address=192.168.2.1/32 interface=freebsd-gif0”

/ip ipsec profile add name=FreeBSD enc-algorithm=des hash-algorithm=sha1 nat-traversal=no
/ip ipsec peer add name=FreeBSD address=A.B.C.D local-address=W.X.Y.Z exchange-mode=main profile=FreeBSD
/ip ipsec identity add peer=FreeBSD auth-method=pre-shared-key secret=verycomplexsecret
/ip ipsec proposal add name=FreeBSD pfs-group=modp1024 lifetime=1h enc-algorithms=des auth-algorithms=sha1
/ip ipsec policy add src-address=W.X.Y.Z dst-address=A.B.C.D ipsec-protocols=esp tunnel=yesno proposal=FreeBSD protocol=ipencap level=require peer=FreeBSD
/ip route add dst-address=192.168.1.0/24 gateway=192.168.1.1

Now I’m trying to find the correct mtu size for tunnel.

At first mtu=1480 was bet set for both FreeBSD gif0 and Mikrotik ipip freebsd-gif0 with both “Clamp TCP MSS=on” and “Allow Fast Path=on”.
Then I tried mtu=1400. I compare the speed of copying information through the tunnel.

How do I calculate the mtu correctly?

The actual mistake was the /32 after the network address. Without it, it works:
_[me@MyTik] > ip address add address=192.168.2.1/32 network=192.168.1.1 interface=ipip-tunnel1
[me@MyTik] > ip address print where interface~“ipip”
Flags: X - disabled, I - invalid, D - dynamic

ADDRESS NETWORK INTERFACE

0 192.168.2.1/32 192.168.1.1 ipip-tunnel1_
But until you need dynamic routing protocols or recursive next-hop search, it makes no difference.


The correct MTU to be set on the tunnel depends on the actual MTU of the interface through which the IPsec transport packets are sent, and on the encryption and authentication algorithms used on the tunnel (some encryption algorithms include the authentication part), so calculation may be complex. Hence it is simpler to set the mtu of /interface ipip to auto, and see the actual-mtu determined by RouterOS which takes all these points into account. Then set the same at FreeBSD end.

Let me clarify which version of RouterOS you are using when entering the command: “ip address add address=192.168.2.1/32 network=192.168.1.1 interface=ipip-tunnel1”.
I’ll try to enter it again. According to https://wiki.mikrotik.com/wiki/Manual:IP/Address perhaps it should fall under this definition:

/ip add address … network

network (IP; Default: 0.0.0.0) IP address for the network. For point-to-point links it should be the address of the remote end. Starting from v5RC6 this parameter is configurable only for addresses with /32 netmask (point to point links)

As for specifying the size mtu - on Mikrotik → mtu=auto, what size of mtu should I choose for the FreeBSD side?

6.45.9, but the same works in 6.47.


Yes, this is the case. However, the default value of 0.0.0.0/0 is suspicious - normally the value is automatically calculated from the IP address itself and its netmask (so it is no configurable in this sense), and only if the IP address itself is a /32 one, the network (actually, remote end) address may be entered “manually”.


As I wrote (maybe not clearly enough), the one which /interface ipip print detail shows as actual-mtu when mtu is set to auto

Yes, its worked for me also, if no add /32 in network field for RouterOS 6.47

 ip address add address=192.168.2.1/32 network=192.168.1.1 interface=freebsd-gif0

At now when tunnel works with psk-key mode, I trying to switch authorization by the rsasig method. Used article https://habr.com/ru/post/496514.

Preparing rsa keys for both side:

FreeBSD side generation rsa key

openssl genrsa -out freebsd.pem 1024
openssl rsa -in freebsd.pem -pubout > freebsd.pub.pem
plainrsa-gen -i freebsd.pem -f freebsd.private
plainrsa-gen -i freebsd.pub.pem -f freebsd.public

Mikrotik side generation rsa key

" IP" - “IP sec” - select tab “Keys” - press “Generate key” - 1024 → mikrotik.private
" IP" - “IP sec” - select tab “Keys” - was selected “mikrotik.private” then pressed " Expor Pub. Key" → mikrotik.pub.pem

File “mikrotik.pub.pem” was copied to FreeBSD then converted for racoon

plainrsa-gen -i mikrotik.pub.pem -f mikrotik.public

All next changes at worked psk-key tunnel was highlighted in a different color

racoon.conf

remote W.X.Y.Z {
exchange_mode main;
my_identifier address “A.B.C.D”;
peers_identifier address “W.X.Y.Z”;
verify_identifier on;
certificate_type plain_rsa “freebsd.private”;
peers_certfile plain_rsa “mikrotik.public”;
proposal_check obey;
proposal {
encryption_algorithm des;
hash_algorithm sha1;
authentication_method rsasig;
dh_group 2;
}
}
sainfo anonymous {
pfs_group 2;
lifetime time 1 hour;
encryption_algorithm des;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}

Mikrotik

/ip ipsec identity
add auth-method=rsa-key key=mikrotik.private peer=FreeBSD generate-policy=no

/ip ipsec key> print
0 PR mikrotik.private 1024-bit
1 R freebsd.public 1024-bit

After this changes ipsec tunnel negotiation gets stuck on Ike phase1 and does not go although the author of the article everything worked.
Also was used “proposal_check claim” options for both side - all without success.

Can you tell - what is wrong and what is rsasig requirements for Mikrotik-FreeBSD(Linux) racoon mode?

Is this still true? http://forum.mikrotik.com/t/ipsec-connection-with-debian-linux-rsasig/1796/1

It is a topic from 2005 :smiley: back then I barely knew that Mikrotik existed as a brand :smiley: Since then, the auth-method=rsa-key has been added.


To me, the biggest surprise is that your Mikrotik does not complain about /ip ipsec identity row with auth-method set to rsa-key but no remote-key specified, or is that just a copy-paste error?

Also, the following line

is redundant - the public freebsd key in .pem format (as generated using openssl) is what the RouterOS wants, so there is no reason to convert it into the .rsa format. The FreeBSD itself is happy with the private key, it doesn’t need its own public key in a separate file. As I can see, you do have the freebsd.public key in 'Tik’s key storage, so you’ve either copied and imported the proper one (.pem), or RouterOS now supports also import of .rsa format. But as @splineship wrote something regarding openssl being unable to convert keys generated by plainrsa-gen to .pem properly, there may be something wrong in the .rsa format generated by plainrsa-gen, so the import on RouterOS may have resulted in a wrong key contents.

So if you’ve copied and imported the FreeBSD’s public key to the 'Tik in .rsa format, remove the existing freebsd.public key from the 'Tik, copy the .pem variant of the FreeBSD’s public key to Mikrotik and import it there under the same key name freebsd.public. Then, specify remote-key=freebsd.public on the /ip ipsec identity row - even if it was there but you’ve re-imported the key, because the identity row doesn’t refer to the key name but to an internal ID which changes when you remove the old key and import another one, even under the same “human-readable” name.

If none of the above makes it work, look for the explanation in the logs. On Mikrotik, do the following:

  • disable the peer representing the FreeBSD machine
  • enter /system logging add topics=ipsec,!packet
  • run /log print follow-only file=ipsec-start where topics~“ipsec”
  • enable the peer representing the FreeBSD machine
  • once the startup fails in a couple of seconds, break the /log print …, download the ipsec-start.txt file, and start reading it

On the FreeBSD, @splineship’s instruction says where to look for log messages.

Also, understand properly the remark regarding file access rights to the certificate files on FreeBSD - if someone else but the user under which racoon is running can read the files, they will most likely not be used as it would be a security hole. So the rights must really be -rw-------, not -rw-r–r-- or something alike.

To me, the biggest surprise is that your Mikrotik does not complain about /ip ipsec identity row with auth-method set to rsa-key but no remote-key specified, or is that just a copy-paste error?

Yes, there was an error when entering text - from the screen of the test computer, the command is of course this:


/ip ipsec identity
add auth-method=rsa-key key=mikrotik.private remote-key=freebsd.public peer=FreeBSD generate-policy=no



So if you’ve copied and imported the FreeBSD’s public key to the 'Tik in .rsa format, remove the existing freebsd.public key from the 'Tik, copy the .pem variant of the FreeBSD’s public key to Mikrotik and import it there under the same key name freebsd.public. Then, specify remote-key=freebsd.public on the /ip ipsec identity row - even if it was there but you’ve re-imported the key, because the identity row doesn’t refer to the key name but to an internal ID which changes when you remove the old key and import another one, even under the same “human-readable” name.

The main problem is to use a key format that is clear to both parties or generate them so that you can exchange public keys and import them to each side.

Quote from the article https://habr.com/ru/post/496514


racoon Daemon uses keys in RSA format, and mikrotik-in PEM format. If you generate keys using the plainrsa-gen utility that comes with racoon, then you can’t use it to convert the public key for Mikrotik to PEM format - it only converts in one direction: PEM to RSA. The generated plainrsa-gen key could not be read by either openssl or ssh-keygen, so they will also not be able to perform the conversion. We will generate the PEM key using openssl, and then convert it to racoon using plainrsa-gen:

From racoon docs:


authentication_method type;
Defines the authentication method used for the phase 1 negotiation.
This directive must be defined. type is one of: pre_shared_key,
rsasig (for plain RSA authentication), gssapi_krb,
hybrid_rsa_server, hybrid_rsa_client, xauth_rsa_server,
xauth_rsa_client, xauth_psk_server or xauth_psk_client.

certificate_type certspec;
Specifies a certificate specification. certspec is one of followings:
x509 certfile privkeyfile; certfile means a file name of a certificate. privkeyfile means a file name of a secret key.
plain_rsa privkeyfile; privkeyfile means a file name of a private key generated by
plainrsa-gen(8). Required for RSA authentication.

Once more docs https://www.shrew.net/static/help-2.1.x/vpnhelp.pdf

FreeBSD - > generating FreeBSD RSA private key (racoon known format) → generate FreeBSD public key → convert to Mikrotik known format, then import
Mikrotik - > generating Mikrotik RSA private key → Export Mikrotik public key → copy to FreeBSD, convert to racoon known format

For example:
#FreeBSD side generate rsa key for racoon x509 mode:
openssl req -new -nodes -newkey rsa:1024 -sha1 -days 3650 -keyform PEM -keyout freebsd.private -outform PEM -out freebsd.pem
openssl x509 -days 3650 -req -in freebsd.pem -signkey freebsd.private -out freebsd.public

Copy freebsd.public to 'Tik then try import freebsd.public:
“ip” - “ipsec” - “key” - press “import” - select file “freebsd.public” and have error “Couldn’t perform action - unable to read key, please supply key in PEM format! (6)|”

Therefore the author of article https://habr.com/ru/post/496514 pointed out this problem and found a method that allows generating keys in plain_rsa format. 'Tik and racoon understand them, but nevertheless authorization on IKE phase1 does not occur. I also didn’t forget about special permissions for key files “chmod 600”.

So if you’ve copied and imported the FreeBSD’s public key to the 'Tik in .rsa format, remove the existing freebsd.public key from the 'Tik, copy the .pem variant of the FreeBSD’s public key to Mikrotik and import it there under the same key name freebsd.public. Then, specify remote-key=freebsd.public on the /ip ipsec identity row - even if it was there but you’ve re-imported the key, because
the identity row doesn’t refer to the key name but to an internal ID which changes when you remove the old key and import another one, even under the same “human-readable” name.

Could you suggest a way to generate and exchange keys FreeBSD (RSA format) < - - - > 'Tik (PEM format). I don’t fully and very clearly understand the meaning of your method.

This used example with one only certificate without remote side certificate http://mikrotik.vetriks.ru/wiki/VPN:IPsec_(аутентификация_с_помощью_сертификата)

Is it possible use this method for FreeBSD (passive only) and 'Tik initiator ipsec?

I think It is a misunderstanding. I’ve just followed the article at Habr - I have zero experience with FreeBSD. There, the author writes that plainrsa-gen can convert an existing key in .pem format into the .rsa format but not vice versa (.pem to .rsa, and that RouterOS only supports .pem format for export and import.

In your step-by-step summary, I’ve seen that you have generated the key at FreeBSD side and created a public key from it using openssl (which is correct), and then you’ve converted both to .rsa. But for FreeBSD itself, its own private key (which has to be in .rsa format) is sufficient, it doesn’t need its own public key in any format for any purpose. So I didn’t see a point in converting the FreeBSD’s public key to .rsa format, as that key is useless for FreeBSD and as that format is unusable (according to the Habr article) for Mikrotik. So I took a suspicion that you’ve copied from FreeBSD to Mikrotik, and imported there, the FreeBSD’s public key in .rsa format, and that the import has silently failed in terms that it hasn’t reported any error but the imported key itself is not working.

If you’ve copied from FreeBSD to Mikrotik, and imported there, the FreeBSD’s public key extracted from the private one using openssl rsa -pubout in the .pem format, you’ve done everything right and there’s no need to redo that.

Preparing for attempt number 2 (in accordance with article https://habr.com/ru/post/496514)

FreeBSD generate RSA private key in PEM format

openssl genrsa -out freebsd.private.pem 1024

Extract public key from RSA private key to PEM format for 'Tik

openssl rsa -in freebsd.private.pem -pubout > freebsd.public.pem

Convert FreeBSD RSA private key to racoon known RSA plain format

plainrsa-gen -i freebsd.private.pem -f freebsd.private

Copy freebsd.public.pem to 'Tik, then importing “ip”-“ipsec”-“import”, in the drop-down list of the “file name” field, we look for the FreeBSD we downloaded freebsd.public.pem.

Mikrotik side generation rsa key

" IP" - “IP sec” - select tab “Keys” - press “Generate key” - 1024 → mikrotik.private
" IP" - “IP sec” - select tab “Keys” - was selected “mikrotik.private” then pressed " Expor Pub. Key" → mikrotik.public.pem

Copy mikrotik.public.pem to FreeBSD, convert to plain RSA known racoon

plainrsa-gen -i mikrotik.public.pem -f mikrotik.public

Change permission

chown 600 freebsd.private
chown 600 mikrotik.public

racoon.conf

remote W.X.Y.Z {
exchange_mode main;
my_identifier address "A.B.C.D";
peers_identifier address "W.X.Y.Z";
verify_identifier on;
certificate_type plain_rsa "freebsd.private";
peers_certfile plain_rsa "mikrotik.public";
proposal_check obey;
proposal {
encryption_algorithm des;
hash_algorithm sha1;
authentication_method rsasig;
dh_group 2;
}
}
sainfo anonymous {
pfs_group 2;
lifetime time 1 hour;
encryption_algorithm des;
authentication_algorithm hmac_sha1;
compression_algorithm deflate;
}

Mikrotik

/ip ipsec identity
add auth-method=rsa-key key=mikrotik.private remote-key=freebsd.public peer=FreeBSD generate-policy=no



/ip ipsec key> print
0 PR mikrotik.private 1024-bit
1 R freebsd.public 1024-bit

I hope I didn’t mess anything up.

The steps seem fine to me (I haven’t looked at the contents of racoon conf, though), except that you have mentioned the delivery of Mikrotik’s public key to the FreeBSD and converting it to .rsa there twice (before and after the steps used to generate it at Mikrotik, and with a different file name in each occurrence).