Page 1 of 1

Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Thu Aug 07, 2014 4:45 am
by jimr
We set out to connect our office LAN to an Amazon AWS Virtual Private Cloud (VPC) using Amazon's VPN facility. After some experimentation, we have a result that is quite usable and secure. Amazon doesn't support Microtik specifically, but they were helpful and not averse to answering questions to help with this setup.

There is a limitation in the Microtik software and a workaround. These will be discussed at the end. Hopefully, Microtik will address this problem in a future software update. In the meantime, it isn't serious obstacle to using AWS VPN.

The network we created is very typical. It could be the "out of the box" default NAT private network behind a single public IP address. The existence of the NAT just makes it interesting.

The first step is to sign-up for AWS. You'll want to create a VPC, which is a prerequisite to using AWS VPN. You'll also want to launch an EC2 machine instance into your VPC so you have something that can respond to "ping" later. The free-tier micro-instance of AWS Linux does this nicely and you never really need to even login or configure it. Just launch it and note the private IP address for testing.

Create a customer gateway and VPN from the web console of AWS. You will supply the public address of your router (or one public address, if you have more than one). In the recipe that follows, I've used x.x.x.x to refer to this address. Use your own!

You also will supply a private AS-number during creation of the VPN and select dynamic routing using BGP. Static routing is another option. For BGP, select any number in the private AS range that doesn't conflict with something you already use. In the example, I selected 65222.

Once AWS has done its magic, your VPN will be "available" and attached to your customer gateway and your VPC. You will then be able to download a configuration file. Choose the generic format (unless you are configuring one of the listed and supported routers). The downloaded file will contain the specific IP-addresses, passwords, and AS-numbers for your VPN. These may differ in small way from what is outlined below. Be sure to substitute your private values for those in the example.

The AWS VPN is implemented as a pair of IPsec tunnels between their facility and your router. The purpose of using two tunnels is to keep traffic flowing during times that IPsec may be negotiating security between the endpoints. Unfortunately, Microtik's limitation partly defeats this goal, but even so, the performance of the resulting setup is quite respectable. More on the limitation and performance at the end.

Referring to the AWS configuration sheet, first assign the customer tunnel endpoints to the public interface of the router. The public address of the router should already be assigned to this interface too.

IP addresses (add):
3   169.254.249.30/30  169.254.249.28  ether1-gateway                                                                     
4   169.254.249.26/30  169.254.249.24  ether1-gateway

Next, we make the default IPsec proposal correspond to AWS requirements. If you don't use these values, you are wasting your own time:
0  * name="default" auth-algorithms=sha1 enc-algorithms=aes-128-cbc lifetime=8m pfs-group=modp1024
Next, you configure five IPsec policies as below. You should be creating six policies, but because of the Microtik limitation, you will have to get along with these five:
0    ;;; AWS Tunnels
      src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

1    src-address=169.254.249.26/32 src-port=any dst-address=169.254.249.25/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0 

2    src-address=169.254.249.30/32 src-port=any dst-address=169.254.249.29/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

3    src-address=169.254.249.25/32 src-port=any dst-address=169.254.249.26/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.119 sa-dst-address=x.x.x.x proposal=default priority=0 

4    src-address=169.254.249.29/32 src-port=any dst-address=169.254.249.30/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.120 sa-dst-address=x.x.x.x proposal=default priority=0


Next, create the IPsec peers that support the Phase-1 handshake and password exchange. Be certain to use the AWS-supplied passwords and don't mix-up the pieces of tunnel #1 and tunnel #2. Use the two public IP-addresses supplied by AWS as a point of reference and keep all the parts for each tunnel together. If you mix them, it won't work.
0   ;;; AWS VPC Tunnel #2
     address=205.251.233.120/32 local-address=x.x.x.x passive=no port=500 auth-method=pre-shared-key 
     secret="<aws-supplied secret>" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no 
     proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s 
     dpd-maximum-failures=3 

1   ;;; AWS VPC Tunnel #1
     address=205.251.233.119/32 local-address=x.x.x.x passive=no port=500 auth-method=pre-shared-key 
     secret="<different aws-supplied secret>" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no 
     proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s 
     dpd-maximum-failures=3
Next, you need some firewall filters to allow IPsec to operate through your public interface. You could abbreviate these a lot, but this long version provides useful counters to provide feedback when it starts working:
1   chain=input action=accept protocol=ipsec-esp src-address=205.251.233.119 dst-address=x.x.x.x in-interface=ether1-gateway 

2   chain=input action=accept protocol=udp src-address=205.251.233.119 dst-address=x.x.x.x in-interface=ether1-gateway src-port=500 
     dst-port=500 

3   chain=input action=accept protocol=ipsec-esp src-address=205.251.233.120 dst-address=x.x.x.x in-interface=ether1-gateway 

4   chain=input action=accept protocol=udp src-address=205.251.233.120 dst-address=x.x.x.x in-interface=ether1-gateway src-port=500 
     dst-port=500

Next, you need a few rules so that BGP from the AWS can talk to your router:
6   chain=input action=accept protocol=tcp src-address=169.254.249.25 dst-address=169.254.249.26 dst-port=179 

7   chain=input action=accept protocol=tcp src-address=169.254.249.29 dst-address=169.254.249.30 dst-port=179
These two rules allow traffic to be routed freely between your AWS VPC and your private network. This is wide open in both directions. The VPC is secure, but you may want to provide a more selective filter.
23   ;;; VPC at AWS us-west-2x
     chain=forward action=accept src-address=172.31.0.0/16 in-interface=ether1-gateway 

24   chain=forward action=accept dst-address=172.31.0.0/16 in-interface=ether2-master-local
Because the local network is private and NAT is running, the following firewall NAT rule allows traffic to arrive at the VPC with the correct (private) source address:
0   ;;; critically important to AWS connectivity that this rule be ahead of "masquerade".
     chain=srcnat action=src-nat to-addresses=192.168.88.0/24 dst-address=172.31.0.0/16

The following firewall NAT filter rule is OPTIONAL and to be used only if the public address you supplied to AWS for your router is different from the public address used to masquerade your private NAT network. Don't use this unless you have two public addresses! And note the exclamation point!!!
1   chain=srcnat action=masquerade src-address=!x.x.x.x out-interface=ether1-gateway

You will need a destination nat rule to defeat any port-mapping rules that may be present in your existing dstnat chain. If none exist, this isn't needed, though it is harmless.
2   ;;; critically important to AWS connectivity that this rule be ahead of any NAT port-mapping
     chain=dstnat action=accept src-address=172.31.0.0/16 in-interface=ether1-gateway

Next, you configure some BGP instances using the addresses from the AWS instructions:
1    name="vgw-1" as=65222 router-id=169.254.249.30 redistribute-connected=no redistribute-static=yes redistribute-rip=no 
      redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table="" 

2    name="vgw-2" as=65222 router-id=169.254.249.26 redistribute-connected=no redistribute-static=yes redistribute-rip=no 
      redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table=""

Next, you define some BGP peers so that your BGP knows where to find AWS. The normal periodic behavior of BGP in connection with these peers will startup IPsec because of the handshake going to the far-end tunnel endpoint.
0 E name="awsvpc1" instance=vgw-1 remote-address=169.254.249.29 remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no 
     route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=169.254.249.30 
     default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no 

1 E name="awsvpc2" instance=vgw-2 remote-address=169.254.249.25 remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no 
     route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=169.254.249.26 
     default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no

Next, you define what local networks you want BGP to advertise to AWS. At minimum, this would be your local private network block. And hopefully, that isn't a 172.31.0.0/16 range network, as these conflict with the monster VPC that AWS allocates to you!
0   network=192.168.88.0/24 synchronize=yes
Assuming BGP is alive and you didn't mix things up, IPsec will install some security associations (SA) that look something like those below. If these don't appear, the tunnels are not working. Check passwords, address, firewall, etc. Also check for the presence of remote-peers, further below. If you see them, then the password and Phase-1 handshake is OK and your problem is firewall or BGP.
0 E  spi=0xF1A1CB0 src-address=205.251.233.120 dst-address=207.38.19.222 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature 
      auth-key="7d08d47b8dd2c46dbc0f8df62b4b899f49bb6a96" enc-key="26e38b12f243133cd8700122443cbf2f" addtime=aug/06/2014 22:20:49 
      expires-in=17m19s add-lifetime=16m/20m current-bytes=2045874 

1 E  spi=0xF7B18F0 src-address=205.251.233.119 dst-address=207.38.19.222 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature 
      auth-key="fd67f15352fae6598c37fa3965ad9990a5354443" enc-key="58c5a47403cb23a77a655135a4ef3263" addtime=aug/06/2014 22:22:10 
      expires-in=18m40s add-lifetime=16m/20m current-bytes=1770 

2 E  spi=0x57469D96 src-address=x.x.x.x dst-address=205.251.233.119 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature 
      auth-key="e2d031639925eb74a270c24941eb70d84b039eaa" enc-key="1dbd99b3b843d62237fe6a2da88b8a17" addtime=aug/06/2014 22:22:10 
      expires-in=18m40s add-lifetime=16m/20m current-bytes=1178 

3 E  spi=0xA099C19C src-address=x.x.x.x dst-address=205.251.233.120 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature 
      auth-key="cba17dab219c1494fe32f0ec034298a2e82274ec" enc-key="1ba56b9109f72a26057cfc5ff2e04afd" addtime=aug/06/2014 22:20:49 
      expires-in=17m19s add-lifetime=16m/20m current-bytes=113976118

Typical IPsec Remote Peers:
0 local-address=x.x.x.x remote-address=205.251.233.119 state=established side=initiator established=2m14s 

1 local-address=x.x.x.x remote-address=205.251.233.120 state=established side=initiator established=3m26s 
If BGP is happy, you will see routing advertisements something like these:
PEER     PREFIX               NEXTHOP          AS-PATH                                                                 ORIGIN     LOCAL-PREF
awsvpc1  192.168.88.0/24      169.254.249.30                                                                           igp       
awsvpc2  192.168.88.0/24      169.254.249.26                                                                           igp


With working BGP, you will also see some additional dynamic routes to AWS VPC in your route table as below. Only one such route is allowed to be active at any given instant. Also note that the gateway address for these routes is the far-end address of the IPsec tunnel, indicating that all the communication is secured.
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          x.x.x.w             1
 1 ADC  192.168.88.0/24    192.168.88.1    ether2-master-l...        0
 2 ADC  169.254.249.24/30  169.254.249.26  ether1-gateway            0
 3 ADC  169.254.249.28/30  169.254.249.30  ether1-gateway            0
 4 ADb  172.31.0.0/16                      169.254.249.25           20
 5  Db  172.31.0.0/16                      169.254.249.29           20
 6 ADC  x.x.x.v/27   x.x.x.x   ether1-gateway            0

If you got this far, you probably have it working.

Don't forget that AWS VPC is a paid service. You pay by the hour for the connection. However, the cost is about US$1 per day, so you're not going to go broke experimenting! If you actually use the tunnel, this is a bargain, especially if you can make it work with a low-cost, high-performance router!

In our case, a 750GL routerboard running v6.15 will see about 50 percent peak CPU utilization (400 MHz) handling sustained encrypted traffic on a 15 megabit/sec link. The effective throughput is about 1.6 Mbyte/sec for long file transfers. Latency is typically below 50 milliseconds between a host on our private network and an instance in our VPC.

The Microtik Limitation Explained

At last, Microtik should fix their little difficulty with IPsec. The problem is, their software does not allow two IPsec policies with identical src- and dst- addresses but different sa-dst-address to be configured. If you attempt to do this, the new policy is declared "invalid" and the whole IPsec policy list becomes unstable.

Following the example above, the missing policy is:
src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0

The presence of this policy would allow the VPN to continue to carry traffic during Phase-2 PFS negotiation periods, which AWS recommends occur every 8 minutes. For our purposes, this is overkill and we've extended the lifetime of Phase-2 a bit so the "pause to refresh" is less frequent. When it happens, the pause is a matter of milliseconds, but even this can be a lot on a fast link or supporting streaming traffic.

It's pretty clear that if you examine the routing table above, only one tunnel is ever actually carrying traffic between your router and AWS. But when Phase-2 timer fires and the active tunnel stops working briefly, the switch to the idle tunnel should be nearly instantaneous and only initiated by the end seeking to transmit (without end-to-end negotiation). The IPsec policy selection employed by Microtik seems to prevent this.

Having said all this the performance of the VPN is not affected, except during the Phase-2 handshake, which is very brief.

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Fri Sep 12, 2014 1:28 pm
by Cubo
Please indicate the settings downladed from Amazon

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Mon Sep 29, 2014 10:36 pm
by KPSI
We are tying to set up the same thing and having similar issues. How did you get ahold of Amazon for support question because we can not seem to get to anyone?

P.S. Could you by any chance look over our configuration and see if you could identify the problem?

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Tue Sep 30, 2014 6:46 pm
by jimr
We are tying to set up the same thing and having similar issues. How did you get ahold of Amazon for support question because we can not seem to get to anyone?

P.S. Could you by any chance look over our configuration and see if you could identify the problem?
I can't advise about AWS support. However, during the time we were getting started with them, we were paying the $100 per month support fee. They were very prompt and helpful with our questions, though they couldn't really help with the Microtik limitation. The two encounters we have were very high quality contacts. They were willing and able to setup screen sharing and watch what I was doing to the Microtik -- even appeared interested. They are not willing to actually configure a customer router, which is understandable.

As for your configuration, it is hard to advise. If you have the AWS configuration file, you would basically follow the steps outlined but substituting your specific addresses and passwords from that file.. Our configuration is still running for us today and it has been very reliable.

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Mon Jan 12, 2015 6:32 am
by sbasnet
hello, thanks for this explicit configuration. following this example i am able to setup vpn. but in my case, only one BGP comes up. the second one never comes up. but in your example you have shown that both BGP as up. so what am i doing wrong. should the second ipsec policy be also turned on? i have currently this in disabled mode as per your mikrotik limitation explanation.

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Mon Jan 12, 2015 10:34 pm
by jimr
Yes. Both IPSEC must be enabled to get both BGP relationships to establish.

It has been sometime since I posted those instructions. We have additional experience to share.

Because of the limitation in the Mikrotik, the VPN setup using a Mikrotik router will result in rare application-level issues that are visible to some users. For example, SMB file sharing to a Windows server will occasionally fail if conducted through the tunnel.

To work around this, we recently purchased a used Juniper ICG1000. This is a gigantic router compared to the Mikrotik, but they are readily available in the used market for about $300. The Juniper is an AWS supported configuration, the dual-tunnel limitations aren't present, and the resulting tunnel is rock-solid. Our electric bill may suffer a little, however.

We also tested a Juniper SCG1400. It worked as well and can be purchased used for about $200.

It would be great to see Mikrotik remove this limitation in their implementation of IPSEC.

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Fri Jan 23, 2015 3:26 pm
by sbasnet
Hello, following your example, the VPN is up. Only one BGP instance is up due to the Mikrotik limitation but BGP is learning prefix advertised by AWS VPC. Referring to the IP addresses used in your example, my PC is having 192.168.88.2/24. From my PC I can ping x.x.x.x, I can ping 169.254.249.29. But no matter what, I cannot ping 172.31.0.3 (the instance IP at AWS). So exactly how do I start troubleshooting the connection? Is it due to my Mikrotik configuration or something at AWS end?? The VPN at AWS is also learning the prefix (192.168.88.0/24) advertised by Mikrotik. I even searched for and added a mangle rule for fragmentation but no luck. Is there some debug in Mikrotik which can display what exactly is happening?
TIA.

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Mon Jan 26, 2015 4:02 pm
by jimr
On the Mikrotik side, confirm that you have a route through to the 172.31.x.x network. Also be certain that the firewall rules are in place to allow this access. Finally, be certain that you have the rule in place to bypass source address replacement on traffic outbound your local network and destined for AWS. You do not want this replacement. Your local "NAT" addresses must go through the tunnel -- not be replaced by the address of the local WAN interface. To stop this, the AWS destination address range much be the first rule in your NAT-tab (ahead of Masquerade or SRCNAT, depending on how you have it setup).

On the AWS side, be certain you have a route back to your local network (BGP should do this automatically), and make certain that the applicable security rules are in place. They exist in two places, one associated with the VPC (a stateless list of rules) and the other associated with each host instance.

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Wed Jan 28, 2015 5:33 am
by jkarras
To get around the two SA same policy issue did you try setting each policy with a different priority?

Re: Amazon AWS VPN -- A Working Configuration Example and Bu

Posted: Wed Jan 28, 2015 6:58 am
by jimr
Yes. Priority doesn't appear to matter. One of the policies is invariably declared "invalid" by RouterOS. Frankly, I don't think the priorities actually work.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed May 13, 2015 1:39 am
by cmurrayis
Hi Guys,

Thanks for the guide - BGP is online and routes being exchanged however I am unable to ping from the local network to the VCP.

I can ping the far side of the tunnel 169 address however when I trace the 172 address it fails at the customers gateway and does not show the hops on the tunnel so I assume my rules are incorrect.

Anything you can recommend?

ip firewall nat print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=dstnat action=accept src-address=172.31.0.0/16 in-interface=ether1-wan-brocade-1/1/8 log=no log-prefix=""

1 ;;; AWS bypass NAT
chain=srcnat action=src-nat to-addresses=192.168.0.0/16 dst-address=172.31.0.0/16 log=no log-prefix=""

ip firewall filter print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=forward action=accept protocol=ipsec-esp src-address=54.240.204.X dst-address=103.17.X.X in-interface=ether1-wan-brocade-1/1/8 log=no log-prefix=""

1 chain=forward action=accept protocol=udp src-address=54.240.204.X dst-address=103.17.X.X in-interface=ether1-wan-brocade-1/1/8 src-port=500 dst-port=500 log=no log-prefix=""

2 chain=forward action=accept protocol=ipsec-esp src-address=54.240.204.X dst-address=103.17.X.X in-interface=ether1-wan-brocade-1/1/8 log=no log-prefix=""

3 chain=forward action=accept protocol=udp src-address=54.240.204.X dst-address=103.17.X.X in-interface=ether1-wan-brocade-1/1/8 src-port=500 dst-port=500 log=no log-prefix=""

4 chain=forward action=accept dst-address=172.31.0.0/16 in-interface=ether3-DC log=no log-prefix=""

5 chain=forward action=accept src-address=172.31.0.0/16 in-interface=ether1-wan-brocade-1/1/8 log=no log-prefix=""

6 chain=forward action=accept dst-address=172.31.0.0/16 in-interface=ether2-mpn-brocade-1/1/9 log=no log-prefix=""



7 chain=input action=accept protocol=tcp src-address=169.254.247.17 dst-address=169.254.247.18 dst-port=179 log=no log-prefix=""

8 chain=input action=accept protocol=tcp src-address=169.254.247.22 dst-address=169.254.247.21 dst-port=179 log=no log-prefix=""

ip ipsec policy print
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default
0 T * group=*2 src-address=::/0 dst-address=::/0 protocol=all proposal=AWS template=yes

1 src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=103.17.X.X sa-dst-address=54.240.204.X proposal=AWS priority=0

2 src-address=169.254.247.18/32 src-port=any dst-address=169.254.247.17/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=103.17.X.X sa-dst-address=54.240.204.X proposal=AWS priority=0

3 src-address=169.254.247.22/32 src-port=any dst-address=169.254.247.21/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=103.17.X.X sa-dst-address=54.240.204.X proposal=AWS priority=0

4 src-address=169.254.247.17/32 src-port=any dst-address=169.254.247.18/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=54.240.204.X sa-dst-address=103.17.X.X proposal=AWS priority=0

5 src-address=169.254.247.21/32 src-port=any dst-address=169.254.247.22/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=54.240.204.X sa-dst-address=103.17.X.X proposal=AWS priority=0

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed May 13, 2015 2:13 pm
by mrz
@jimr
The presence of this policy would allow the VPN to continue to carry traffic during Phase-2 PFS negotiation periods, which AWS recommends occur every 8 minutes. For our purposes, this is overkill and we've extended the lifetime of Phase-2 a bit so the "pause to refresh" is less frequent. When it happens, the pause is a matter of milliseconds, but even this can be a lot on a fast link or supporting streaming traffic.
There shouldn't be any pause because new SAs are re-established when soft-timer expires (old SAs are still active and used for forwarding). Only when new SAs has "mature" state, old ones goes to "dying" state and get removed only after hard-timeout expires.

Test between two MT routers did not show any pause at all during SA renegotiation.

Are you actually see the "pause" until new SAs are established after lifetime expires, or it is just taken from AWS recommendation articles?

Also I do not see this as a requirement for any other implementations on Linux:
http://www.cloudhowto.org/amazon-vpc-wi ... -solution/
http://bleikertz.com/blog/amazon_vpc_with_linux.html
http://www.heitorlessa.com/working-with ... pn-part-2/

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed May 27, 2015 5:27 pm
by jimr
@mrz

Sorry about the delay on this. The notification disappeared in a pile of other e-mail...

I can assure you that the dual-tunnel requirement of AWS cannot be supported by the Mikrotik implementation. I'm not looking at it in the same level of implementation detail as you, but from an outward perspective. The missing rule I described is needed. This rule allows both tunnels to have a common end-point on the Mikrotik side. Any attempt to define this rule creates instability and is declared invalid in RouterOS. Without the rule, the AWS VPN support is crippled. During tunnel renegotiation, most persistent IP connections will time-out (e.g., Windows shares).

Unfortunately, our solution was to substitute a Juniper ScreenOS router that implements the required dual-tunnel perfectly. Juniper is one of about 10 different brands that AWS supports. The ScreenOS tunnel has been running for 116 days without trouble, as of this morning.

I'd suggest that the easiest way for MikroTik to resolve this problem would be to create an AWS VPN to test against. Last time I looked, the cost of one of these VPNs was $15 per month, with no minimum commitment.
@jimr
The presence of this policy would allow the VPN to continue to carry traffic during Phase-2 PFS negotiation periods, which AWS recommends occur every 8 minutes. For our purposes, this is overkill and we've extended the lifetime of Phase-2 a bit so the "pause to refresh" is less frequent. When it happens, the pause is a matter of milliseconds, but even this can be a lot on a fast link or supporting streaming traffic.
There shouldn't be any pause because new SAs are re-established when soft-timer expires (old SAs are still active and used for forwarding). Only when new SAs has "mature" state, old ones goes to "dying" state and get removed only after hard-timeout expires.

Test between two MT routers did not show any pause at all during SA renegotiation.

Are you actually see the "pause" until new SAs are established after lifetime expires, or it is just taken from AWS recommendation articles?

Also I do not see this as a requirement for any other implementations on Linux:
http://www.cloudhowto.org/amazon-vpc-wi ... -solution/
http://bleikertz.com/blog/amazon_vpc_with_linux.html
http://www.heitorlessa.com/working-with ... pn-part-2/

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed May 27, 2015 5:36 pm
by jimr
@mrz

A further comment --

All three of the example implementations cited in your post illustrate the same requirement: a single address on the Mikrotik size is used to terminate both tunnels. This is not possible in RouterOS. Just try it.

The problem exists whether or not one uses BGP for routing (an option with AWS VPN).

My comments about the re-negotiation delay have more to do with why two tunnels are desirable. In RouterOS, the problem is that you can't define two tunnels to one destination that terminate on a common IP-address on the near side.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Thu May 28, 2015 3:42 pm
by mrz
My comments about the re-negotiation delay have more to do with why two tunnels are desirable. In RouterOS, the problem is that you can't define two tunnels to one destination that terminate on a common IP-address on the near side.
But according to all examples from link I provided address ranges are different for both policies:
Outside IP Addresses:
  - Customer Gateway:        : 1.2.3.4
  - VPN Gateway              : 72.21.209.225

Inside IP Addresses
  - Customer Gateway         : 169.254.255.2/30
  - VPN Gateway              : 169.254.255.1/30

Outside IP Addresses:
  - Customer Gateway:        : 1.2.3.4
  - VPN Gateway              : 72.21.209.193

Inside IP Addresses
  - Customer Gateway         : 169.254.255.6/30
  - VPN Gateway              : 169.254.255.5/30

Other example too:
Tunnel 1:
Pre-Shared Key : AAAA
Outside IP Addresses:
- Customer Gateway : 1.1.1.1
- Virtual Private Gateway : 2.2.2.2
Inside IP Addresses
- Customer Gateway : 169.254.248.10/30
- Virtual Private Gateway : 169.254.248.9/30
- Customer Gateway ASN : 65000
- Virtual Private Gateway ASN : 7224
- Neighbor IP Address : 169.254.248.9

Tunnel 2:
Pre-Shared Key : BBBB
Outside IP Addresses:
- Customer Gateway : 1.1.1.1
- Virtual Private Gateway : 3.3.3.3
Inside IP Addresses
- Customer Gateway : 169.254.248.14/30
- Virtual Private Gateway : 169.254.248.13/30
- Customer Gateway ASN : 65000
- Virtual Private Gateway ASN : 7224
- Neighbor IP Address : 169.254.248.13

Such configurations works also on RouterOS without any problems, or am I missing something?

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Tue Jun 02, 2015 12:15 pm
by riaanhuman
Hi Guys

Been reading through this thread as i am also battling with the amazon stuff.

I am configuring this for a client of mine.

My setup is slightly different in the sense that Amazon is not accessing a private ip on my clients network they are accessing a public IP assigned to a windows machine........ :mrgreen: :shock:

I have got the 1 tunnel up and a BGP is established. The subnet of 10.0.0.0/16 is announced back to me and i went as far as setting up the bgp network for the correct subnet range and not announcing a default route back to amazon.

According to BGP the advertising is all good and the BGP session is established.

I cannot ping the Amazon side ( client gave me 10.0.1.42 as the IP )

Because amazon is accessing a public portion of the network i do not need the NAT bypass rule ( altough i did include it ).

Suggestions are VERY welcome please.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Fri Jun 26, 2015 12:01 am
by giannello
Hi jimr, and thanks for the great post - it's the most complete I've seen so far about the topic.

There's a couple of things that are not really clear to me:
0    ;;; AWS Tunnels
      src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 
Why is this SA needed? Packets from the 192.168.88.0/24 are src-nat'ed using the Inside IP Address of the Customer Gateway - provided by Amazon. The only SAs should be between the two Inside IP Addresses, as all the traffic is tunneled through them.

also
0   ;;; critically important to AWS connectivity that this rule be ahead of "masquerade".
     chain=srcnat action=src-nat to-addresses=192.168.88.0/24 dst-address=172.31.0.0/16
isn't the to-address supposed to be the Inside IP Address configured on our Mikrotik, so that the packets can be properly SNATed through it?

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Thu Aug 20, 2015 4:07 pm
by sthibault
Hi Guys

Thanks jimr for the example, I now have a working connection with my VPC using Mikrotik. I will use it for a while to see if the Mikrotik limitation impacts the traffic between our Lan and the VPC.

I hope Mikrotik will take care of the issue in the future.

This is potentially for them a feature that could help get their product known and increase sales. Let me explained, I'm working for SaaS provider and all our infrastructure is hosted on Amazon AWS. When it was the time to acquire a Firewall for our new office, one of the key feature I was looking for was AWS VPN support. Amazon can generate VPN configuration file for the most several vendors such as Cisco, Juniper, Fortinet, Vyatta, etc. (https://aws.amazon.com/vpc/faqs/#C9)

I have been a Mikrotik user for a few years but since it wasn't supported and I had read this thread explaining the limitation with the 2 ipsec tunnel to AWS, I decided agains't purchasing a Cloud Core Router and went with a Juniper SSG-320. This limitation cost Mikrotik a sale at that time and I'm sure that I'm not the only one. Having Mikrotik listed on the Amazon AWS VPN supported device list and included in the automatic generation of config file would be a major advertisement, endorsement and would lead to increase sale.

I went ahead with the purchase of the Juniper SSG, I was please with the ease of setting the AWS VPN connection. It literally takes seconds and it's stable.

I eventually got tired of the complexity of their ScreenOS and decided to order a CCR. I do prefer RouterOS for everything and the support from the community is great. I will keep the SSG and might use it only to connect to our different VPC using several AWS VPN connection, if this doesn't work correctly.

I really hope that Mikrotik look at this issue with another perpective and fix this issue.

Serge

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Fri Sep 25, 2015 2:31 pm
by ulysses
Year after, this post is still of a great value, thank you very much! Most of the things are straight-forward, but there is one thing that I have struggled to understand for already 20 hours in a row, searched high and low and still can't get my head around it..

First, something minor.
0   ;;; critically important to AWS connectivity that this rule be ahead of "masquerade".
     chain=srcnat action=src-nat to-addresses=192.168.88.0/24 dst-address=172.31.0.0/16
I was puzzled with this notation, since as I understand the idea behind this is that for the BGP routed network we want to avoid masquerading. However the way it is accomplished is rather strange: you change the source address to... itself? Or even, in case the packet arrives from another network that this local router is serving, say, 192.168.89.0/24, it will snat it, mapping to the 88 network, and may introduce conflicts (as I suspect). Instead, I used the return action at this point to simply leave the source address as is. Also, I had to add another rule to allow for the BGP traffic
 2    chain=srcnat action=return dst-address=172.31.0.0/16 log=no log-prefix="" 
 3    chain=srcnat action=return dst-address=169.254.0.0/16 log=yes log-prefix="" 
 
Next, my biggest unresolved question
why do we need to create mirror ipsec policies for the BGP link network, and creating mirror for the main site to site policy breaks everything?
I mean the following snippet, where policies 1-3 and 2-4 are mirroring each other, but when I try to create a mirror policy for 0, even with explicit addresses, it just fails and the packets silently die during the RouthingDecision http://wiki.mikrotik.com/wiki/Manual:Packet_Flow phase (I see the packet on mangle prerouting and nat dstnat, and then it disapears; if I add explicit rule in routing to mark the destination as unreachable then it works by returning the corresponding ICMP message to the issuer). This makes it even harder for me to understand, because it means that changes to the ipsec policy implicitly influences the way the packet is routed, although it is not shown on the packet flow
0    ;;; AWS Tunnels
      src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

1    src-address=169.254.249.26/32 src-port=any dst-address=169.254.249.25/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0 

2    src-address=169.254.249.30/32 src-port=any dst-address=169.254.249.29/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

3    src-address=169.254.249.25/32 src-port=any dst-address=169.254.249.26/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.119 sa-dst-address=x.x.x.x proposal=default priority=0 

4    src-address=169.254.249.29/32 src-port=any dst-address=169.254.249.30/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.120 sa-dst-address=x.x.x.x proposal=default priority=0
I suspect that there is something I don't know about the basics of IPSec implementation in RouterOS, but there is absolutely no information on how it works.
I understand, that one policy should be enough for both encoding and decoding traffic, and in case of decoding it is applied in "reverse", i.e. a policy is working for a received packet where src-address of the policy is matched to the destination address in the received packet. That seems valid assumption since in the examples only one rule is added for a site-to-site tunneling, with single mirrored rule on the other side of the tunnel (different router). But then this should be working the same for the BGP networks! I tried removing rules 3 and 4 above, and everything worked just fine. However the fact that mirror policies for BGP networks work fine, and mirror policy for the main network crashes routing is something that doesn't let me sleep

Now, let's see one big issue with the proposed solution.

The author sets up both VPN tunnels and both BGP instances, but only one site to site policy is installed (active), due to the described RouterOS limitation. But that results in a very bad situation that I have observed.

Since Amazon side sees two working tunnels it's routing at one moment decides to go over the other channel, not the one that is being selected as active on our side. As long as the ipsec policy that would be used to decode the response is in inactive state (NB, "I" stands for inactive, not invalid), or simply missing (like the author suggested), then we observe unanswered ping. At the same time, on the other side our ping is received and even answered, but it returns over the channel that we can't decode. The way I am detecting this, is by monitoring the installed-as current bytes property, which obviously increases with every pair of ICMP messages, and the response goes over the other channel.

The only way to fix this is to shut down one of the BGP instances and the corresponding ipsec policy. Probably a script can be developed that would switch channels.

I am currently investigating options to control the AWS BGP decision on the route selection, so that I can actively control it based on the currently active ipsec policy on my side.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Sat Oct 17, 2015 11:16 pm
by mark2015
I followed all the steps; however, I still cannot ping over the tunnel. AWS says the tunnel are up and everything looks ok from my side. It even seems like the ICMP packet goes out, but nothing ever comes back, and this is from both sides. I can try to ping from AWS to my local network and the router sees the packet, but there's never a response.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Mon Oct 19, 2015 10:33 am
by ulysses
the router sees the packet, but there's never a response.
Make sure you accept the packets in the /ip firewall filter forward chain

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Sun Nov 01, 2015 6:47 pm
by itcoop
Regardless if this thread is stale, it offers some good intelligence. I believe the trouble in your ipsec configuration is this:

level=require

it should be changed to:

level=unique

Mikrotik defaults an ipsec policy to level=require which allows a single pair of bidirectional crypto tunnels (SPI's) to be reused for different ipsec policies; this works when VPN is used between Mikrotik to Mikrotik but not to any other vendor's equipment. Using this level, "unique" SPI's will now be established for every IP range (NAT'd or otherwise).

As a rule-of-thumb for using Mikrotik ipsec VPNs to differing vendors equipment (Cisco, Juniper, AWS, or unknown), I always set level=unique before troubleshooting the tunnel. This simply makes the tunnel much more compliant with the RFC.

Next, you configure five IPsec policies as below. You should be creating six policies, but because of the Microtik limitation, you will have to get along with these five:
0    ;;; AWS Tunnels
      src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

1    src-address=169.254.249.26/32 src-port=any dst-address=169.254.249.25/32 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0 

2    src-address=169.254.249.30/32 src-port=any dst-address=169.254.249.29/32 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

3    src-address=169.254.249.25/32 src-port=any dst-address=169.254.249.26/32 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.119 sa-dst-address=x.x.x.x proposal=default priority=0 

4    src-address=169.254.249.29/32 src-port=any dst-address=169.254.249.30/32 dst-port=any protocol=all action=encrypt level=unique
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.120 sa-dst-address=x.x.x.x proposal=default priority=0
      
5	src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Sun Nov 01, 2015 11:38 pm
by ulysses
As a rule-of-thumb for using Mikrotik ipsec VPNs to differing vendors equipment (Cisco, Juniper, AWS, or unknown), I always set level=unique before troubleshooting the tunnel. This simply makes the tunnel much more compliant with the RFC.
AWS requires that you aren't creating more than two SA per tunnel, so in case with AWS it is not allowed to set this value to "unique". A quote from http://docs.aws.amazon.com/AmazonVPC/la ... quirements:
You are limited to 1 unique Security Association (SA) pair per tunnel (1 inbound and 1 outbound), and therefore 2 unique SA pairs in total for 2 tunnels (4 SAs). Some devices use policy-based VPN and will create as many SAs as ACL entries. Therefore, you may need to consolidate your rules and then filter so you don't permit unwanted traffic.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed Nov 25, 2015 2:27 pm
by dne
Quick and dirty but seems to work.
:global activeGatewayAWS [/ip route get [/ip route find dst-address=172.31.0.0/16 bgp active] gateway]
:global saDstAddress
:if ($activeGatewayAWS=ACTIVE_BGP_GATEWAY_ADDR) do={:global saDstAddress ADDR_GW_1} else={:global saDstAddress ADDR_GW_2}
/ip ipsec policy set [/ip ipsec policy find dst-address=172.31.0.0/16] sa-dst-address=$saDstAddress

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed Nov 25, 2015 2:32 pm
by ulysses
Quick and dirty but seems to work.
:global activeGatewayAWS [/ip route get [/ip route find dst-address=172.31.0.0/16 bgp active] gateway]
:global saDstAddress
:if ($activeGatewayAWS=ACTIVE_BGP_GATEWAY_ADDR) do={:global saDstAddress ADDR_GW_1} else={:global saDstAddress ADDR_GW_2}
/ip ipsec policy set [/ip ipsec policy find dst-address=172.31.0.0/16] sa-dst-address=$saDstAddress
What is the trigger for the script? Or it is just to manually switch between the channels?

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed Nov 25, 2015 3:40 pm
by dne
/system scheduler
add interval=1s name=setIPSecPolicy on-event=setIPSecPolicy ...
:global activeGatewayAWS
:global saDstAddress
:global IPSecPeer1 ADDR_GW_1
:global IPSecPeer2 ADDR_GW_2

:for timer from=1 to=6 step=1 do={
:global activeGatewayAWS [/ip route get [/ip route find dst-address=172.31.0.0/16 bgp active] gateway]
:if ($activeGatewayAWS=BGP_GATEWAY_ADDR1) do={:global saDstAddress $IPSecPeer1} else={:global saDstAddress $IPSecPeer2}

:if ( $saDstAddress!=[/ip ipsec policy get [/ip ipsec policy find dst-address=172.31.0.0/16] sa-dst-address]) do={
/ip ipsec policy set [/ip ipsec policy find dst-address=172.31.0.0/16] sa-dst-address=$saDstAddress
}
:delay 100ms
}
but i canĀ“t guarantee anything, because this is my first ros script an i did not use it in production

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed Nov 25, 2015 3:48 pm
by ulysses
Nice..

But it has to be tested well, since I am afraid there may be positive loopback between the activation of the correct IPSec policy and the automatic active BGP route selection. It may even get into constant oscillation switching back and forth. Probably that is the reason you have a loop with 100ms sleep to have it stabilised...

In any case, good work. Maybe I will try this with my tunnel.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed Nov 25, 2015 4:02 pm
by dne
To prevent flapping between both path we used a bgp filter to set a lower distance to the preferred path.

the 100ms delay is only to switch policy faster then one second.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Tue Feb 02, 2016 6:54 pm
by danielharkerdp
Sorry if I am bumping an old thread here, I can't see a date on the thread from my phone... Odd.

Regardless, does setting level=unique get around the MikroTik Limitation??

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Thu Feb 04, 2016 12:47 pm
by ulysses
Sorry if I am bumping an old thread here, I can't see a date on the thread from my phone... Odd.

Regardless, does setting level=unique get around the MikroTik Limitation??
Nope :(. Even otherwise, it is recommended against, since AWS explicitly states that it will only allow two SAs per VPN channel. I mentioned it somewhere up the thread, with a reference to the docs

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Thu Apr 14, 2016 5:26 pm
by italotejada
Hi,

Thanks for the guide, really is the only one that has worked well using my router Mikrotik. Currently I have three EC2 public instances running within my VPC, and once you establish the IPSec tunnel following the here mentioned steps, everything works perfect, but I can not access the public IP's of this instances, only private IP. If the IPSec tunnel is disabled, i can ping and access to the public IP's, but not private (which seems logical). I need create a route or rule within my Mikrotik to access the public IP's of my instances, once the IPSec tunnel established? I could give a hand with this? Thank you very much

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Fri Aug 12, 2016 11:27 am
by ptofevski
Hi all,

Has anyone tried the configuration with version 6.36 is there any change to the config, because I have a problem bringing up the tunnels, i can ping 169.x.x.x IP addresses but amazon says tunnel down, and can not ping the IPs inside the VPC.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed Sep 28, 2016 11:46 am
by lordzmd
Hello all,

I have done the following config on my Mikrotik
ip address add address=169.254.40.38/30 disabled=no interface=WAN_BR network=169.254.40.36

ip ipsec proposal add name="aws" auth-algorithms=sha1 enc-algorithms=aes-128-cbc lifetime=8m pfs-group=modp1024

ip ipsec policy add src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=MyPublicIP sa-dst-address=52.57.15.8 proposal=aws priority=0

ip ipsec policy add src-address=169.254.40.38/32 src-port=any dst-address=169.254.40.37/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=MyPublicIP sa-dst-address=52.57.15.8 proposal=aws priority=0

ip ipsec peer add address=52.57.15.8/32 local-address=MyPublicIP passive=no port=500 auth-method=pre-shared-key \
secret="ZtkhRAD9Oehw5PvFWMCgxEssgb_Yo97A" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no \
proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s \
dpd-maximum-failures=3
An I do not get any SA installed.

I see although some IPSEC traffic in the logs.

Also I am not ussing BGP but static routes, and indicate my local net on the AWS side when creating VPN.

Also here is my firewall config.
add action=accept chain=input comment="Permit ICMP" protocol=icmp
add action=accept chain=input comment="Permit already established packages" \
    connection-state=established,related
add action=accept chain=input comment=PPTP dst-port=1723 protocol=tcp
add action=accept chain=input protocol=gre
add action=accept chain=input comment="Allow IKE" dst-port=500 in-interface=\
    WAN_BR protocol=udp
add action=accept chain=input comment="Allow IPSec-esp" in-interface=WAN_BR \
    protocol=ipsec-esp
add action=accept chain=input comment="Allow IPSec-ah" protocol=ipsec-ah
add action=accept chain=input dst-port=1234 protocol=udp
add action=drop chain=input comment="Drop all other traffic" in-interface=\
    WAN_BR
add action=accept chain=forward comment=\
    "Accept established connections in Forward" connection-state=\
    established,related
add action=drop chain=forward comment="Drop all Invalid on Forward" \
    connection-state=invalid
add action=drop chain=forward comment="Drop New connections in forward" \
    connection-nat-state=!dstnat connection-state=new in-interface=WAN_BR
/ip firewall nat
add action=masquerade chain=srcnat comment=NAT out-interface=WAN_BR
I specialy did not connect the second tunnel and also can say that Mikrotik is still having the bug mentioned above.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Wed Sep 28, 2016 2:50 pm
by lordzmd
The problem of the SA was solved. Now I am looking into the routing problems. I am able to ping machines in my Data Center from the AWS cloud but not able to do the oposite. Investigating.

UPDATE:
Problem solved.Seems Static routing has a slightly configuration.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Tue Apr 04, 2017 5:33 am
by itcoop
I've been running a modification of this configuration for a month now. The major revision I made to this config is to terminate each AWS VPN tunnel to separate 1100AHX2's as opposed to only one: this removed the "bug" and, most importantly, allows me to perform Mikrotik maintenance without taking the AWS VPN down.

Today I found that the performance was lacking during a large file transfer. Over a 2Gb raw connection, I was only seeing 30Mb going over the tunnel. Running pcap, I found many retransmissions.

Adding the following mangle rule adjusted the MTU over each tunnel; I'm now getting close to 750Mb on each tunnel:
/ip firewall mangle
add action=change-mss chain=forward dst-address=[my.vpc.net.aws] new-mss=1379 \
passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=!0-1379

The MMS adjustment was suggested in the IPSec Generic document from AWS:

IPSec ESP (Encapsulating Security Payload) inserts additional
headers to transmit packets. These headers require additional space,
which reduces the amount of space available to transmit application data.
To limit the impact of this behavior, we recommend the following
configuration on your Customer Gateway:
- TCP MSS Adjustment : 1379 bytes
- Clear Don't Fragment Bit : enabled
- Fragmentation : Before encryption

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Tue Apr 18, 2017 9:57 pm
by mark2015
I have setup a VPN from my Mikrotik router to my AWS VPC as per instruction above (and oddly enough have gotten it work previously). Today however it is not working. I cannot get a full round ping to work from either AWS->myHost or myHost->AWS.

When pinging from AWS->myHost I see that packets are hitting the
1 chain=dstnat action=accept src-address=10.10.0.0/16 in-interface=ether1-gateway
rule as the packet count increases as I keep the pings going; however, they never reach my host (I'm running a tcpdump).

On the other side of the test myHost->AWS, I can actually see the ping hit the ec2 instance (via tcpdump) however, it never arrives back at my source. It feels like the router is dropping packets from AWS destined to my internal network for some reason. Any suggestion on what the issue maybe or how to debug?

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Fri Aug 11, 2017 10:38 pm
by Xblackheart
I have setup a VPN from my Mikrotik router to my AWS VPC as per instruction above (and oddly enough have gotten it work previously). Today however it is not working. I cannot get a full round ping to work from either AWS->myHost or myHost->AWS.
That is worrisome. I am having a very similar issue. We took our vpn down for business reasons, and now, I'm getting BGP propagation, installed SA's everything that you would think. but no routing, no pings, no ssh.

I've been staring at this all day, At least I'm not the only one.

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Mon Oct 09, 2017 1:28 am
by mark2015
I have created a simple parameterized script to help with this whole process. All you need to do is populate the local vars at the top of the script.
The first two vars are specified in CIDR, and all the other vars are just IPs

Code: Select all

:local awsVpcCidr
:local onPremCidr

:local tunnelOneInsideIPVirtualPrivateGateway
:local tunnelOneOutsideIPVirtualPrivateGateway
:local tunnelOneInsideIPCustomerGateway

:local tunnelTwoInsideIPCustomerGateway
:local tunnelTwoInsideIPVirtualPrivateGateway
:local tunnelTwoOutsideIPVirtualPrivateGateway

:local myPublicIP [/ip address get [find interface="ether1-gateway"] address];

# add ip address $customer gateway
/ip address add address=$tunnelTwoInsideIPCustomerGateway/30 interface=ether1-gateway
/ip address add address=$tunnelOneInsideIPCustomerGateway/30 interface=ether1-gateway


/ip ipsec proposal set 0 auth-algorithms=sha1 enc-algorithms=aes-128-cbc lifetime=8m pfs-group=modp1024


# AWS Tunnels - ipsec policy (the first one doesn't work for me) - see if this ever does start to work
#/ip ipsec policy add src-address=0.0.0.0/0 src-port=any dst-address=$awsVpcCidr dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \
# sa-src-address=$myPublicIP sa-dst-address=$tunnelTwoOutsideIPVirtualPrivateGateway proposal=default priority=0

#
/ip ipsec policy add src-address=0.0.0.0/0 src-port=any dst-address=$awsVpcCidr dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \
sa-src-address=$myPublicIP sa-dst-address=$tunnelOneOutsideIPVirtualPrivateGateway proposal=default priority=0

/ip ipsec policy add src-address=$tunnelOneInsideIPCustomerGateway/32 src-port=any dst-address=$tunnelOneInsideIPVirtualPrivateGateway/32 \
dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \
sa-src-address=$myPublicIP sa-dst-address=$tunnelOneOutsideIPVirtualPrivateGateway proposal=default priority=0

/ip ipsec policy add src-address=$tunnelTwoInsideIPCustomerGateway/32 src-port=any dst-address=$tunnelTwoInsideIPVirtualPrivateGateway/32 \
dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \
sa-src-address=$myPublicIP sa-dst-address=$tunnelTwoOutsideIPVirtualPrivateGateway proposal=default priority=0

/ip ipsec policy add src-address=$tunnelOneInsideIPVirtualPrivateGateway/32 src-port=any dst-address=$tunnelOneInsideIPCustomerGateway/32 \
dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \
sa-src-address=$tunnelOneOutsideIPVirtualPrivateGateway sa-dst-address=$myPublicIP proposal=default priority=0

/ip ipsec policy add src-address=$tunnelTwoInsideIPVirtualPrivateGateway/32 src-port=any dst-address=$tunnelTwoInsideIPCustomerGateway/32 \
dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \
sa-src-address=$tunnelTwoOutsideIPVirtualPrivateGateway sa-dst-address=$myPublicIP proposal=default priority=0


# ip sec peers:
#0 AWS VPC Tunnel #2
/ip ipsec peer add address=$tunnelTwoOutsideIPVirtualPrivateGateway/32 local-address=$myPublicIP passive=no port=500 auth-method=pre-shared-key \
secret="5f5qC810t6mrrKxGdpIbu0gmDQUyI9rk" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no \
proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s \
dpd-maximum-failures=3

#1 AWS VPC Tunnel #1
/ip ipsec peer add address=$tunnelOneOutsideIPVirtualPrivateGateway/32 local-address=$myPublicIP passive=no port=500 auth-method=pre-shared-key \
secret="26X18YmCMDhybMqh8QpcT7yLqshY4r4c" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no \
proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s \
dpd-maximum-failures=3


#firewall filters
/ip firewall filter add chain=input action=accept protocol=ipsec-esp src-address=$tunnelOneOutsideIPVirtualPrivateGateway dst-address=$myPublicIP in-interface=ether1-gateway
/ip firewall filter add chain=input action=accept protocol=udp src-address=$tunnelOneOutsideIPVirtualPrivateGateway dst-address=$myPublicIP in-interface=ether1-gateway src-port=500 dst-port=500
/ip firewall filter add chain=input action=accept protocol=ipsec-esp src-address=$tunnelTwoOutsideIPVirtualPrivateGateway dst-address=$myPublicIP in-interface=ether1-gateway
/ip firewall filter add chain=input action=accept protocol=udp src-address=$tunnelTwoOutsideIPVirtualPrivateGateway dst-address=$myPublicIP in-interface=ether1-gateway src-port=500 dst-port=500

#firewall filter
/ip firewall filter add chain=input action=accept protocol=tcp src-address=$tunnelOneInsideIPVirtualPrivateGateway dst-address=$tunnelOneInsideIPCustomerGateway dst-port=179
/ip firewall filter add chain=input action=accept protocol=tcp src-address=$tunnelTwoInsideIPVirtualPrivateGateway dst-address=$tunnelTwoInsideIPCustomerGateway dst-port=179

#firewall filter
# VPC at AWS us-west-2x
/ip firewall filter add chain=forward action=accept src-address=$awsVpcCidr in-interface=ether1-gateway
/ip firewall filter add chain=forward action=accept dst-address=$awsVpcCidr in-interface=ether2-master-local


#nat rule
# critically important to AWS connectivity that this rule be ahead of "masquerade".
/ip firewall nat add chain=srcnat action=src-nat to-addresses=$onPremCidr dst-address=$awsVpcCidr


#The following firewall NAT filter rule is OPTIONAL and to be used only if the public address you supplied to AWS for your router is different from the public address used to masquerade your private NAT network. Don't use this unless you have two public addresses! And note the exclamation point!!!
#i don't have 2 public ip addresses...
#chain=srcnat action=masquerade src-address=!$myPublicIP out-interface=ether1-gateway


#nat rule
# critically important to AWS connectivity that this rule be ahead of any NAT port-mapping
/ip firewall nat add chain=dstnat action=accept src-address=$awsVpcCidr in-interface=ether1-gateway


#bgp instances
/routing bgp instance add name="vgw-1" as=65530 router-id=$tunnelTwoInsideIPCustomerGateway redistribute-connected=no redistribute-static=yes redistribute-rip=no \
redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table=""

/routing bgp instance add name="vgw-2" as=65530 router-id=$tunnelOneInsideIPCustomerGateway redistribute-connected=no redistribute-static=yes redistribute-rip=no \
redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table=""


#bgp peers
/routing bgp peer add name="awsvpc1" instance=vgw-1 remote-address=$tunnelTwoInsideIPVirtualPrivateGateway remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no \
route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=$tunnelTwoInsideIPCustomerGateway \
default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no

/routing bgp peer add name="awsvpc2" instance=vgw-2 remote-address=$tunnelOneInsideIPVirtualPrivateGateway remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no \
route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=$tunnelOneInsideIPCustomerGateway \
default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no


# bpg to advertise network:
/routing bgp network add network=$onPremCidr synchronize=yes

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Thu Jan 24, 2019 12:37 pm
by indnti
Another solution without mirror-policies and a supernet config to bypass the mikrotik one network limitation on IPsec policies:
http://biplane.com.au/blog/?p=406

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Fri Jun 21, 2019 10:10 pm
by nitrag
wowsers! spent a long time on troubleshooting routing + timeout issues. Make sure on the AWS side (VPC > Routing Tables > [Your VPC]) that you are propagating the routes back to your router!
Screen Shot 2019-06-21 at 3.06.20 PM.jpg
Also, I updated the script for the dynamic routing config to support 6.44.3 here:
https://github.com/nitrag/aws-vpn-mikro ... ter-config

Re: Amazon AWS VPN -- A Working Configuration Example and Bug

Posted: Mon Oct 14, 2019 8:03 pm
by radrexvalera
I'm a total noob on running the github files that I downloaded.

How am I going to run these files ( script )?