UPDATE:
In the original post, you would need to drop into FRR’s VTYSH to configure a portion of the configuration since the necessary address-family was not exposed to the VyOS CLI. That has since been remedied under this feature request: https://vyos.dev/T5913.
NOTE: There appears to be a bug in this code that will remove the MPLS interface configuration after a reboot. If this happens, just run this command to put that config back.
rollback-soft 0
commit
In part one of this series, we created a basic Multipoint VPN solution using ZeroTier. You can find that post here:
Dynamic Multipoint VPN with ZeroTier and VyOS
This is a great solution if you have a single tenant, but what if you you need multiple tenants. Or what if you wanted to do microsegmentation of your routing? Because ZeroTier builds an emulated LAN for all of your nodes, you can run anything you can run over normal ethernet networks. This means we can achieve that multitenancy/microsegmentation with MPLS. Our config won’t even look too different than what we had in part one. We’re still going to use iBGP with route-reflection. We’re still going to use BGP Listeners for dynamic neighbors.
MPLS in Cisco DMVPN
We also talked a little bit about Cisco’s DMVPN in part 1. DMVPN also supports MPLS (originally called 2547oDMVPN after RFC2547; later just called MPLSoDMVPN), though it was in a different, more limiting way. MPLS Imposition was handled by NHRP, which worked great to maintain the dynamic spoke-to-spoke capability for IPv4 traffic, but it had some limitations. It couldn’t distribute labels beyond the DMVPN topology, which meant you couldn’t maintain label switched paths further into your network, forcing the DMVPN routers to always act as PEs. They did ultimately add a way to use spokes as P nodes (added in IOS-XE 17.2.1), but it is a bit clunky as it requires importing NHRP learned labels into the IPv4 Labeled Unicast address family.
You also couldn’t do Spoke-to-Spoke using IPv6 or MVPN, all of that traffic still needed to traverse the Hub since that traffic didn’t trigger any NHRP actions.
Our Topology
Our Topology will be mostly the same as we saw in Part 1. We are going to add a second hub, since it will allow us more practice in setting up the nodes.
Here is a flyby review of the steps needed to setup another node:
- Download VyOS
- We’re using 1.4.0-rc3 which can be found in this blog post: https://blog.vyos.io/vyos-1.4.0-rc3-release-candidate
- Boot the ISO image and install the image to a drive
- Configure internet routing and name resolution
- Install ZeroTier
- Map ZeroTier Interface to Ethernet Interface
- Join a ZeroTier Network
Installing VyOS to Disk: Follow the guided prompts
vyos@vyos:~$ install image
Configure basic routing and name resolution
vyos@vyos:~$ configure
vyos@vyos# set interfaces ethernet eth0 address dhcp
vyos@vyos# set system name-server 4.2.2.2
vyos@vyos# commit
Install ZeroTier
vyos@vyos# sudo su
root@vyos:/home/vyos# curl -s https://install.zerotier.com | sudo bash
Map ZeroTier Interface to Ethernet Interface and restart ZeroTier service
root@vyos:/var/lib/zerotier-one# cat >devicemap
# Example: <networkID>=<interface>
xxxxxxxxxxxxxxxx=eth10
^C
root@vyos:/var/lib/zerotier-one# sudo /etc/init.d/zerotier-one restart
Join a ZeroTier Network and authorize the new Hub
root@vyos:/var/lib/zerotier-one# zerotier-cli join xxxxxxxxxxxxxxxx
200 join OK
A note about Hubs
In DMVPN, the Hubs served not just as a way for spokes to know how to get to each other, but also served as a critical routing component in the hub and spoke topology. Traffic would need to originally traverse the hub always, and if a spoke-to-spoke connection could not be established, traffic would always traverse the hub.
In our design, the Hub is really just serving as a route server (though it can be an edge node for a site), which allows you to put it at any site that you’d like. You can put it in the cloud, and fully support an entirely on-premise solution.
Configuring VRFs
In this lab, we’re going to use 3 VRFs. You can obviously use more (or less) if that’s your requirement. Let’s configure the VRFs and LAN Interfaces first.
Key to be used for IP addressing of Spoke sites:
Spoke1 – Site Code 11
Spoke2 -Site Code 12
The VRFs get mapped to a routing table in Linux, which is what the numerical value after table is indicating.
set vrf name A table '101'
set vrf name B table '102'
set vrf name C table '103'
set interfaces ethernet eth1 vif 10 address '10.11.1.1/24'
set interfaces ethernet eth1 vif 10 vrf 'A'
set interfaces ethernet eth1 vif 20 address '10.11.2.1/24'
set interfaces ethernet eth1 vif 20 vrf 'B'
set interfaces ethernet eth1 vif 30 address '10.11.3.1/24'
set interfaces ethernet eth1 vif 30 vrf 'C'
Easy enough, now let’s configure the VPNv4 configuration for the VRFs.
Note on Route Distinguishers
The route distinguishers will be the same for each node per VRF. This doesn’t present a problem in this example, but it can limit what prefixes are advertised, so you may want to make them unique per site. Remember, these 2 prefixes are not the same:
1:1:0.0.0.0/0
1:2:0.0.0.0/0
By default, BGP will only forward it’s best path, so if both of those were 1:1:0.0.0.0/0, only 1 would reach the spokes (unless using addpath).
Now let’s get back to configuring our VRF VPNv4 configuration.
set vrf name A protocols bgp address-family ipv4-unicast export vpn
set vrf name A protocols bgp address-family ipv4-unicast import vpn
set vrf name A protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name A protocols bgp address-family ipv4-unicast rd vpn export '1:1'
set vrf name A protocols bgp address-family ipv4-unicast redistribute connected
set vrf name A protocols bgp address-family ipv4-unicast route-target vpn both '1:1'
set vrf name A protocols bgp system-as '65000'
set vrf name B protocols bgp address-family ipv4-unicast export vpn
set vrf name B protocols bgp address-family ipv4-unicast import vpn
set vrf name B protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name B protocols bgp address-family ipv4-unicast rd vpn export '2:2'
set vrf name B protocols bgp address-family ipv4-unicast redistribute connected
set vrf name B protocols bgp address-family ipv4-unicast route-target vpn both '2:2'
set vrf name B protocols bgp system-as '65000'
set vrf name C protocols bgp address-family ipv4-unicast export vpn
set vrf name C protocols bgp address-family ipv4-unicast import vpn
set vrf name C protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name C protocols bgp address-family ipv4-unicast rd vpn export '3:3'
set vrf name C protocols bgp address-family ipv4-unicast redistribute connected
set vrf name C protocols bgp address-family ipv4-unicast route-target vpn both '3:3'
set vrf name C protocols bgp system-as '65000'
The only thing that will change in the VRF config between spokes will be the LAN Subnets.
Configuring Hub BGP
Our 2 hubs will be VPNv4 route reflectors for our solution. The Hub’s will peer together to make a route-reflector Cluster, and both serve as route-reflectors for clients (our spokes).
We are going to need a loopback to build our VPNv4 adjacencies to. In VyOS, it is called a dummy interface. We’re going to pull IPs from the 10.0.0.0/24 subnet to populate the dummy interfaces on our nodes:
Hub1:
set interfaces dummy dum0 address '10.0.0.1/32'
Hub2:
set interfaces dummy dum0 address '10.0.0.2/32'
Let’s set the system AS for BGP. Like in Part 1, we’re going to be doing iBGP using AS 65000.
set protocols bgp system-as '65000'
Also like we did in Part 1 of this blog, we need to create a BGP listener to dynamically create peerings with spokes. This config will be identical on both Hubs.
set protocols bgp listen range 10.13.0.0/16 peer-group 'ZT_PEERS'
set protocols bgp peer-group ZT_PEERS remote-as '65000'
set protocols bgp peer-group ZT_PEERS address-family ipv4-labeled-unicast route-reflector-client
We will also need to create a listener for our VPNv4 peerings to the dummy interfaces.
set protocols bgp listen range 10.0.0.0/24 peer-group 'VPNv4_PEERS'
set protocols bgp peer-group VPNv4_PEERS address-family ipv4-vpn route-reflector-client
set protocols bgp peer-group VPNv4_PEERS remote-as '65000'
set protocols bgp peer-group VPNv4_PEERS update-source 'dum0'
Now we need to import our dummy interfaces into the ipv4-labeled-unicast address-family. We’re going to do this with “redistribute connected” like was shown in part 1 of this series. This make it so our hub configs will be mostly the same, which makes configuration management much easier. We filter our ZeroTier subnet from being imported so the Hub’s don’t get countless copies of that subnet.
set policy prefix-list ZT_LAN_PFX rule 10 action 'permit'
set policy prefix-list ZT_LAN_PFX rule 10 prefix '10.13.0.0/16'
set policy route-map DENY_ZT_LAN rule 10 action 'deny'
set policy route-map DENY_ZT_LAN rule 10 match ip address prefix-list 'ZT_LAN_PFX'
set policy route-map DENY_ZT_LAN rule 20 action 'permit'
set protocols bgp address-family ipv4-labeled-unicast
set protocols bgp address-family ipv4-unicast redistribute connected route-map 'DENY_ZT_LAN'
You may notice that we’re not actually importing into the ipv4-labeled-unicast address-family but the ipv4-unicast, which seems unintuitive, but that config is correct.
Create Peering between Hubs
We need to create our cluster of route-reflectors with our hubs. We will build an IPv4 Labeled Unicast peering, as well as a VPNv4 peering. In our lab, we’ll consider the hubs to not be at the same site, so the IPv4 Labeled Unicast peering will be using ZeroTier. If these were at the same site, you could use a physical connection. Notice that these are not configured with “route-reflector-client”, since these are themselves route-reflectors and not clients.
Hub1:
set protocols bgp neighbor 10.13.0.2 address-family ipv4-labeled-unicast
set protocols bgp neighbor 10.13.0.2 remote-as '65000'
set protocols bgp neighbor 10.0.0.2 address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.2 remote-as '65000'
set protocols bgp neighbor 10.0.0.2 update-source 'dum0'
Hub2:
set protocols bgp neighbor 10.13.0.1 address-family ipv4-labeled-unicast
set protocols bgp neighbor 10.13.0.1 remote-as '65000'
set protocols bgp neighbor 10.0.0.1 address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.1 remote-as '65000'
set protocols bgp neighbor 10.0.0.1 update-source 'dum0'
Configure MPLS
Label Imposition is already being handled with the IPv4 Labeled Unicast address-family, so all we really need to do is allow MPLS on our interfaces that will serve as a label switched path. The only interface in this lab that fits that description is eth10, which is the ZeroTier interface. If you do a physical interface between your Hubs, you’ll also need to add it here.
set protocols bgp interface eth10 mpls forwarding
set protocols mpls interface 'eth10'
Configuring The Spokes
All nodes will get an identical VRF config up to this point, so you can apply what was shown in the previous section, remembering to change the IPs for the LAN interfaces.
Let’s configure the dummy interface.
Spoke1:
set interfaces dummy dum0 address '10.0.0.11/32'
Spoke2:
set interfaces dummy dum0 address '10.0.0.12/32'
Now the policy for redistributing connected routes. This the same for all nodes.
set policy prefix-list ZT_LAN_PFX rule 10 action 'permit'
set policy prefix-list ZT_LAN_PFX rule 10 prefix '10.13.0.0/16'
set policy route-map DENY_ZT_LAN rule 10 action 'deny'
set policy route-map DENY_ZT_LAN rule 10 match ip address prefix-list 'ZT_LAN_PFX'
set policy route-map DENY_ZT_LAN rule 20 action 'permit'
Now let’s configure BGP. This will be identical for all spokes, whether it’s 2 or 200. Though we didn’t do it here, you can create peer-groups to clean up this section a little.
set protocols bgp address-family ipv4-labeled-unicast
set protocols bgp address-family ipv4-unicast redistribute connected route-map 'DENY_ZT_LAN'
set protocols bgp neighbor 10.0.0.1 address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.1 remote-as '65000'
set protocols bgp neighbor 10.0.0.1 update-source 'dum0'
set protocols bgp neighbor 10.0.0.2 address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.2 remote-as '65000'
set protocols bgp neighbor 10.0.0.2 update-source 'dum0'
set protocols bgp neighbor 10.13.0.1 address-family ipv4-labeled-unicast nexthop-self
set protocols bgp neighbor 10.13.0.1 remote-as '65000'
set protocols bgp neighbor 10.13.0.2 address-family ipv4-labeled-unicast nexthop-self
set protocols bgp neighbor 10.13.0.2 remote-as '65000'
set protocols bgp system-as '65000'
And now we need to enable MPLS on our interfaces….we’re getting close!
set protocols bgp interface eth10 mpls forwarding
set protocols mpls interface 'eth10'
And that’s it, our config is done.
Verification
Let’s check BGP:
vyos@Hub1:~$ show ip bgp summary
IPv4 VPN Summary (VRF default):
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
*10.0.0.11 4 65000 25 31 43 0 0 00:19:29 3 9 N/A
*10.0.0.12 4 65000 25 31 43 0 0 00:19:28 3 9 N/A
10.0.0.2 4 65000 51 62 43 0 0 00:42:31 6 9 N/A
Total number of neighbors 3
* - dynamic neighbor
2 dynamic neighbor(s), limit 100
IPv4 Labeled Unicast Summary (VRF default):
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
*10.13.0.11 4 65000 24 27 44 0 0 00:19:31 2 5 N/A
*10.13.0.12 4 65000 24 27 44 0 0 00:19:30 2 5 N/A
10.13.0.2 4 65000 55 59 44 0 0 00:48:38 4 4 N/A
Total number of neighbors 3
* - dynamic neighbor
2 dynamic neighbor(s), limit 100
This looks good so far (notice the ‘*’ before the peers in both address-families, indicating they’re dynamic neighbors) We can see we’re receiving 3 prefixes from each Spoke (10.0.0.11 and 10.0.0.12). We can also see we’re getting 6 prefixes from our other Hub (10.0.0.2), which are the 3 prefixes from each of our Spokes, totaling 6.
Let’s check the VPNv4 on a spoke to make sure route-reflection is working:
vyos@Spoke1:~$ show bgp ipv4 vpn
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1:1
*> 10.11.1.0/24 0.0.0.0@14< 0 32768 ?
UN=0.0.0.0 EC{1:1} label=82 type=bgp, subtype=5
* i10.12.1.0/24 10.0.0.12 0 100 0 ?
UN=10.0.0.12 EC{1:1} label=80 type=bgp, subtype=0
*>i 10.0.0.12 0 100 0 ?
UN=10.0.0.12 EC{1:1} label=80 type=bgp, subtype=0
Route Distinguisher: 2:2
*> 10.11.2.0/24 0.0.0.0@15< 0 32768 ?
UN=0.0.0.0 EC{2:2} label=85 type=bgp, subtype=5
* i10.12.2.0/24 10.0.0.12 0 100 0 ?
UN=10.0.0.12 EC{2:2} label=85 type=bgp, subtype=0
*>i 10.0.0.12 0 100 0 ?
UN=10.0.0.12 EC{2:2} label=85 type=bgp, subtype=0
Route Distinguisher: 3:3
*> 10.11.3.0/24 0.0.0.0@16< 0 32768 ?
UN=0.0.0.0 EC{3:3} label=84 type=bgp, subtype=5
* i10.12.3.0/24 10.0.0.12 0 100 0 ?
UN=10.0.0.12 EC{3:3} label=84 type=bgp, subtype=0
*>i 10.0.0.12 0 100 0 ?
UN=10.0.0.12 EC{3:3} label=84 type=bgp, subtype=0
Displayed 6 routes and 9 total paths
Everything looks great, this should mean that everything will work. Next we can configure our Customer Edge Connections. We should probably make sure it’s working at all first. Let’s ping the “VRF A” gateways between the spokes.
vyos@Spoke1:~$ ping 10.12.1.1 vrf A source-address 10.11.1.1
PING 10.12.1.1 (10.12.1.1) from 10.11.1.1 : 56(84) bytes of data.
--- 10.12.1.1 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4077ms
And a whole lot of nothing. Maybe the routes never actually made it into the VRF’s routing table.
vyos@Spoke1:~$ show ip route vrf A
VRF A:
C>* 10.11.1.0/24 is directly connected, eth1.10, 02:28:35
B> 10.12.1.0/24 [200/0] via 10.0.0.12 (vrf default) (recursive), label 80, weight 1, 00:29:10
* via 10.13.0.12, eth10 (vrf default), label implicit-null/80, weight 1, 00:29:10
Nope, that’s not it. Maybe it’s MPLS.
vyos@Spoke1:~$ show mpls table
Inbound Label Type Nexthop Outbound Label
-------------------------------------------------
81 BGP 10.13.0.1 implicit-null
82 BGP A -
83 BGP 10.13.0.2 implicit-null
84 BGP C -
85 BGP B -
86 BGP 10.13.0.12 implicit-null
No, labels look fine. We can also see that we have a label to the destination prefix in the FIB:
root@Spoke1:/home/vyos# ip route show table 101 | grep 10.12.1.0
10.12.1.0/24 nhid 384 encap mpls 80 via 10.13.0.12 dev eth10 proto bgp metric 20
So what’s happening? There’s a gotcha when using ZeroTier. ZeroTier has it’s own rules engine for traffic, and by default the only traffic allowed is ARP, IPv4, and IPv6. We can see these rules under our network section in ZeroTier Central.
We need to make sure that MPLS is allowed. MPLS is ethertype 0x8847. Let’s add that and save the changes.
You can read more about ZeroTier’s Rule’s Engine here: https://docs.zerotier.com/rules/
Now let’s retry our ping test.
vyos@Spoke1:~$ ping 10.12.1.1 vrf A source-address 10.11.1.1
PING 10.12.1.1 (10.12.1.1) from 10.11.1.1 : 56(84) bytes of data.
64 bytes from 10.12.1.1: icmp_seq=1 ttl=63 time=24.1 ms
64 bytes from 10.12.1.1: icmp_seq=2 ttl=63 time=2.00 ms
64 bytes from 10.12.1.1: icmp_seq=3 ttl=63 time=2.05 ms
And we have success. Things work much better when traffic is allowed.
Configure Customer Edge BGP Peerings
BGP configuration for CE connections is under the VRF config section. For simplicity, we’re only going to configure a CE peering for one of our VRFs, but the process would just be repeated as needed. We’re using the Site Codes defined earlier to populate the AS for the CEs.
Spoke1:
set vrf name A protocols bgp neighbor 10.11.1.2 address-family ipv4-unicast
set vrf name A protocols bgp neighbor 10.11.1.2 remote-as '65011'
Spoke2:
set vrf name A protocols bgp neighbor 10.12.1.2 address-family ipv4-unicast
set vrf name A protocols bgp neighbor 10.12.1.2 remote-as '65012'
Now we’ll configure some CE routers. I’m going to use some lightweight Cisco IOU routers for this, but they can really be anything that speaks eBGP.
Site1 CE:
interface Ethernet0/0.10
encapsulation dot1Q 10
ip address 10.11.1.2 255.255.255.0
!
router bgp 65011
neighbor 10.11.1.1 remote-as 65000
redistribute connected
Site2 CE:
interface Ethernet0/0.10
encapsulation dot1Q 10
ip address 10.12.1.2 255.255.255.0
!
router bgp 65011
neighbor 10.12.1.1 remote-as 65000
redistribute connected
You should see BGP come up on our CE devices.
*Jan 9 21:38:37.527: %BGP-5-ADJCHANGE: neighbor 10.12.1.1 vpn vrf A Up
Let’s see if we have routes for the other site’s VRF A.
Cust1#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 10.11.1.0/24 0.0.0.0 0 32768 ?
*> 10.12.1.0/24 10.11.1.1 0 0 65000 65012 ?
Looks good, final test, can we ping the other site’s CE router.
Cust1#ping 10.12.1.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.12.1.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 3/8/32 ms
And that’s it! There’s a lot more that you can do with this, now that you have the MPLS and ZeroTier part configured. Want to add VPNv6, it’s already there for you.
Conclusion
It’s pretty amazing how far Linux routing has come. This really only scratches the surface on what can be done with VyOS 1.4.
In the next part of this series, we will look at how to easily increase the scalability of this design if you have a massive network using regionalized designs.
Leave a Reply