In part 1 of this series, we went over the basic operations of MPLS, and built a simple lab to show the forwarding of labels across a network. But MPLS alone on an island doesn’t accomplish much. The power of MPLS comes in it being transparent to overlay traffic, allowing for multiple protocols to traverse the network without the MPLS environment needing to understand that traffic at every hop. In this post, we’re going to build services called L3VPNs across our MPLS environment.
A Note on VPNs
The term VPN has become synonymous with an encrypted path between 2 endpoints, but strictly speaking, a VPN has no requirement for encryption. I know, BLASPHEMY! But in simple terms a VPN really just forwards overlay traffic across an underlay. All of the earliest VPNs didn’t have encryption.
L3VPNs are VPNs in the traditional sense, but not the modern sense. You should never consider L3VPNs a security measure, outside of segmenting traffic from each other. All traffic will be unencrypted unless you add encryption.
What Are L3VPNs
L3VPNs traverse an MPLS environment to communicate at Layer3 between Provider Edge (PE) routers. All of the labels we’ve discussed so far have been what are known as Transport labels (also referred to as “outer” and “top of stack” labels). L3VPNs introduce a new label that is added to the stack called a VPN label (also referred to as “inner”, “s bit”, and “bottom of stack” labels). This label will map to an IP prefix (or service) on the PE router.
In part one of this series, I mentioned that once all of the labels are popped, an IP lookup will be performed. You may ask why that VPN label is needed at all in that case. The addition of the VPN labels creates a mechanism to see how and where that lookup is performed. It enables the use of multiple Virtual Routing and Forwarding (VRF) instances. Not just that, it allows for multi-address family support (like IPv4, IPv6, and even EVPN if supported). The label can say “perform the lookup for IPv6 in VRF A”.
NOTE: Sadly, using MPLS as a dataplane for EVPN is not currently supported in VyOS. If it ever is, it’ll open up a whole new ballgame for connectivity.
Operation of MPLS with L3VPN
We went over the push-swap-pop function of MPLS in part one of this series. The operation of MPLS in an environment with L3VPNs is mostly the same, except the packet will now have 2 MPLS shim headers. One is the transport label, and the other is the VPN label.
The transport label will be pushed by the head-end PE, and then swapped on every hop along the path. Once we get to the next-to-last hop in the path, the transport label will be fully popped (remember PHP from part one). Once the packet reaches the tail-end PE , the only label left in the stack will now be the VPN label. The PE will perform the lookup for the service in the address-family the label indicates, and will finally pop the VPN label as the packet goes towards that services.
Let’s build something to see this in action.
Toplogy
For this topology, we’re going to have 2 ‘PE’ routers, and 2 ‘P’ routers. We’re also going to introduce a new type of router referred to as a Customer Edge (CE) router. Each PE will have 1 ‘CE’ hanging off of it.
Network Info
NOTE: Some of these terms may be unfamiliar right now, but will be covered as we configure them.
Like in part one, all MPLS devices are interconnected using IPs based on a device number. In this case we’ll use the last octet of the loopbacks (e.g. PE1 and P1 are connected with the 10.1.2.0/24 network; P1 and P2 will connect using the 10.2.3.0/24 network, and P2 and PE2 will connect using 10.3.4.0/24)
- Loopbacks:
- PE1 – 10.0.0.1/32
- P1 – 10.0.0.2/32
- P2 – 10.0.0.3/32
- PE2 – 10.0.0.4/32
- VRF:
- Name – A
- RD – 65000:1
- RT – 65000:1
- CE Network Info:
- CE1 Peering Subnet – 172.16.0.0/30
- CE2 Peering Subnet – 172.16.0.4/30
- CE1 Client Subnet – 10.0.1.0/24
- CE2 Client Subnet – 10.0.2.0/24
- CE1 ASN – 65001
- CE2 ASN – 65002
Just like in part one, we need to build OSPF between all of our Label Switched Routers (LSR), this includes P and PE routers.
PE1:
set interfaces dummy dum0 address '10.0.0.1/32'
set interfaces ethernet eth0 address '10.1.2.1/24'
set protocols ospf interface dum0 area '0'
set protocols ospf interface eth0 area '0'
set protocols ospf interface eth0 network 'point-to-point'
P1:
set interfaces dummy dum0 address '10.0.0.2/32'
set interfaces ethernet eth0 address '10.1.2.2/24'
set interfaces ethernet eth1 address '10.2.3.2/24'
set protocols ospf interface dum0 area '0'
set protocols ospf interface eth0 area '0'
set protocols ospf interface eth1 area '0'
set protocols ospf interface eth0 network 'point-to-point'
set protocols ospf interface eth1 network 'point-to-point'
P2:
set interfaces dummy dum0 address '10.0.0.3/32'
set interfaces ethernet eth0 address '10.3.4.3/24'
set interfaces ethernet eth1 address '10.2.3.3/24'
set protocols ospf interface dum0 area '0'
set protocols ospf interface eth0 area '0'
set protocols ospf interface eth1 area '0'
set protocols ospf interface eth0 network 'point-to-point'
set protocols ospf interface eth1 network 'point-to-point'
PE2:
set interfaces dummy dum0 address '10.0.0.4/32'
set interfaces ethernet eth0 address '10.3.4.4/24'
set protocols ospf interface dum0 area '0'
set protocols ospf interface eth0 area '0'
set protocols ospf interface eth0 network 'point-to-point'
We should see all loopbacks on any of our LSRs.
vyos@PE1# run show ip route ospf
O 10.0.0.1/32 [110/1] via 0.0.0.0, dum0 onlink, weight 1, 1d18h26m
O>* 10.0.0.2/32 [110/65536] via 10.1.2.2, eth0, weight 1, 00:00:21
O>* 10.0.0.3/32 [110/65537] via 10.1.2.2, eth0, weight 1, 00:00:21
O>* 10.0.0.4/32 [110/65538] via 10.1.2.2, eth0, weight 1, 00:00:21
LDP Configuration
Now we need to configure label imposition. Like part one, we’re going to use Label Distribution Protocol (LDP).
PE1:
set protocols mpls interface 'eth0'
set protocols mpls ldp discovery transport-ipv4-address '10.0.0.1'
set protocols mpls ldp interface 'eth0'
set protocols mpls ldp router-id '10.0.0.1'
P1:
set protocols mpls interface 'eth0'
set protocols mpls interface 'eth1'
set protocols mpls ldp discovery transport-ipv4-address '10.0.0.2'
set protocols mpls ldp interface 'eth0'
set protocols mpls ldp interface 'eth1'
P2:
set protocols mpls interface 'eth0'
set protocols mpls interface 'eth1'
set protocols mpls ldp discovery transport-ipv4-address '10.0.0.3'
set protocols mpls ldp interface 'eth0'
set protocols mpls ldp interface 'eth1'
set protocols mpls ldp router-id '10.0.0.3'
PE2:
set protocols mpls interface 'eth0'
set protocols mpls ldp discovery transport-ipv4-address '10.0.0.4'
set protocols mpls ldp interface 'eth0'
set protocols mpls ldp router-id '10.0.0.4'
Let’s make sure we have labels assigned to all of our loopbacks.
vyos@PE1# run show mpls ldp binding
AF Destination Nexthop Local Label Remote Label In Use
ipv4 10.0.0.1/32 10.0.0.2 imp-null 16 no
ipv4 10.0.0.2/32 10.0.0.2 16 imp-null yes
ipv4 10.0.0.3/32 10.0.0.2 17 17 yes
ipv4 10.0.0.4/32 10.0.0.2 18 18 yes
Everything looks good! Let’s get to configure the L3VPNs.
L3VPN Config
For this section, we’re going to specifically configure L3VPNs in the VPNv4 address family. As the name implies, these will be for IPv4 prefixes.
The first thing we need to configure is the VRFs. These will be the same for each of the PE routers.
set vrf name A table '101'
set vrf name A protocols bgp address-family ipv4-unicast export vpn
set vrf name A protocols bgp address-family ipv4-unicast import vpn
set vrf name A protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name A protocols bgp address-family ipv4-unicast rd vpn export '65000:1'
set vrf name A protocols bgp address-family ipv4-unicast route-target vpn both '65000:1'
set vrf name A protocols bgp system-as '65000'
A lot of that may be hieroglyphs to you if you haven’t seen them before. Let’s break down the config into smaller parts.
This command will create a VRF called A, and map it to linux table 101.
set vrf name A table '101'
These commands will allow the IPv4 addresses to be imported/exported between the local VRF and the VPNv4 address family.
set vrf name A protocols bgp address-family ipv4-unicast export vpn
set vrf name A protocols bgp address-family ipv4-unicast import vpn
This command will tell VPNv4 what VPN label to assign for this traffic. If auto is selected, a label will be assigned by the Zebra Routing Daemon. You can statically define these if desired.
set vrf name A protocols bgp address-family ipv4-unicast label vpn export 'auto'
What Is A Route Distingusher (RD)
When using VRFs, you’re able to use the same subnet on many different VRFs. You can have the 10.0.0.0/24 prefix exist 1000 times in an environment. BGP however, only likes to forward on the best path.
By default, BGP will only advertise 1 of those thousand routes (likely the oldest in this case). VPNv4 needs a way to make each of those 1000 prefixes be unique.
That is the point of a route-distinguisher. It is a measure to ensure that each prefix is unique. If a VRF has an RD of 65000:1, it’ll make the full VPNv4 prefix look like this: 65000:1:10.0.0.0/24.
RDs are 64-bit values typically configured as colon-separated values in these formats:
- IP:16-bit:
- Example: 10.0.0.1:1
- 32-bit:16-bit:
- Example: 123456:1
set vrf name A protocols bgp address-family ipv4-unicast rd vpn export '65000:1'
What is a Route Target
Like with route distinguishers, route-targets help solve the issue with having multiple identical prefixes in different VRFs.. When a remote PE receives those 1000 different 10.0.0.0/24 prefixes, it needs some way to know which BGP table those routes need to get dumped into. That is the purpose of route-targets.
Route-targets get attached to BGP Prefixes (NLRIs) as extended-communities. When the remote PE receives the NLRI, it is now able to look at it’s VRF config, and see which VRFs these prefixes need to get dumped into.
The actions that are taken with route-targets is import and export.
An export just says that any VRF prefix that it advertises to remote PEs will add a route-target to the NLRI.
An import says that an NLRI received from a remote PE is allowed to be placed in the BGP table for that VRF.
NOTE: It is possible to import multiple route-targets in any VRF. This is the basis for route leaking. Each VRF just needs to put the other VRF’s route-target and leaking has occured.
The both command indicates that both an import and export action should be performed on this route-target.
set vrf name A protocols bgp address-family ipv4-unicast route-target vpn both '65000:1'
VPNv4 BGP Config
iBGP is configured between the loopback addresses on the PEs in the ipv4-vpn (VPNv4) address-family. iBGP is desired in a VPNv4 deployment due to its native next-hop-unchanged behavior. When scaling the network, a route-reflector is typically used. iBGP allows the reflected routes to still have their original next-hops, allowing for traffic to go directly between route-reflector clients, instead of needing to traverse the route-reflector like hub and spoke.
PE1:
set protocols bgp address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.4 address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.4 remote-as '65000'
set protocols bgp neighbor 10.0.0.4 update-source 'dum0'
set protocols bgp system-as '65000'
PE2:
set protocols bgp address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.1 address-family ipv4-vpn
set protocols bgp neighbor 10.0.0.1 remote-as '65000'
set protocols bgp neighbor 10.0.0.1 update-source 'dum0'
set protocols bgp system-as '65000'
PE to CE BGP Config
The connection between the PE and CE will be a very simple eBGP peering.
PE1:
set interfaces ethernet eth2 address '172.16.0.1/30'
set interfaces ethernet eth2 vrf 'A'
set vrf name A protocols bgp address-family ipv4-unicast network 172.16.0.0/30
set vrf name A protocols bgp neighbor 172.16.0.2 address-family ipv4-unicast
set vrf name A protocols bgp neighbor 172.16.0.2 remote-as '65001'
PE2:
set interfaces ethernet eth2 address '172.16.0.5/30'
set interfaces ethernet eth2 vrf 'A'
set vrf name A protocols bgp address-family ipv4-unicast network 172.16.0.4/30
set vrf name A protocols bgp neighbor 172.16.0.6 address-family ipv4-unicast
set vrf name A protocols bgp neighbor 172.16.0.6 remote-as '65002'
CE1:
set interfaces dummy dum0 address '10.0.1.1/24'
set interfaces ethernet eth0 address '172.16.0.2/30'
set protocols bgp address-family ipv4-unicast redistribute connected
set protocols bgp neighbor 172.16.0.1 address-family ipv4-unicast
set protocols bgp neighbor 172.16.0.1 remote-as '65000'
set protocols bgp system-as '65001'
CE2:
set interfaces dummy dum0 address '10.0.2.1/24'
set interfaces ethernet eth0 address '172.16.0.6/30'
set protocols bgp address-family ipv4-unicast redistribute connected
set protocols bgp neighbor 172.16.0.5 address-family ipv4-unicast
set protocols bgp neighbor 172.16.0.5 remote-as '65000'
set protocols bgp system-as '65002'
Let’s look at one of the CE BGP tables.
vyos@CE1# run show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 10.0.1.0/24 0.0.0.0 0 32768 ?
*> 10.0.2.0/24 172.16.0.1 0 0 65000 65002 ?
* 172.16.0.0/30 0.0.0.0 0 32768 ?
*> 172.16.0.4/30 172.16.0.1 0 0 65000 65002 ?
Like we did in part 1, let’s look at some packet captures as a ping traverses the network.
vyos@CE1# run ping 10.0.2.1 source-address 10.0.1.1 count 1
64 bytes from 10.0.2.1: icmp_seq=1 ttl=62 time=7.50 ms
PE1 to P1
We can see there are now 2 labels in the stack. You can see the S Bit is set to 1 for the second label, indicating it is the bottom of the stack label. Let’s see where we’re getting these. First, the transport (top of stack) label.
vyos@PE1# run show ip bgp vrf A
Network Next Hop Metric LocPrf Weight Path
*> 10.0.2.0/24 10.0.0.4@0< 0 100 0 65002 ?
vyos@PE1# run show mpls ldp binding 10.0.0.4/32
AF Destination Nexthop Local Label Remote Label In Use
ipv4 10.0.0.4/32 10.0.0.2 18 18 yes
We can see that to get to 10.0.2.0/24, we need to go to 10.0.0.4. If we look at our LDP Bindings, we can see that the address 10.0.0.4/32 has a remote Label of 18, so this is what we need to put as a transport label.
Let’s see how the VPN (bottom of stack) label is determined.
vyos@PE1# run show bgp ipv4 vpn 10.0.2.0/24
BGP routing table entry for 172.16.0.5:2:10.0.2.0/24, version 14
Extended Community: RT:65000:1
Remote label: 144
We can see the VPNv4 route has a VPN label of 144.
NOTE: You can also see the RD appended to the IP prefix to create the VPNv4 prefix we mentioned earlier.
NOTE: You can also see the route-target extended community that has been attached to the NLRI.
If you want a quick way to see these values, you can see them in the “show ip route vrf” command.
vyos@PE1# run show ip route vrf A
VRF A:
B> 10.0.2.0/24 [200/0] via 10.0.0.4 (vrf default) (recursive), label 144, weight 1, 00:18:23
* via 10.1.2.2, eth0 (vrf default), label 18/144, weight 1, 00:18:23
P1 to P2
We see the exact same labels as the first hop. You might be even be tricked into thinking you’re looking at the wrong capture, but this is from the correct segment. This further illustrates what we saw in part one on how the labels are locally significant, and not global values. We can see that the TTL decremented by one, which is the indicator that this is a different capture.
Let’s quickly verify with the LDP Binding table.
vyos@P1# run show mpls ldp binding 10.0.0.4/32
AF Destination Nexthop Local Label Remote Label In Use
ipv4 10.0.0.4/32 10.0.0.3 18 18 yes
Yep, the labels are what we should expect.
P2 to PE2
We can see here that the only label we have is now the VPN label. Notice the S-Bit?
This should mean that we’re recieving implicit null from PE2 on P2, let’s verify.
vyos@P2# run show mpls ldp binding 10.0.0.4/32
AF Destination Nexthop Local Label Remote Label In Use
ipv4 10.0.0.4/32 10.0.0.4 18 imp-null yes
Yep! As expected, we’re receiving implicit null. That’s the full LSP, everything is just normal IP switching to the CE IP after that.
One good thing to look at to observe the VPN nature of this traffic is to look at the routing table for P1 or P2.
vyos@P1# run show ip route vrf all
VRF default:
O>* 10.0.0.1/32 [110/2] via 10.1.2.1, eth0, label implicit-null, weight 1, 1d01h02m
O 10.0.0.2/32 [110/1] via 0.0.0.0, dum0 onlink, weight 1, 1d01h03m
C>* 10.0.0.2/32 is directly connected, dum0, 1d01h03m
O>* 10.0.0.3/32 [110/2] via 10.2.3.3, eth1, label implicit-null, weight 1, 1d01h01m
O>* 10.0.0.4/32 [110/3] via 10.2.3.3, eth1, label 18, weight 1, 03:16:17
O 10.1.2.0/24 [110/1] is directly connected, eth0, weight 1, 1d01h03m
C>* 10.1.2.0/24 is directly connected, eth0, 1d01h03m
O 10.2.3.0/24 [110/1] is directly connected, eth1, weight 1, 1d01h02m
C>* 10.2.3.0/24 is directly connected, eth1, 1d01h02m
You can see the P nodes have no idea of the VRF A prefixes (10.0.1.0/24 and 10.0.2.0/24), yet CE1 and CE2 are fully able to communicate.
Conclusion
That’s it for this post! Hopefully MPLS is becoming a little less intimidating at this point. For all of the power of MPLS, the configuration isn’t all that difficult.
Leave a Reply