From: https://github.com/Mellanox/mlxsw/wiki/L3-Tunneling
Introduction
Since L3 tunneling is fundamentally a routing technology, the switch where tunnels should to be configured needs to have routing enabled.
#enable routing
echo 1 > /proc/sys/net/ipv4/ip_forward
echo 1 > /proc/sys/net/ipv6/conf/all/forwarding
sysctl -w net.ipv4.ip_forward = 1
sysctl -w net.ipv6.conf.all.forwarding = 1
Topology
In abstract, the reason to create an IP-in-IP tunnel is to connect two IP networks separated by another IP network. In the example here, the two domains to be connected are represented by two hosts with arbitrarily-chosen addresses 192.168.1.33 resp. 192.168.2.33. The two hosts are each connected to a tunnel endpoint, addressed 1.2.3.4/31, which wraps up the host traffic and delivers it through a tunnel to the other endpoint. The encapsulated traffic travels over a transport network, here addressed 192.168.99.0/24.
In tunneling parlance, the traffic flowing between the two separated IP domains is called overlay traffic, and correspondingly the network where it flows overlay network. The encapsulated traffic on the other hand is called underlay traffic, and the network where it flows underlay network.
+--------------+ +--------------+
| | | |
| host1 | | host2 |
| | | |
| 192.168.1.33 | | 192.168.2.33 |
| + | | + |
| | | | | |
+--------------+ +--------------+
| |
+--------------+ +--------------+
| | | | | |
| + | | + | Overlay
| 192.168.1.1 | | 192.168.2.1 | - - - - - -
| | | | Underlay
| switch1 | | switch2 |
| | | |
| 1.2.3.4 | | 1.2.3.5 |
| + | | + |
| | | | | |
| 192.168.99.1 | | 192.168.99.2 |
| + | | + |
| | | | | | | |
+--------------+ +--------------+
| |______________________| |
'--------------------------'
Overlay Configuration
#host1
ip link set eth0 up
ip addr add 192.168.1.33/24 dev eth0
ip route add 192.168.2.0/24 via 192.168.1.1
#host2
ip link set eth0 up
ip addr add 192.168.2.33/24 dev eth0
ip route add 192.168.1.0/24 via 192.168.2.1
#switch1
ip link set sw1p49 up
ip addr add 192.168.1.1/24 dev sw1p49
#switch2
ip link set sw1p49 up
ip addr add 192.168.2.1/24 dev sw1p49
Tunnel Configuration
There are two ways that GRE tunnel endpoint can be set up. Either overlay and underlay are each in a different VRF (which we call hierarchical configuration), or they share the same VRF (flat configuration).
flat configuration
+------------------( switch )-------------------+
| |
| overlay GRE transport |
---|-+ 192.168.1.1 1.2.3.4 +-- 192.168.99.1 +=|===
| |
+-----------------------------------------------+
#sw1
ip tunnel add name g mode gre local 1.2.3.4 remote 1.2.3.5 tos inherit
ip link set g up
ip addr add 1.2.3.4/32 dev g
ip link set sw1p50 up
ip addr add 192.168.99.1/24 dev sw1p50
ip route add 1.2.3.5/32 via 192.168.99.2
ip route add 192.168.2.0/24 dev g
#sw2
ip tunnel add name g mode gre local 1.2.3.5 remote 1.2.3.4 tos inherit
ip link set g up
ip addr add 1.2.3.5/32 dev g
ip link set sw1p50 up
ip addr add 192.168.99.2/24 dev sw1p50
ip route add 1.2.3.4/32 via 192.168.99.1
ip route add 192.168.1.0/24 dev g
Hierarchical Configuration
This is similar in spirit to the flat configuration, however now the GRE netdevice has a bound device that selects a VRF to use for underlay traffic. Typically this would be a different VRF than the one with the GRE netdevice itself, but it does not have to be.
+------------------( switch )-------------------+
| | <-- VRF ol
| overlay GRE |
---|-+ 192.168.1.1 ^ |
| | |
| - - - - - - - - - - -|- - - - - - - - - - - - |
| v | <-- VRF ul
| dummy transport |
| 1.2.3.4 +-- 192.168.99.1 +=|===
| |
+-----------------------------------------------+
#First, create the VRFs themselves.
ip link add name ol type vrf table 10
ip link set ol up
ip link add name ul type vrf table 20
ip lik set ul up
#Second,create the dummy device to use to select the underlay VRF.
ip link add name d type dummy
ip link set d up
ip link set d master ul
ip addr add 1.2.3.4/32 dev d //1.2.3.5 for sw2
#Third, create tunnel
#sw1
ip tunnel add name g mode gre local 1.2.3.4 remote 1.2.3.5 dev d tos inherit
ip link set g master ul
ip link set g up
#sw2
ip tunnel add name g mode gre local 1.2.3.5 remote 1.2.3.4 dev d tos inherit
ip link set g master ul
ip link set g up
#Fourth, config route
#sw1
ip route add vrf ol 192.168.2.0/24 dev g
ip link set sw1p50 up
ip addr add 192.168.99.1/24 dev sw1p50
ip route add 1.2.3.5/32 via 192.168.99.2
ip link set sw1p49 master ol
ip link set sw1p50 master ul
#sw2
ip route add vrf ol 192.168.2.0/24 dev g
ip link set sw1p50 up
ip addr add 192.168.99.2/24 dev sw1p50
ip route add 1.2.3.4/32 via 192.168.99.1
ip link set sw1p49 master ol
ip link set sw1p50 master ul