Redhat7 Static Route for Two Interfaces
Overview
Virtual machines (VMs) in Azure can have multiple virtual network interface cards (NICs) attached to them. A common scenario is to have different subnets for front-end and back-end connectivity.
To send to or from a secondary network interface, you have to manually add persistent routes to the operating system for each secondary network interface.
In this article, we experiment adding static route for multiple NICs in Azure Readhat VM.
Secondary NIC in Azure VM
Add Two NICs
You may create Azure VM with multiple NIC via powershell, CLI and ARM.
"networkProfile": {
"networkInterfaces": [
{
"id": "[resourceId('Microsoft.Network/networkInterfaces', parameters('networkInterfaceName1'))]",
"properties": {
"primary": true
}
},
{
"id": "[resourceId('Microsoft.Network/networkInterfaces', parameters('networkInterfaceName2'))]",
"properties": {
"primary": false
}
}
]
},Secondary NIC Not Working
Outbound from second interface to Internet does not work, while it works for the primary network interface.
$curl -v --interface eth0 bing.com
* About to connect() to bing.com port 80 (#0)
* Trying 13.107.21.200...
* Local Interface eth0 is ip 10.99.0.7 using address family 2
* SO_BINDTODEVICE eth0 failed with errno 1: Operation not permitted; will do regular bind
* Local port: 0
* Connected to bing.com (13.107.21.200) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: bing.com
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently$ curl -v --interface eth1 bing.com
[sudo] password for clouduser:
* About to connect() to bing.com port 80 (#0)
* Trying 13.107.21.200...
* Local Interface eth1 is ip 10.99.0.164 using address family 2
* Local port: 0
Also second interface is not reachable from another internal network.
ping 10.99.0.164 -c 4
PING 10.99.0.164 (10.99.0.164) 56(84) bytes of data.--- 10.99.0.164 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3059ms
From another internal network, run tcpdump, output shows no response from the second interface IP address.
$sudo tcpdump icmp -i eth0 -xx >dump1.txt
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
4 packets received by filter
0 packets dropped by kernel
$ cat dump1.txt
01:25:39.728790 IP myvm.internal.cloudapp.net > 10.99.0.164: ICMP echo request, id 8672, seq 1, length 64
01:25:40.753708 IP myvm.internal.cloudapp.net > 10.99.0.164: ICMP echo request, id 8672, seq 2, length 64
01:25:41.777766 IP myvm.internal.cloudapp.net > 10.99.0.164: ICMP echo request, id 8672, seq 3, length 64
01:25:42.801777 IP myvm.internal.cloudapp.net > 10.99.0.164: ICMP echo request, id 8672, seq 4, length 64
Add Static Route
Run IP Command to Add Route and Rule
#dev - device/interface tab - table id
$sudo ip route add 10.99.0.0/25 dev eth0 tab 1
$sudo ip route add 10.99.0.160/28 dev eth1 tab 2#each interface has its own default gateway
$sudo ip route add default via 10.99.0.1 dev eth0 tab 1
$sudo ip route add default via 10.99.0.161 dev eth1 tab 2#route packets based on source IP
$sudo ip rule add from 10.99.0.7/32 tab 1 priority 100
$sudo ip rule add from 10.99.0.164/32 tab 2 priority 200
$sudo ip route flush cache
Validation
Open google site from secondary NIC
$ curl -I --interface eth1 www.google.com
HTTP/1.1 200 OK $ping 10.99.0.164 -c 4
PING 10.99.0.164 (10.99.0.164) 56(84) bytes of data.
64 bytes from 10.99.0.164: icmp_seq=1 ttl=64 time=2.02 ms
64 bytes from 10.99.0.164: icmp_seq=2 ttl=64 time=1.18 ms
64 bytes from 10.99.0.164: icmp_seq=3 ttl=64 time=1.16 ms
64 bytes from 10.99.0.164: icmp_seq=4 ttl=64 time=1.41 ms
Persist Route Changes
Let’s add route and rule to configuration files.
$cd /etc/sysconfig/network-scripts$ cat route-eth0
10.99.0.0/25 dev eth0 tab 1
default via 10.99.0.1 dev eth0 tab 1$ cat rule-eth0
from 10.99.0.7/32 tab 1 priority 100$ cat route-eth1
10.99.0.160/28 dev eth1 tab 2
default via 10.99.0.161 dev eth1 tab 2$ cat rule-eth1
from 10.99.0.164/32 tab 2 priority 200
It’s unclear to me whether it’s Azure VM issue or Redhat limitation, after reboot, the configuration is not picked up.
Redhat docs says “NetworkManager supports policy-routing, but rules are not supported yet. The rules must be configured by the user running a custom script.”
Anyway, it is not super complicated to execute shell script at startup.
$sudo su - root#cat /var/scripts/configroute.sh
#!/bin/bash
sudo ip route add 10.99.0.0/25 dev eth0 tab 1
sudo ip route add 10.99.0.160/28 dev eth1 tab 2
sudo ip route add default via 10.99.0.1 dev eth0 tab 1
sudo ip route add default via 10.99.0.161 dev eth1 tab 2sudo ip rule add from 10.99.0.7/32 tab 1 priority 100
sudo ip rule add from 10.99.0.164/32 tab 2 priority 200sudo ip route flush cache#cat /etc/systemd/system/configroute.service
[Unit]
Description=configure routes
After=network.target[Service]
Type=idle
ExecStart=/var/scripts/configroute.sh
TimeoutStartSec=0[Install]
WantedBy=default.target#systemctl daemon-reload
#systemctl enable configroute.service
#systemctl start configroute.service
After reboot VM, let’s check the route and rule. Route and rule are added, since the custom script is executed during startup.
$ip route show tab 1
default via 10.99.0.1 dev eth0
10.99.0.0/25 dev eth0 scope link $ip route show tab 2
default via 10.99.0.161 dev eth1
10.99.0.160/28 dev eth1 scope link $ip rule show
0:from all lookup local
100:from 10.99.0.7 lookup 1
200:from 10.99.0.164 lookup 2
32766:from all lookup main
32767:from all lookup default
References
Cheat Sheet
tcpdump can capture multiple protocol packets, including tcp, udp, icmp.
#catpure packet to file
tcpdump -s 0 -w /path/nameofthecapture.pcap#if you want ICMP traffic
tcpdump icmp -i eth0 -xx
(-xx print both header and data of each packet)#if you want TCP traffic
tcpdump -c 100 tcp and port 80 or 443-c <n> | only capture n number of packets
-s 0 -w <file.pcap> | write a packet capture to file instead of standard out-s decreases the amount of packet buffering, set this to zero-n | avoid DNS lookup-nn | don't resolve DNS or Service
TCP Flags Flags [P.]. Typical values for this field include:
TCP Flags Flags [P.]. Typical values for this field include:Value Flag Type Description
S SYN Connection Start
F FIN Connection Finish
P PUSH Data push
R RST Connection reset
. ACK Acknowledgment
TCP Sequence and Acknowledgement Numbers Explained — MadPackets6 advanced tcpdump formatting options | Enable Sysadmin (redhat.com)
Create a Linux VM in Azure with multiple NICs — Azure Virtual Machines | Microsoft Docs
4.4. Configuring Static Routes with ip commands Red Hat Enterprise Linux 7 | Red Hat Customer Portal
4.5. Configuring Static Routes in ifcfg files Red Hat Enterprise Linux 7 | Red Hat Customer Portal
CentOS / RHEL 7 : How to create custom script to run automatically during boot — The Geek Diary
Besides running startup script through service, you may try https://docs.microsoft.com/en-us/troubleshoot/azure/virtual-machines/swap-file-not-recreated-linux-vm-restart
