Skip to main content

Linux Link Aggregation (Bonding)

This guide provides instructions for configuring link aggregation in Linux. Link aggregation, or bonding, combines multiple network interfaces for improved performance and resilience. Learn how to set up and optimize link aggregation configurations efficiently.

Required Linux Modules​

root@dent-1:~# lsmod
bonding               245760  0
tls                   151552  1 bonding

Topology:​

topology2

Bond Modes​

We will go through the following bond modes configuration on dent-1 and dent-2 devices:

Active-Backup​

Active-Backup mode utilizes one active interface with another standby interface for fault tolerance. Provides failover only.

the mode uses arp request for monitoring link availability, alternatively it can use MIIMON for monitoring.

Configuration​

  • create bond1 with following parameters:
    • Active-Backup as bonding mode.
    • enp0s4 as primary link.
    • arp_interval 1000ms to send arp request every 1 sec.
    • arp_ip_target is 1.1.1.2 which is the br1 ip on dent-2, if no arp response from 1.1.1.2 switchover will be triggered.
  • set bond1 as master on enp0s4 and enp0s5.
  • set add bond1 and enps011 (where PC1 is connected) to bridge br1.
dent-1# conf t
dent-1(config)# links-iproute2
dent-1(config-links-iproute2)# bond bond1
dent-1(config-bond[name='bond1'])# bond-info mode active-backup
dent-1(config-bond[name='bond1'])# bond-info primary enp0s4
dent-1(config-bond[name='bond1'])# bond-info arp_interval 1000
dent-1(config-bond[name='bond1'])# bond-info arp_ip_target 1.1.1.2
dent-1(config-bond[name='bond1'])# master br1
dent-1(config-bond[name='bond1'])# exit
dent-1(config-links-iproute2)# link enp0s4
dent-1(config-[name='enp0s4'])# master bond1
dent-1(config-[name='enp0s4'])# exit
dent-1(config-links-iproute2)# link enp0s5
dent-1(config-[name='enp0s5'])# master bond1
dent-1(config-[name='enp0s5'])# exit
dent-1(config-links-iproute2)# link enp0s11
dent-1(config-[name='enp0s11'])# master br1
dent-1(config-[name='enp0s11'])# admin-status up
dent-1(config-[name='enp0s11'])# commit
note

Links must be DOWN before adding them to the bond device.

Verify​

  • We can see that the active link is the primary one enp0s4
root@dent-1:~# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: enp0s4 (primary_reselect always)
Currently Active Slave: enp0s4

  • if we try to ping from PC1 (192.168.10.1) to PC2 (192.168.10.3):
PC1> ping 192.168.10.3
64 bytes from 192.168.10.3: icmp_seq=373 ttl=64 time=3.47 ms
64 bytes from 192.168.10.3: icmp_seq=374 ttl=64 time=3.56 ms
64 bytes from 192.168.10.3: icmp_seq=375 ttl=64 time=3.54 ms
64 bytes from 192.168.10.3: icmp_seq=376 ttl=64 time=1.53 ms
64 bytes from 192.168.10.3: icmp_seq=377 ttl=64 time=3.77 ms
64 bytes from 192.168.10.3: icmp_seq=378 ttl=64 time=3.72 ms
  • Currently, the bond1 link is continusouly monitoring the availablility of enp0s4 (primary/active) link using arp requests to the specified arp_ip_target:

arp_ip_capture

  • Now we shut down enp0s4 (primary/active) on dent-2, and check if the failover happen on dent-1 and if enp0s5 become active link.
dent-2(config-links-iproute2)# link enp0s4
dent-2(config-[name='enp0s4'])# admin-status down
dent-2(config-[name='enp0s4'])# commit

from the logs we can see the switchover is triggered and enp0s5 takes over:

root@dent-1:~# 
[25711.548944] bond1: (slave enp0s4): link status definitely down, disabling slave
[25711.554258] device enp0s4 left promiscuous mode
[25711.568937] bond1: now running without any active interface!
[25711.579690] br1: port 1(bond1) entered disabled state
[25711.583592] br1: topology change detected, propagating
[25712.636845] bond1: (slave enp0s5): link status definitely up
[25712.642074] bond1: (slave enp0s5): making interface the new active one
[25712.646451] device enp0s5 entered promiscuous mode
[25712.671080] bond1: active interface up!
[25712.672767] br1: port 1(bond1) entered blocking state
[25712.673948] br1: port 1(bond1) entered listening state
[25728.125383] br1: port 1(bond1) entered learning state
[25743.484653] br1: port 1(bond1) entered forwarding state
[25743.485585] br1: topology change detected, sending tcn bpdu

if we check the /proc/net/bonding/bond1, we can see enp0s5 is active now:

root@dent-1:~# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: enp0s4 (primary_reselect always)
Currently Active Slave: enp0s5
tip

When STP is enabled and the switchover triggered, there will be some packet drops, and the connectivity will be lost for few seconds. if we disable the STP on br1 (stp_state = 0) the link will immediately move from blocking to forwarding state resulting in no drops during failover.

  • Let's bring enp0s4 up again, and we should see that enp0s4 become active again, as we have primary_reselect = always:
dent-2(config-links-iproute2)# link enp0s4
dent-2(config-[name='enp0s4'])# admin-status up
dent-2(config-[name='enp0s4'])# commit

And here we can see that enp0s4 become active again:

root@dent-1:~# [25847.868834] bond1: (slave enp0s4): link status definitely up
[25847.874298] bond1: (slave enp0s4): making interface the new active one
[25847.879784] device enp0s5 left promiscuous mode
[25847.894837] device enp0s4 entered promiscuous mode
[25850.492669] br1: port 1(bond1) neighbor 8000.0c:8c:30:dd:00:01 lost
[25851.004572] bond1: (slave enp0s5): link status definitely down, disabling slave

Balance-rr​

This mode Sequentially sends packets over available interfaces for increased throughput, but may cause TCP out-of-order issues. Offers load balancing and failover.

Configuration:​

dent-1# conf t
dent-1(config)# links-iproute2
dent-1(config-links-iproute2)# bond bond1
dent-1(config-bond[name='bond1'])# bond-info mode balance-rr
dent-1(config-bond[name='bond1'])# bond-info arp_interval 1000
dent-1(config-bond[name='bond1'])# bond-info arp_ip_target 1.1.1.2
dent-1(config-bond[name='bond1'])# master br1
dent-1(config-bond[name='bond1'])# exit
dent-1(config-links-iproute2)# link enp0s4
dent-1(config-[name='enp0s4'])# master bond1
dent-1(config-[name='enp0s4'])# exit
dent-1(config-links-iproute2)# link enp0s5
dent-1(config-[name='enp0s5'])# master bond1
dent-1(config-[name='enp0s5'])# exit
dent-1(config-links-iproute2)# link enp0s11
dent-1(config-[name='enp0s11'])# master br1
dent-1(config-[name='enp0s11'])# admin-status up
dent-1(config-[name='enp0s11'])# commit

Verify​

  • When flapping enp0s5 on dent-2:
[30398.588782] bond1: (slave enp0s5): interface is now down
[30424.188809] bond1: (slave enp0s5): interface is now up
  • We can see the current active link member:
root@dent-1:~# cat /proc/net/bonding/bond1

Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0
ARP Polling Interval (ms): 1000
ARP IP target/s (n.n.n.n form): 1.1.1.2

Slave Interface: enp0s4 <--
MII Status: up          <--
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 1
Permanent HW addr: 0c:c7:c2:08:00:01
Slave queue ID: 0

Slave Interface: enp0s5 <--
MII Status: up          <--
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 2
Permanent HW addr: 0c:c7:c2:08:00:02
Slave queue ID: 0

Balance-xor​

This mode Uses a hash policy to select interface for packet transmission, ensuring specific traffic goes over the same interface. Offers load balancing and failover.

Configuration​

We configured the bond1 as balance-xr and set the hashing policy to use layer3 and layer4 info.

dent-1# conf t
dent-1(config)# links-iproute2
dent-1(config-links-iproute2)# bond bond1
dent-1(config-bond[name='bond1'])# bond-info mode balance-xor
dent-1(config-bond[name='bond1'])# bond-info xmit_hash_policy layer3+4
dent-1(config-bond[name='bond1'])# bond-info arp_interval 1000
dent-1(config-bond[name='bond1'])# bond-info arp_ip_target 1.1.1.2
dent-1(config-bond[name='bond1'])# master br1
dent-1(config-bond[name='bond1'])# exit
dent-1(config-links-iproute2)# link enp0s4
dent-1(config-[name='enp0s4'])# master bond1
dent-1(config-[name='enp0s4'])# exit
dent-1(config-links-iproute2)# link enp0s5
dent-1(config-[name='enp0s5'])# master bond1
dent-1(config-[name='enp0s5'])# exit
dent-1(config-links-iproute2)# link enp0s11
dent-1(config-[name='enp0s11'])# master br1
dent-1(config-[name='enp0s11'])# admin-status up
dent-1(config-[name='enp0s11'])# commit

Verify​

In the following capture we can see that same link is used for a single flow, here we are testing with ping which will count L3 info only:

balance-xor-capture

Broadcast​

This mode Sends everything over all standby interfaces. Provides failover only.

Configuration​

dent-1# conf t
dent-1(config)# links-iproute2
dent-1(config-links-iproute2)# bond bond1
dent-1(config-bond[name='bond1'])# bond-info mode broadcast
dent-1(config-bond[name='bond1'])# bond-info arp_interval 1000
dent-1(config-bond[name='bond1'])# bond-info arp_ip_target 1.1.1.2
dent-1(config-bond[name='bond1'])# master br1
dent-1(config-bond[name='bond1'])# exit
dent-1(config-links-iproute2)# link enp0s4
dent-1(config-[name='enp0s4'])# master bond1
dent-1(config-[name='enp0s4'])# exit
dent-1(config-links-iproute2)# link enp0s5
dent-1(config-[name='enp0s5'])# master bond1
dent-1(config-[name='enp0s5'])# exit
dent-1(config-links-iproute2)# link enp0s11
dent-1(config-[name='enp0s11'])# master br1
dent-1(config-[name='enp0s11'])# admin-status up
dent-1(config-[name='enp0s11'])# commit

Verify​

From PC1 icmp outputs, can see that the packet now is being duplicated and sent out via all active links

PC1> ping 192.168.10.3
4 bytes from 192.168.10.3: icmp_seq=159 ttl=64 time=3.78 ms
64 bytes from 192.168.10.3: icmp_seq=159 ttl=64 time=4.90 ms (DUP!)
64 bytes from 192.168.10.3: icmp_seq=159 ttl=64 time=6.40 ms (DUP!)
64 bytes from 192.168.10.3: icmp_seq=159 ttl=64 time=6.70 ms (DUP!)
64 bytes from 192.168.10.3: icmp_seq=160 ttl=64 time=1.56 ms
64 bytes from 192.168.10.3: icmp_seq=160 ttl=64 time=2.05 ms (DUP!)
64 bytes from 192.168.10.3: icmp_seq=160 ttl=64 time=2.26 ms (DUP!)
64 bytes from 192.168.10.3: icmp_seq=160 ttl=64 time=2.38 ms (DUP!)
note

The request is duplicated to two packets, and response as well, that's why we see the (DUP!) flags.

802.3ad​

This mode Bundles multiple connections in parallel for redundancy and load balancing. Requires IEEE 802.1AX support on switches. Provides redundancy and load balancing.

Configuration​

dent-1# conf t
dent-1(config)# links-iproute2
dent-1(config-links-iproute2)# bond bond1
dent-1(config-bond[name='bond1'])# bond-info lacp_active on
dent-1(config-bond[name='bond1'])# bond-info lacp_rate slow 
dent-1(config-bond[name='bond1'])# master br1 
dent-1(config-bond[name='bond1'])# admin-status up 
dent-1(config-bond[name='bond1'])# exit
dent-1(config-links-iproute2)# link enp0s4
dent-1(config-[name='enp0s4'])# master bond1 
dent-1(config-[name='enp0s4'])# exit
dent-1(config-links-iproute2)# link enp0s5
dent-1(config-[name='enp0s5'])# master bond1
dent-1(config-[name='enp0s5'])# commit
info

802.3ad is dynamic link aggregation mode, it detects link down by exchanging LACP messages, unlike previous modes where we had to configure arp_ip_target and arp_interval for monitoring link availability.

Verify​

  • We can see the LACP messages exchanged between dent-1 and dent-2:

lacp_slow

  • Now PC1 can ping PC2:
PC1> ping 192.168.10.3
PING 192.168.10.3 (192.168.10.3) 56(84) bytes of data.
64 bytes from 192.168.10.3: icmp_seq=1 ttl=64 time=1.14 ms
64 bytes from 192.168.10.3: icmp_seq=2 ttl=64 time=1.41 ms
64 bytes from 192.168.10.3: icmp_seq=3 ttl=64 time=4.82 ms
64 bytes from 192.168.10.3: icmp_seq=4 ttl=64 time=3.00 ms
64 bytes from 192.168.10.3: icmp_seq=5 ttl=64 time=3.52 ms
  • Now if we shutdown enp0s4(the current select link for the ping test) on dent-2, dent-1 will failover to enp0s5:
dent-2(config-links-iproute2)# link enp0s4
dent-2(config-[name='enp0s4'])# admin-status down
dent-2(config-[name='enp0s4'])# commit
PC1>
64 bytes from 192.168.10.3: icmp_seq=22 ttl=64 time=3.00 ms
64 bytes from 192.168.10.3: icmp_seq=23 ttl=64 time=2.83 ms
64 bytes from 192.168.10.3: icmp_seq=24 ttl=64 time=3.14 ms
64 bytes from 192.168.10.3: icmp_seq=25 ttl=64 time=2.83 ms
From 192.168.10.1 icmp_seq=47 Destination Host Unreachable
From 192.168.10.1 icmp_seq=48 Destination Host Unreachable
From 192.168.10.1 icmp_seq=49 Destination Host Unreachable
<...>
From 192.168.10.1 icmp_seq=88 Destination Host Unreachable
From 192.168.10.1 icmp_seq=91 Destination Host Unreachable
From 192.168.10.1 icmp_seq=92 Destination Host Unreachable
64 bytes from 192.168.10.3: icmp_seq=93 ttl=64 time=7.99 ms
64 bytes from 192.168.10.3: icmp_seq=94 ttl=64 time=3.84 ms
64 bytes from 192.168.10.3: icmp_seq=95 ttl=64 time=18.3 ms
64 bytes from 192.168.10.3: icmp_seq=96 ttl=64 time=4.26 ms
64 bytes from 192.168.10.3: icmp_seq=97 ttl=64 time=3.77 ms
tip

We can see the failover happen, however it took around 1 minutes to failover causing many packet drops, to solve this we set lacp_rate = fast:

dent-1(config-links-iproute2)# bond bond1
dent-1(config-bond[name='bond1'])# bond-info lacp_rate fast

now the failover will be instant:

PC1> ping 192.168.10.3
64 bytes from 192.168.10.3: icmp_seq=100 ttl=64 time=4.40 ms
64 bytes from 192.168.10.3: icmp_seq=101 ttl=64 time=3.62 ms
64 bytes from 192.168.10.3: icmp_seq=102 ttl=64 time=3.55 ms
64 bytes from 192.168.10.3: icmp_seq=103 ttl=64 time=3.66 ms
64 bytes from 192.168.10.3: icmp_seq=104 ttl=64 time=3.74 ms
64 bytes from 192.168.10.3: icmp_seq=105 ttl=64 time=11.1 ms <- failover
64 bytes from 192.168.10.3: icmp_seq=106 ttl=64 time=4.09 ms
64 bytes from 192.168.10.3: icmp_seq=107 ttl=64 time=4.19 ms
64 bytes from 192.168.10.3: icmp_seq=108 ttl=64 time=2.80 ms
64 bytes from 192.168.10.3: icmp_seq=109 ttl=64 time=1.47 ms

Balance-tlb​

Adaptive transmit load balancing creates channel bonding without needing any unique switch assistance. Outbound traffic gets divided based on the current load on each slave, which is calculated relative to its speed. Incoming traffic is handled by the current active slave, and if it fails, another slave assumes the MAC address of the failed one. Offers load balancing and failover.

Topology​

topology3

Configuration​

dent-1# conf t
dent-1(config)# links-iproute2
dent-1(config-links-iproute2)# bond bond1
dent-1(config-bond[name='bond1'])# bond-info mode balance-tlb
dent-1(config-bond[name='bond1'])# master br1 
dent-1(config-bond[name='bond1'])# admin-status up 
dent-1(config-bond[name='bond1'])# exit
dent-1(config-links-iproute2)# link enp0s4
dent-1(config-[name='enp0s4'])# master bond1 
dent-1(config-[name='enp0s4'])# exit
dent-1(config-links-iproute2)# link enp0s5
dent-1(config-[name='enp0s5'])# master bond1
dent-1(config-links-iproute2)# link enp0s6
dent-1(config-[name='enp0s6'])# master bond1
dent-1(config-[name='enp0s6'])# commit

Verify​

  • From /proc/net/bonding/bond1 we can see that enp0s4 is chosen for receive, and transmitting will be load-balanced on enp0s5 and enp0s6.
root@dent-1:~# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: transmit load balancing
Primary Slave: None
Currently Active Slave: enp0s4
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

Slave Interface: enp0s4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 0c:c7:c2:08:00:01
Slave queue ID: 0

Slave Interface: enp0s5
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 0c:c7:c2:08:00:02
Slave queue ID: 0

Slave Interface: enp0s6
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 0c:c7:c2:08:00:03
Slave queue ID: 0
info

Balance-tlb only load balance the transmitted traffic, the incoming traffic will be handled by the Currently Active Slave link.

  • Let's start ping from PC1 to PC2:
PC1> ping 192.168.10.3
64 bytes from 192.168.10.3: icmp_seq=100 ttl=64 time=4.40 ms
64 bytes from 192.168.10.3: icmp_seq=101 ttl=64 time=3.62 ms
64 bytes from 192.168.10.3: icmp_seq=102 ttl=64 time=3.55 ms
  • By taking a tcpdump on enp0s6 we can see it's transmitting the pings requests, while enp0s4 is receiving the pings replies.
root@dent-1:~# tcpdump -i enp0s6
[65658.968933] device enp0s6 entered promiscuous mode
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s6, link-type EN10MB (Ethernet), capture size 262144 bytes
12:39:45.657158 IP 192.168.10.1 > 192.168.10.3: ICMP echo request, id 24, seq 189, length 64
12:39:46.657179 IP 192.168.10.1 > 192.168.10.3: ICMP echo request, id 24, seq 190, length 64
12:39:47.658181 IP 192.168.10.1 > 192.168.10.3: ICMP echo request, id 24, seq 191, length 64
12:39:48.659210 IP 192.168.10.1 > 192.168.10.3: ICMP echo request, id 24, seq 192, length 64
12:39:49.659534 IP 192.168.10.1 > 192.168.10.3: ICMP echo request, id 24, seq 193, length 64
12:39:50.661078 IP 192.168.10.1 > 192.168.10.3: ICMP echo request, id 24, seq 194, length 64


root@dent-1:~# tcpdump -i enp0s4
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s4, link-type EN10MB (Ethernet), capture size 262144 bytes
12:39:29.659580 IP 192.168.10.3 > 192.168.10.1: ICMP echo reply, id 24, seq 173, length 64
12:39:30.659454 IP 192.168.10.3 > 192.168.10.1: ICMP echo reply, id 24, seq 174, length 64
12:39:31.659542 IP 192.168.10.3 > 192.168.10.1: ICMP echo reply, id 24, seq 175, length 64
12:39:32.659508 IP 192.168.10.3 > 192.168.10.1: ICMP echo reply, id 24, seq 176, length 64
12:39:33.659140 IP 192.168.10.3 > 192.168.10.1: ICMP echo reply, id 24, seq 177, length 64
12:39:34.659288 IP 192.168.10.3 > 192.168.10.1: ICMP echo reply, id 24, seq 178, length 64
tip

Balance-tlb will use the configured xmit_hash_policy to load balance the transmitted traffic, if you want to load balance the transmitted traffic based on link load you can set tlb_dynamic_lb = 1, but this requires support in the base drivers to retrieve the speed of each physical interface.

caution

Balance-tlb can't be configured at both device ends at the same time (dent-1 and dent-2), as it try to enforce egress and ingress traffic.

Balance-alb​

Adaptive load balancing includes both balance-transmit load balancing and receive-load balancing for IPv4 traffic, without needing any special switch support. Receive-load balancing is accomplished through ARP negotiation. The bonding driver intercepts ARP replies sent by the local system and replaces the source hardware address with the unique hardware address of one of the slaves in the bond. Consequently, various peers utilize distinct hardware addresses for the server. Provides adaptive load balancing (ALB) for IPv4 traffic.

Topology​

topology3

Configuration​

dent-1# conf t
dent-1(config)# links-iproute2
dent-1(config-links-iproute2)# bond bond1
dent-1(config-bond[name='bond1'])# bond-info mode balance-alb
dent-1(config-bond[name='bond1'])# master br1 
dent-1(config-bond[name='bond1'])# admin-status up 
dent-1(config-bond[name='bond1'])# exit
dent-1(config-links-iproute2)# link enp0s4
dent-1(config-[name='enp0s4'])# master bond1 
dent-1(config-[name='enp0s4'])# exit
dent-1(config-links-iproute2)# link enp0s5
dent-1(config-[name='enp0s5'])# master bond1
dent-1(config-links-iproute2)# link enp0s6
dent-1(config-[name='enp0s6'])# master bond1
dent-1(config-[name='enp0s6'])# commit

Verify​

  • We can see that all slaves are active:
root@dent-1:~# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: adaptive load balancing
Primary Slave: None
Currently Active Slave: enp0s4
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

Slave Interface: enp0s4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 0c:c7:c2:08:00:01
Slave queue ID: 0

Slave Interface: enp0s5
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 0c:c7:c2:08:00:02
Slave queue ID: 0

Slave Interface: enp0s6
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 0c:c7:c2:08:00:03
Slave queue ID: 0

info

Balance-alb does exactly like Balance-tlb, but it also tries to load balance the incoming traffic by intercepting locally generated arp reply and insert one of the bond link's phy address in the answer.

If we ping from PC2 to 192.168.10.11 ( br1 ip on dent-1),
then if we take a look at the arp entries on PC2 we will see the MAC address of enp0s4 not br1:

PC2> ping 192.168.10.11
PING 192.168.10.11 (192.168.10.11) 56(84) bytes of data.
64 bytes from 192.168.10.11: icmp_seq=1 ttl=64 time=35.5 ms
64 bytes from 192.168.10.11: icmp_seq=2 ttl=64 time=1.27 ms
64 bytes from 192.168.10.11: icmp_seq=3 ttl=64 time=2.41 ms

PC2> arp -a
? (192.168.10.11) at 0c:c7:c2:08:00:01 [ether] on eth0 
root@Toolbox-3:~#

0c:c7:c2:08:00:01 is the mac address of enp0s4 on dent-1.