Understanding ACI packet encapsulation and ACI packet format is an essential topic for network professionals who aim to gain a deep knowledge of Cisco ACI.
In this video article, I explore ACI iVXLAN encapsulation and explain the ACI packet fields and format.
Summary
- Cisco ACI is an overlay technology that uses a custom VXLAN called iVXLAN for packet encapsulation.
- ACI iVXLAN uses a custom UDP destination port, 48879, instead of the standard VXLAN UDP port 4789.
- When forwarding in the ACI fabric, all traffic is normalized as iVXLAN packets.
- The additional ACI iVXLAN encapsulation is 50 bytes.
- The Cisco Application Centric Infrastructure (ACI) Design Guide provides more knowledge about Cisco ACI.
Very nice explanation. Thank you for that!
But a question: You wrote “The additional ACI iVXLAN encapsulation is 50 bytes.”
Isn’t it more? At 08:20 you explain:
8 Bytes iVXLAN header
8 Bytes UDP header
20 Bytes IP header
14 Bytes Ethernet header “with 802.1Q Tag”
An untagged Ethernet header is 14 Bytes long. When you add a VLAN tag you need additional 4 Bytes to store the 802.1Q header. You also explained that the outer 802.1Q header is mandatory because of the subinterfaces.
I think it must be like this: “The additional ACI iVXLAN encapsulation is 50 bytes.””The additional ACI iVXLAN encapsulation is 54 bytes.”
Hi Lukas,
You’re right, however, I didn’t want to confuse the audiance because in all Cisco documents/books everyone mentioned the iVXLAN header length is 50 bytes. Here is a full detailed packet encapsulation that I took very long ago from the IPN device (out from the spine with iVXLAN encapsulation).
Sorry to be pedantic, but at 9:02 in the video you show the Outer Header of the ACI packet with an outer Dst IP address of Leaf4, but a Dst MAC address of Spine MAC – the dst MAC should be Leaf4 MAC, unless it was treated as a destination unknown, in which case the Dst IP would be the IPV4PROXY IP and the Dst MAC would be a special MAC address that the spine would accept.
Hello RedNectar,
I hope you are doing well, mate. Knowing that you are one of the most (if not the most) professionals posted in the Cisco community, I’m happy that you commented on my humble website, and I encourage you to do so.
Regarding your comment, I think you mixed the outer and inner header addressing! What you mentioned (IPV4PROXY) is correct for the inner addresses. However, the destination MAC of the outer header for the iVXLAN packet that Leaf1 sent to Leaf4 CANNOT be Leaf4’MAC because the links between leaves and spines are L3 links (routed subinterfaces using the infra vlan). Also, the src and dst outer IPs are all in different subnets since they all are /32 addresses (configured on loopback 0 for every VTEP/Leaf), which means the dst MAC should belong to the dst device (like the IP).
Technically, the MAC address in the outer header is a hop-by-hop address for each link; it can’t go beyond the direct link, and you can confirm that by having a packet capture in the spine switches.
Please pay attention to the fact that I was talking about the packet sent from Leaf 1 toward the spines; it hasn’t yet reached the spines.
Again, I’m happy to see you here, Chris. Wish you the very best.
You are absolutely right – not sure what planet I was on when I made that comment. I’ll put it down as a “senior moment” and thank you for your kind words. And as I become even more “senior” (I’m 71 yo) I’m really happy to see folk like yourself putting effort into the community forum.
Anand
Brilliant explanation and very well-done videos.
Thanks for your comment, appreciated.
Very good, thank you
You’re welcome.