Every now and then a question comes up which I REALLY should know the answer to, but I don't. It's often one of those things that's so fundamental it's embarassing to ask.
For example, what's the largest MTU of an Ethernet packet? Or, how about when it's tunnelled through L2TP over a PPPoE connection?
Googling can help but there's no substitute for a bit of classroom experimentation so I recently did a little back-to-basics experiment with Wireshark to remind myself what I should already know. It resulted in a pretty cool demo of how packets on the Internet are built up so I thought I'd share it.
MTU
Networking 101: Data on an Ethernet network (and indeed the Internet) is sent by breaking it up into trillions of little packets. Each packet has a header which describes the packet, says where it's come from and where it's going to, and a payload, which is the actual data carried.
The Maximum Transmission Unit of a packet is the size of the data payload the packet can carry. This varies depending on the technology used in the network and what the packet is carrying.
Standard packets on an Ethernet network can be up to 1514 bytes. Ethernet itself uses a 14 byte header which leaves 1500 bytes for the data. This is the magic number I always remembered. But what's in that 1500 bytes?
Larger data is sent by breaking it up into multiple small packets, each packet needing its own set of headers. This fragmentation process is generally invisible to the user and works well for the most part, but optimising the MTU will minimise fragmentation and a poorly configured MTU can cause weird edge cases where things don't work in subtle ways. There's also the warm glow of fully understanding something and getting your configuration totally right.
Plain ICMP
Let's test this with a regular ICMP (ping) message. Ping clients let you specify the size of data payload so are perfect for this job. You run the client, tell it to generate packets that must not be fragmented, and increase the payload to the point the packets stop getting through.
On an Ethernet network, trial and error with ICMP shows the maximium ICMP payload is 1472 bytes. We know the packet can hold 1500 bytes so there must be 28 bytes of header.
Let's look at the packet with Wireshark. The Ethernet frame has 1514 bytes:
The Ethernet header is, as expected, 14 bytes:
After that is the first header. ICMP is an IP protocol so we see an IPv4 header. This is 20 bytes:
Within the IP packet we see the ICMP packet. This has an 8 byte header. After the header we can see the ICMP packet payload, which we can note is the English alphabet from a-w in this case.
Tunnel-tastic
With the basics proven, the real point of my investigation is to determine the MTU of a packet that takes a more complicated path: I'm setting up a backup Internet connection (more on that another time) where the network path involves:
PPPoE connection over an ADSL broadband connection from a cheap ISP
L2TP tunnel through the cheap ISP to my proper ISP
The tunnel means I can use my IP blocks from my 'proper ISP' even if my primary Internet connection dies, so various networky things I do (VPNs, access by ACL etc.) continue to work. Most ISPs would laugh at you if you ask about a backup L2TP tunnel which is why I am absolutely delighted to have found the excellent Andrews & Arnold who are as geeky as I am about all this stuff.
The help page for their L2TP connection makes a point of warning of low MTUs so I was wondering how low I needed to go.
The answer is: the largest ICMP payload I can get through the tunnel is 1426 bytes, and as we saw there are 28 bytes of header on an ICMP packet, so the MTU of the L2TP connection is 1454 bytes.
But why? Let's look:
Sorry about the mad redacting; these grabs include actual IP addresses. The un-redacted addresses are publically listed in the Andrews & Arnold knowledgebase.
Here's the Ethernet packet. 1514 bytes, so fully stuffed:
14 bytes of Ethernet Header:
This packet was sniffed from the interface pointing to the VDSL modem, so at this point the internet is encapsulated in a PPPoE session as is common to a lot of broadband Internet connections. This shows in Wireshark as two nested protocols.
There are 6 bytes of PPPoE:
And 2 bytes of PPP:
Now we're done with the ISP's essential wrapper for our packet, so what follows is 'ours' to play with. As mentioned, on this connection I'm using an L2TP tunnel. L2TP is a common (though a bit old) way of tunnelling data across the Internet. It runs in UDP packets over IP, so we expect to see in reverse order an IP header, a UDP header and finally the L2TP header. And, we do!
20 bytes of IP header:
8 bytes of UDP header:
Ah-ha! Now we're at the L2TP protocol. It has 6 bytes of header:
And within this are, this time, 4 bytes of PPP:
That is a lot of wrapping just to get packets to/from my primary ISP but it shows how all this stuff works. From this point in the packet, the remaining data payload is all we have to play with. It's the equivalent to our 1500 byte Ethernet frames, but in the case of L2TP is:
1514 - 14 (Ethernet) - 6 (PPPoE) - 2 (PPP) - 20 (IP) - 8 (UDP) - 6 (L2TP) - 4 (PPP) = 1454 bytes.
Obviously we need more headers around the data we're actually sending. This is a ICMP (ping) packet which operates over IP. So we have 20 bytes of IP header:
And finally we get to our ICMP packet with its 8 bytes of header and 1426 bytes of payload (yellow highlight):
And there we go. A payload of 1426 bytes (in an ICMP packet) is the most we can get, un-fragmented, through the backup tunnel.
Ethernet frame | 1514 bytes |
Ethernet header | 14 bytes |
PPPoE header | 6 bytes |
PPP header | 2 bytes |
IP (L2TP tunnel) header | 20 bytes |
UDP header | 8 bytes |
L2TP header | 6 bytes |
PPP header (again, for L2TP) | 4 bytes |
IP header (of our actual payload packet) | 20 bytes |
ICMP header (as our packet is ICMP) | 8 bytes |
Total | 88 bytes of header |
88 bytes of header out of a 1514 byte packet is only 5.8% of the packet's total size, which is really pretty small. And this is a fairly extreme example of wrapping packets up in tunnels.
Notes
If we were using a different protocol (e.g. TCP) the payload is further reduced. The operating system and network stack in your PC/phone/whatever is good at figuring out what the limit is through a process called Path MTU Discovery. By setting my L2TP tunnel to cap MTU at 1454, which I know will fit through the tunnel, the client device will figure this out and can send the largest possible payload in each packet. This is optimal.
L2TP can be used for VPN tunnels but here is used without encryption. In fact, traffic though ISPs is generally not encrypted - this little demo makes it clear how someone on the Internet eavesdropping on the packets can see all the data. Always use secure layers like TLS on top of all this!
With IPv6 the payloads are all smaller because the IP header is 40 bytes instead of 20. But one set of screenshots is enough for this post, which has turned out to be rather longer than I originally planned.
Comments