Wanted to make a quick post about enabling Jumbo Frames in my VMware environment! My switches (Cisco SG-200 & SG-500) support Jumbo Frames so I thought why not? It would surely help push more data through the network for faster for VSAN and vMotions. I’ll do some speed tests later to compare!
Hardware Requirements:
Before you start out on this quest ensure your physical switch support jumbo frames. When enabling it on my Cisco switches a switch reboot was required:
The next step is to ensure the NICs in your servers support jumbo frames. If they are server NICs then they should. My Dell T620/R520 both have Broadcom NetXtreme 5709 which in fact support Jumbo Frames.
Enable Jumbo Frames on Virtual Switch:
Now that you confirmed you can physically support Jumbo Frames your ready to enable it in vSphere. If you are using Distributed Switches right click the vDS then Edit Settings:
On the Properties tab click Advanced. In the Maximum MTU box change it from 1500 to 9000 then click Ok:
For Standard Switches: Go to vSphere Standard Switch -> Properties -> with vSwitch selected click Edit -> then change MTU to 9000:
Source: VMware KB 1038828 – Enabled Jumbo Frames on virtual distributed switches
Enable Jumbo Frames on VMKernel:
Now that the switch supports up Jumbo Frames let’s enable it on a vmk virtual adapter. Go to one of your hosts -> Configuration -> Networking -> vSphere Distributed Switch -> Manage Virtual Adapters -> Select a vmk -> Edit:
Check if Jumbo Frames are working:
If you have the VSAN health check enabled it does a health check on MTU size:
If the computer your on has Jumbo Frames enabled you can also perform a ping test. Open a command window and type:
|
1 2 |
ping -l 9000 <IP ADDRESS> |
Excellent guide. I specifically needed someone to tell me that the MTU 9000 setting needs to go on the virtual switch before that setting goes on the VMK interfaces. I have a switch that supports 10K jumbo frames natively, and I tried setting this up once before on my production vSAN, almost with disastrous consequences because whatever documentation I started from neglected to tell that you can hose your whole network by doing it wrong. Fortunately I was able to reverse the change and put everything back without serious issue.
I’m still going to check if the rest of my hardware supports jumbo frames before I try to turn it on. But knowing that my physical switch was not the cause of my problem, I’m hopeful that I can get a performance boost thanks to this tip.
Thank you for sharing. I have some feedback on the validation piece though. As provided this ping test will allow fragmentation of packets, allowing the ping to be broken into smaller frames for retransmission. This will allow the ping to succeed even if jumbo frames aren’t configured correctly on the network. Also there is ping packet overhead of 28 bytes meaning that your ping frame is 9028 not 9000. Lastly the ping command uses the management network context and doesn’t guarantee that the correct vmk port will be used. This is the correct test for Jumbo frames:
vmkping -d -s 8972 dest_ip
vmkping makes sure that the vmkernel interface is used and not the management network context. -d indicates “do not fragment” which will prevent false positives. 8972 subtracts the ping overhead resulting in a frame that is actually 9000 rather than 9028 (which is necessarily fragmented).
Misconfiguring jumbo frames can completely crash a network as the fragmentation overhead slowly fills buffers and consumes switch CPU cycles. Be very careful of false positive tests in a production environment.
Jason, when trying your test I do not get any ping responses. It is strange that if i use 8973 i get a sendto() failed (Message too long) but when I use 8972, it just sits there no responses of any kind.
Try 1472 and if it succeeds then you know the device will respond to pings and works fine at 1500 MTU. If you remove -d from your jumbo ping and it works but doesn’t work with -d then your network isn’t set up correctly someplace.
You may need to change the max MTU on your switches to 9216 to allow for COS and or VLAN tags.
Make sure the vmknic, vswitch, and port group all allow jumbo frames. Make sure your storage array is configured for jumbo frames. Make sure you aren’t doing any suboptimal pathing (network detours you’re not aware of)
The ping command to test jumbo frames is missing the -f argument to force the DO NOT FRAGMENT flag.