Tuesday 3 December 2013

Network speed issues in Windows Server 2012 R2 Hyper-V with Broadcom NICs

It’s been well documented that there are various problems with Broadcom network drivers in implementations of Hyper-V.
  
Some of these examples are:
Microsoft KB2902166 – Recommendation to disable VMQ with Broadcom NICs - http://support.microsoft.com/kb/2902166
Guest Clustering Issues - http://www.hyper-v.nu/archives/pnoorderijk/2013/06/virtual-guest-cluster-and-nic-teaming-in-the-host-results-in-an-evicted-cluster-node-broadcom-emulex/
Guest Clustering Issues - http://systemscentre.blogspot.co.uk/2013/05/problems-clustering-virtual-machines-on.html
Updated Dell Driver for Broadcom NICs - http://datacenter-flo.de/?tag=broadcom
Various other posts that a simple Bing search will find you - http://www.bing.com/search?q=broadcom+hyper-v&qs=n&form=QBLH&filt=all&pq=broadcom+hyper-v&sc=3-16&sp=-1&sk=

I was hoping that with the release of Windows Server 2012 R2 that these might be a thing of the past and the fixes introduced in the latest 2012 RTM drivers carried across.
How wrong could I be…

After deploying a 2 node 2012 R2 Hyper-V cluster I started to immediately notice slow network performance both deploying new VM’s and copying files between guest virtual machines.
To further confuse me, the problems were heavily present when copying to the host, or VM’s running on the host, that wasn’t the CSV owner.  This originally started me looking down the wrong path.

So, after trying multiple things, I came full circle back round to retesting VMQ & Broadcom settings.

At the moment it looks like the problem that I (and others) had experienced in the past with having VMQ enabled on Broadcom adapters is present with the inbox driver in R2 (version 15.6.1.3).

As well as enabling/disabling VMQ I also stepped the driver down to the previous 2012 RTM version driver (15.6.0.10) and it works fine with VMQ enabled.
I can now even swap between drivers without a reboot and show speed impact.

With VMQ Enabled, poor transfer speed between VM’s:
 VMQEnabled

With VMQ Disabled, consistent (and better) transfer speeds regardless of VM/Node placement (Live Migration while copying):
 VMQDisabled

In my environment that I was testing, I have Broadcom NetXtreme  B5720 Quad Port NICs in my blades and all firmware is up to date

Obviously I don’t really want to miss out on the VMQ features so for a while I ran the down level driver, hoping that a fix would appear.

Well, Broadcom have recently released an updated driver directly to their site.
http://www.broadcom.com/support/ethernet_nic/netxtreme_server.php
image

This driver is dated 5th November 2013 and version 16.2.0.4

I’ve flattened my environment and let VMM install the updated driver during bare metal deployment and, touch wood, so far all VMQ related speed issues are fixed.

Looks like it’s something to bear in mind that the in-box Broadcom driver in R2 is broken while the current 16.2.0.4 direct from Broadcom works.

9 comments:

Anonymous said...

What a find! Thank you very much!!! We've been battling with inexplicably slow network speeds and the updated Broadcom driver has done the trick beautifully.

Anonymous said...

BRILLIANT - thanks, worked for us too!

Granger said...

This was exactly what we needed!

We had the same issue with a new Windows 2012 r2 cluster. Copying files from/to the VMs running on the cluster would top out at around 3MB/sec, but copying files from/to the host partition (on the node hosting the VM in question) was normal (~90-110MB/sec)---same physical NIC.

Updated the drivers and the VMs are now up to ~60-90MB/sec. I expect there's more I could do, but that's good enough for what I need now.

Thanks for publishing this!

nathan woodcock said...

Awesome stuff. I had this issue on a brand new Dell server with all current Dell/MS firmware and drivers. Both my Server 2012 R2 and SBS2011 hyper-v guests had atrocious network speeds accessing network shares. Updated the driver as suggested and shares now accessed at full speed. Thanks.

Brian Klish said...

I have latest NIC firmware and drivers, but I'm still having a problem unfortunately. Turned off VMQ and it works fine. Any thoughts on why it wouldn't work for me?

Anonymous said...

You're a genius. I deployed 2 Hyper-V Server 2012 r2 servers over the weekend on my HP Proliant Gen8 servers. I had made sure to use updated firmware and drivers from HP as well! Just woke up Monday morning to a list of complaints regarding network speed. Back to copying at 50MB/s. Thanks heaps!

Virtu Techs said...

Even after doing this my speed not even crossing 1 MBPS. Kindy adviceme what needs to be done

sathish@itspl.com

Anonymous said...

Nice work. Thank you very much!

Randy said...

This worked perfectly for me. I had the problematic drivers that came in 2012 R2. Swapped them out for the ones from the Broadcom site. Suddenly my file server was accessible again.