Tuesday, 14 May 2013

Problems Clustering Virtual Machines on Windows Server 2012 Hyper-V

I was re-building our lab environment at work the other week in preparation for our big Summit13 event, that and the lab had been trashed over the last year...

As part of the re-build I had decided to implement a couple of virtual machine clusters, one for a scale-out file server and one as a SQL cluster.

I'd deployed the virtual machines for the cluster nodes using Service Templates in SCVMM and as part of that template chosen to use an availability set to ensure the VM's were separated across hosts (a cluster doesn't provide much High Availability if they all reside on the same host that's failed!).

When I started to create the cluster I ran straight into a problem with the Failover Cluster Manager reporting problems due to timeouts when creating the cluster.

Creating a single node cluster worked fine, but then would again fail when trying to add in another node.

I happened to put one of the Hyper-V hosts into maintenance mode for something and it migrated all the VM's onto the same host, at which point creating the cluster worked flawlessly, yay!

However, when the Hyper-V host came back out of maintenance mode and the availability sets kicked in during optimisation forcing a VM node back onto a separate physical host, the clusters broke again, not yay :(



So after some Googling Binging about and a shout on Twitter (Thanks @hvredevoort and @WorkingHardInIT) an issue with Broadcom NICs was brought to my attention and I came across this TechNet Forum post talking about the same issue.

Sophia_whx suggested trying to use Disable-NetAdapterChecksumOffload on the NICs to help with the issue.

So, first things first.  Use the Get-NetAdapterChecksumOffload to see just what the configuration was and sure enough Checksum Offload is enabled for just about all services across the majority of the NICs.

I then used the Disable-NetAdapterChecksumOffload * -TcpIPv4 command which resulted in this:

 
A reboot later and then perform it on the second host and whoa....
For some reason, the virtual switch really didn't like having that done to it.
I wish I had some screenshots, but I went into "get it fixed fast" mode.
 
Basically the switch via powershell was showing as up, the NIC Teaming GUI was showing it down and all the bound adapters as failed. SCVMM had lost all configuration for the switch altogether.
Deleting the switch from SCVMM didn't delete it from the host, but brought it back to life on the host but was missing in SCVMM.  SCVMM then wouldn't redetect it or let you build it again as it was still there, apparently???
 
I had to manually remove the team from a remote NIC Teaming GUI (I could of PowerShell'd it I know!) and then recreated via SCVMM.
 
Anyway... at first this looked to have fixed the clustering within virtual machine issues, but it only delayed the symptoms i.e. it took longer to evict nodes and randomly brought them back online.
 
So next I tried disabling Checksum Offload for all services, being careful not to touch the Virtual Switch this time.
 
Rather than going adapter by adapter I used the following command:
 
Get-NetAdapter | Where-Object {$_.Name -notlike "Converged*"} | Disable-NetAdapterChecksumOffload
 
This resulted in the Checksum Offload being disabled for the various services as shown, except for my virtual switch.


After doing this on the other host and giving them a reboot, my clustered virtual machines appear to be nice and stable when split across physical hosts. Yay! Problem fixed.

Just as another side note about Broadcom adapters, there have also been reports of performance issues when the Virtual Machine Queue (VMQ) setting is enabled, despite it being a recommended setting.

A quick check of my hosts showed it was enabled:


Another quick PowerShell line later and it wasn't:

Get-NetAdapterVmq -InterfaceDescription Broad* | Disable-NetAdapterVmq

9 comments:

Alex Smith said...

Activating McAfee is a quick and simple process. You just need to copy your 25 characters alphanumeric product key and paste it in the activation tab. Let’s check out the step-by-step process #mcafee #activate mcafee.com/activate | mcafee.com/activate | mcafee.com/activate | mcafee.com/activate

Alex Smith said...

Thank you so much for sharing such superb information's with us. Download the Microsoft Office setup on your Windows or Mac computer. Make sure that the Office product key is already copied for the activation procedure. office.com/setup mcafee.com/activate|office.com/setup

Anonymous said...


Visit www.mcafee.com/activate for download and activate your mcafee product, and if you have any issues you can directly call at mcafee toll free number.

mcafee activation key support
activation key for mcafee
mcafee activation
how to activate mcafee
mcafee product activation
how to activate mcafee
mcafee com setup activation
mcafee antivirus code
mcafee activation support
mcafee activation code
mcafee activation key

Anonymous said...

Visit Norton Setup website www.norton.com/setup or norton.com/setup. Click on Enter Product key, Enter Your Email and Password to proceed, Enter your 25 Key.

norton.com/setup

www.norton.com/setup

norton product key

setup norton

norton setup with product key

norton com setup

norton setup

norton install

install norton antivirus

norton product key activation

Anonymous said...

Thank you so much for sharing all this wonderful info with the how-to’s!!!! It is so appreciated!!!

Parks and Recreation: 5 Most Realistic Storylines
NASA Will Conduct Green Run Hot Fire Tests for Artemis – I Mission
Everything To Know About Android Skins
Take Advantage Of Stock Android Without Rooting
Surprising Facts About Lord Voldemort Which Only Book Fans Know

kati martin said...

Ik vond dit een informatieve blog en ook erg nuttig en deskundig. Bellen google

ashleyleio123 said...

Zeer informatieve blog. bedankt voor het delen met ons.Bellen Outlook

naina robert said...

Positieve webpagina-bron, waar heb je de informatie over dit bericht gevonden? Bedankt voor het delen. telefoon bellen Instagram

Kala Jadu Specialist said...

Powerful Black Magic Specialist is there you have a difficult you basically can't get past you can leave him in his grasp and he will deal with it. You take help of a particularly Powerful Black Magic Specialist you can trust as you are in the most secure hands on the planet. In increases, He has helped hundreds and thousands of individuals across the globe. What's more, is quite famous for his extraordinary astrology and black magic work commitments also.
He was conceived and raised in the family who had astrology and black magic work potential which is brimming with capacities in his blood. He has been rehearsing black magic from a youthful age. Also, likewise has built up an expertise for such works through his dad and progenitors. Subsequently, he is somebody whom you can trust with full confidence.