Cluster Networking
This is a sample of the event you will see in the System Event Log in Event Viewer:
One specific problem that I have seen a few times lately is with the VMXNET3 adapters dropping inbound network packets because the inbound buffer is set too low to handle large amounts of traffic. We can easily find out if this is a problem by using Performance Monitor to look at the “Network Interface\Packets Received Discarded” counter.
Once you have added this counter, look at the Average, Minimum and Maximum numbers and if they are any value higher than zero, then the receive buffer needs to be adjusted up for the adapter. This problem is documented in VMWare’s Knowledge Base:
Large packet loss at the guest OS level on the VMXNET3 vNIC in ESXi 5.x / 4.x
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2039495
Resolution
To resolve this issue, ensure that there is no traffic filtering occurring (for example, with a mail filter). After eliminating this possibility, slowly increase the number of buffers in the guest operating system.To reduce burst traffic drops in Windows Buffer Settings:
- Click Start > Control Panel > Device Manager.
- Right-click vmxnet3 and click Properties.
- Click the Advanced tab.
- Click Small Rx Buffers and increase the value. The default value is 512 and the maximum is 8192.
- Click Rx Ring #1 Size and increase the value. The default value is 1024 and the maximum is 4096.
Notes:
- These changes will happen on the fly, so no reboot is required. However, any application sensitive to TCP session disruption can likely fail and have to be restarted. This applies to RDP, so it is better to do this work in a console window.
- This issue is seen in the Windows guest operating system with a VMXNET3 vNIC. It can occur with versions besides 2008 R2.
- It is important to increase the value of Small Rx Buffers and Rx Ring #1 gradually to avoid drastically increasing the memory overhead on the host and possibly causing performance issues if resources are close to capacity.
- If this issue occurs on only 2-3 virtual machines, set the value of Small Rx Buffers and Rx Ring #1 to the maximum value. Monitor virtual machine performance to see if this resolves the issue.
- The Small Rx Buffers and Rx Ring #1 variables affect non-jumbo frame traffic only on the adapter.