Packet Drops on DOWN VMNIC
« on: September 16, 2015, 08:46:10 AM »
All,
As of yet I have no solution but am documenting this issue on MPS portfolio.
The issue is simple (or so we thought). We are seeing packet drops on vmnics that are administratively down in VMware.
The Infrastructure
Two HS23 (both exhibiting same issues)
Running ESXi 5.5 Update 2
Emulex 4-port One Connect Ethernet Adapter (driver version - 10.2.298.5)
The Issue
As you can see below. When I pull statistics from vmnic0 it is showing a large amount packets received and a large amount of packet drops. However, as mentioned above, this vmnic is administratively down as you will see below.
esxcli network nic stats get -n vmnic0
NIC statistics for vmnic0
Packets received: 31547633303
Packets sent: 0
Bytes received: 16187874324373
Bytes sent: 0
Receive packets dropped: 1482863150
Transmit packets dropped: 0
Total receive errors: 0
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 0
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0
As you can see, I have no vmnic0/1 in any vSwitch.
/usr/sbin/esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 5632 9 128 1500 vmnic2,vmnic3
PortGroup Name VLAN ID Used Ports Uplinks
Production 0 2 vmnic2,vmnic3
VMotion 0 1 vmnic2,vmnic3
Management Network 0 1 vmnic2,vmnic3
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitchUSB0 5632 4 128 1500 vusb0
PortGroup Name VLAN ID Used Ports Uplinks
IMM_Network0 0 1 vusb0
If I list the statistics of vmnic2 right after I see that packets received on this vmnic IS ALMOST THE SAME AS vmnic0!
esxcli network nic stats get -n vmnic2
NIC statistics for vmnic2
Packets received: 31547645008
Packets sent: 23805666719
Bytes received: 16187878339822
Bytes sent: 39966911268856
Receive packets dropped: 0
Transmit packets dropped: 0
Total receive errors: 0
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 0
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0
When I run the below command and use WireShark to review the .pcap file I see (as confirmed by VMware technical support) that the file shows packets that mirror what vmnic2 is receiving. Almost as if the vmnic ports are replicating their traffic across ports??
pktcap-uw --uplink vmnic0 ?o /vmfs/volumes/ESXFshare-01/vmnic0_pktcap.pcap
Currently this is unresolved. VMware has asked that I open a hardware ticket for my Emulex card to see if there is a hardware defect causing this "packet replication" issue. I will update when/if a resolution is reached.
Thanks!