All,

As of yet I have no solution but am documenting this issue on MPS portfolio.

The issue is simple (or so we thought). We are seeing packet drops on vmnics that are administratively down in VMware.

The Infrastructure
Two HS23 (both exhibiting same issues)
Running ESXi 5.5 Update 2
Emulex 4-port One Connect Ethernet Adapter (driver version - 10.2.298.5)


The Issue
As you can see below. When I pull statistics from vmnic0 it is showing a large amount packets received and a large amount of packet drops. However, as mentioned above, this vmnic is administratively down as you will see below.

esxcli network nic stats get -n vmnic0
NIC statistics for vmnic0
   Packets received: 31547633303
   Packets sent: 0
   Bytes received: 16187874324373
   Bytes sent: 0
   Receive packets dropped: 1482863150
   Transmit packets dropped: 0
   Total receive errors: 0
   Receive length errors: 0
   Receive over errors: 0
   Receive CRC errors: 0
   Receive frame errors: 0
   Receive FIFO errors: 0
   Receive missed errors: 0
   Total transmit errors: 0
   Transmit aborted errors: 0
   Transmit carrier errors: 0
   Transmit FIFO errors: 0
   Transmit heartbeat errors: 0
   Transmit window errors: 0


As you can see, I have no vmnic0/1 in any vSwitch.

/usr/sbin/esxcfg-vswitch -l
Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         5632        9           128               1500    vmnic2,vmnic3

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  Production            0        2           vmnic2,vmnic3
  VMotion               0        1           vmnic2,vmnic3
  Management Network    0        1           vmnic2,vmnic3

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitchUSB0      5632        4           128               1500    vusb0

  PortGroup Name        VLAN ID  Used Ports  Uplinks
  IMM_Network0          0        1           vusb0

If I list the statistics of vmnic2 right after I see that packets received on this vmnic IS ALMOST THE SAME AS vmnic0!
esxcli network nic stats get -n vmnic2
NIC statistics for vmnic2
   Packets received: 31547645008
   Packets sent: 23805666719
   Bytes received: 16187878339822
   Bytes sent: 39966911268856
   Receive packets dropped: 0
   Transmit packets dropped: 0
   Total receive errors: 0
   Receive length errors: 0
   Receive over errors: 0
   Receive CRC errors: 0
   Receive frame errors: 0
   Receive FIFO errors: 0
   Receive missed errors: 0
   Total transmit errors: 0
   Transmit aborted errors: 0
   Transmit carrier errors: 0
   Transmit FIFO errors: 0
   Transmit heartbeat errors: 0
   Transmit window errors: 0

When I run the below command and use WireShark to review the .pcap file I see (as confirmed by VMware technical support) that the file shows packets that mirror what vmnic2 is receiving. Almost as if the vmnic ports are replicating their traffic across ports??

pktcap-uw --uplink vmnic0 ?o /vmfs/volumes/ESXFshare-01/vmnic0_pktcap.pcap


Currently this is unresolved. VMware has asked that I open a hardware ticket for my Emulex card to see if there is a hardware defect causing this "packet replication" issue. I will update when/if a resolution is reached.

Thanks!

Hello - Did you ever find out what the cause of this issue was? I am having almost the exact same thing happen to me with some HS23 systems running 5.5

Thanks

Hi Stephen,

Good morning, thanks for the question. No unfortunately we never got a solid answer on what this was happening. We believe it was a hardware/firmware issue with the Emulex adapter but neither IBM or VMware took credit for being the root cause of the problem. We began to ignore the issue since we know those VMnics are down.