VMWare VM shows duplicate IP for same host after Migrating (VMotion) the VM to another ESXi Storage Host

In the following example I ran into an issue whereby I was attempting to:

Migrate VMware Virtual Machine (VM) 2012_DC01.BINTRI.AD from ESXi Host-A to ESXi Host-B

The Migration was successful.

However after the VMotion completed, when I attempted to ‘ping’ my VM host IP Address it showed that there was a Duplicate IP address!

# ping -I ens192 172.16.241.187
PING 172.16.241.187 (172.16.241.187) from 172.16.242.196 ens192: 56(84) bytes of data.
64 bytes from 172.16.241.187: icmp_seq=1 ttl=128 time=0.459 ms
64 bytes from 172.16.241.187: icmp_seq=1 ttl=128 time=0.460 ms (DUP!)
64 bytes from 172.16.241.187: icmp_seq=2 ttl=128 time=0.529 ms
64 bytes from 172.16.241.187: icmp_seq=2 ttl=128 time=0.529 ms (DUP!)

I knew my client (2012_DC01.BINTRI.AD) only had a single ethernet interface.

Initially thinking someone grabbed my IP address on the network, I ran the following to display the MAC address

# arping -c 1 -I ens192 172.16.241.187
ARPING 172.16.241.187 from 172.16.242.51 ens192
Unicast reply from 172.16.241.187 [00:50:56:AF:7A:19]  1.390ms
Unicast reply from 172.16.241.187 [00:50:56:AF:7A:19]  1.432ms
Sent 1 probes (1 broadcast(s))
Received 2 response(s)

This told me that there was the SAME MAC address (00:50:56:AF:7A:19) responding to IP 172.16.241.187

Because I wanted to be doubly sure, I used nmap to tell me the OS type/etc and I was able to verify it is a Windows Server 2012 Client MAC Address (00:50:56:AF:7A:19)

# nmap -FO --system-dns 172.16.241.187

Starting Nmap 6.40 ( http://nmap.org ) at 2022-01-24 16:02 PST
Nmap scan report for dc01.bintri.ad (172.16.241.187)
Host is up (0.00033s latency).
Not shown: 95 closed ports
PORT      STATE SERVICE
53/tcp    open  domain
80/tcp    open  http
139/tcp   open  netbios-ssn
445/tcp   open  microsoft-ds
49152/tcp open  unknown
MAC Address: 00:50:56:AF:7A:19 (VMware)
Device type: general purpose
Running: Microsoft Windows 7|2012
OS CPE: cpe:/o:microsoft:windows_7:::ultimate cpe:/o:microsoft:windows_2012
OS details: Microsoft Windows 7 or Windows Server 2012
Network Distance: 1 hop

OS detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 4.47 seconds

Because I knew MAC Address (00:50:56:AF:7A:19) belonged to (2012_DC01.BINTRI.AD) there was ‘something’ in ESXi that believed there were 2 instances running!

I powered off VM 2012_DC01.BINTRI.AD using vCenter.

Yet I was still able to ping the IP address.

I went to the source ESXi Host-A and verified that it did not exist.

This told me that something was still resident in memory on either ESXi Host-A, or ESXi Host-B

I ssh’d into ESXi Host-A and ran the following command:

# esxcli vm process list

I looked for any processes open for host 2012_DC01.BINTRI.AD. I saw none.

I then ssh’d into ESXi Host-B and also ran

# esxcli vm process list

Except this time I saw in the output:

# esxcli vm process list
AD_2012_DC01.BINTRI.AD
   World ID: 136755
   Process ID: 0
   VMX Cartel ID: 136754
   UUID: 42 2f 8d 75 b6 84 51 78-90 5b 06 84 b2 e4 d5 22
   Display Name: AD_2012_DC01.BINTRI.AD
   Config File: /vmfs/volumes/d81de9d4-9c2bbc60/AD_2012_BENDC01.BINTRI.AD/AD_2012_BENDC01.BINTRI.AD.vmx

Because I powered off the VM, this told me that there was a process that needed to be stopped.

Therefore I used the ‘World ID’ and ran the following:

# esxcli vm process kill -t force -w=136755

Re-running esxcli vm process list now showed the process was gone.

I attempted to ping the IP address and this time it did not respond; as expected.

I then powered on the VM, and now when I ping, there is only a single IP Address

# ping 172.16.241.187
PING 172.16.241.187 (172.16.241.187) 56(84) bytes of data.
64 bytes from 172.16.241.187: icmp_seq=1 ttl=128 time=1.68 ms
64 bytes from 172.16.241.187: icmp_seq=2 ttl=128 time=0.348 ms
64 bytes from 172.16.241.187: icmp_seq=3 ttl=128 time=0.606 ms