Objective 10.4 – Troubleshoot common connectivity issues

Principles

  1. Review netcpa logs for control plane connectivity issues
  2. Verify VXLAN, VTEP, MAC, and ARP mapping tables
  3. List VNI configuration
  4. View VXLAN connection tables and statistics
  5. Perform VTEP connectivity tests

References

  1. NSX Administration Guide

http://pubs.vmware.com/NSX-62/topic/com.vmware.ICbase/PDF/nsx_62_admin.pdf

  1. NSX Command Line Interface Reference

http://pubs.vmware.com/NSX-62/topic/com.vmware.ICbase/PDF/nsx_62_cli.pdf

  1. vSphere Command Line Interface Concepts and Examples

https://vdc-download.vmware.com/vmwb-repository/dcr-public/0a40d9c5-4d4b-490d-8efa-e373a0ff2109/43a3c005-3878-4e05-8b60-35aca804d61d/vsphere-esxi-vcenter-server-65-command-line-interface-concepts-examples-guide.pdf

Review netcpa logs for control plane connectivity issues

netcpa

  • ESXi Control Plane Agent logs: /var/log/netcpa.log

Connection from ESXi to Controller made through netcpa

  • Message Bus Client/ VSFWD logs: /var/log/vsfwd.log

Capture activities of a firewall agent

  • VMkernel logs: /var/log/vmkernel.log
    NSX Controller Logs

    • cloudnet/cloudnet_java-vnet-controller.<start-time-stamp>.log

Manages configuration and internal API server

    • cloudnet/cloudnet.nsx-controller.log

Main controller process log

    • cloudnet/cloudnet_cpp.log.nsx-controller.log

Manages clustering and bootstrap

    • cloudnet/cloudnet_cpp.log.ERROR

Verify VXLAN, VTEP, MAC, and ARP mapping tables

VXLAN

Display VNI records:

show control-cluster logical-switches vni-table

VNI Controller BUM-Replication ARP-Proxy Connections VTEPs Active

5000 192.168.1.104 Enabled Enabled 1 1 true

5001 192.168.1.104 Enabled Enabled 0 0 true

5002 192.168.1.104 Enabled Enabled 1 1 true

5003 192.168.1.104 Enabled Enabled 0 0 false

VTEP

VTEP records of the specified VNI

show control-cluster logical-switches vtep-table vni

show control-cluster logical-switches vtep-table 5000

VNI IP Segment MAC Connection-ID Is-Active Out-Of-Sync

5000 192.168.10.1 192.168.10.0 00:50:56:60:3a:4a 11 YES NO

MAC

Show MAC records of the specified VNI

show control‐cluster logical‐switches mac‐table vni

nsx-controller # show control-cluster logical-switches mac-table 5000

VNI MAC VTEP-IP Connection-ID

5000 00:50:56:9b:07:df 192.168.10.1 11

5000 00:50:56:9b:e8:c5 192.168.10.1 11

Show the MAC records updated from the specified connection (VTEP)

show control‐cluster logical‐switches mac‐records VTEP-ipAddress

show control‐cluster logical‐switches mac‐records 192.168.110.52
VNI MAC VTEP‐IP Connection‐ID
5000 00:50:56:8e:f5:8b 192.168.150.52 2
5000 00:50:56:8e:6a:04 192.168.150.52 2
5000 00:50:56:8e:9d:88 192.168.150.52 2

ARP

Show the ARP records for the specified VNI

show control‐cluster logical‐switches arp‐table vni

Show control‐cluster logical‐switches arp‐table 5000
VNI IP MAC Connection‐ID
5000 192.168.10.6 00:50:56:8e:f5:8b 2
5000 192.168.10.1 00:50:56:8e:6a:04 2
5000 192.168.10.2 00:50:56:8e:9d:88 2

Show the ARP records updated from the specified connection (VTEP)

show control‐cluster logical‐switches arp‐records VTEP-ipAddress

show control‐cluster logical‐switches arp‐records 192.168.110.52
VNI IP MAC Connection‐ID
5000 192.168.10.6 00:50:56:8e:f5:8b 2
5000 192.168.10.1 00:50:56:8e:6a:04 2
5000 192.168.10.2 00:50:56:8e:9d:88 2

List VNI configuration

Controller, configuration, and status of the specified VNI

show control-cluster logical-switches vni vni

show control-cluster logical-switches vni 5000

VNI Controller BUM-Replication ARP-Proxy Connections VTEPs Active

5000 192.168.1.104 Enabled Enabled 1 1 true

View VXLAN connection tables and statistics

Connections joined to the specified VNI

show control-cluster logical-switches connection-table vni

show control-cluster logical-switches connection-table 5000

Host-IP Port ID

192.168.1.101 41033 11

VTEP records updated from the specified connection (VTEP)

show control-cluster logical-switches vtep-records <vtep_ip>

show control‐cluster logical‐switches vtep‐records 192.168.110.52
VNI IP Segment MAC Connection‐ID
5000 192.168.150.52 192.168.150.0 00:50:56:60:1e:dd 2

Perform VTEP connectivity tests

ESXi:

vmkping ++netstack=vxlan -d -s <packet size> <vmknic IP>

Set Packet Size => 1572

[root@esx01:~] vmkping ++netstack=vxlan -d -s 1572 192.168.10.1

PING 192.168.10.1 (192.168.10.1): 1572 data bytes

1580 bytes from 192.168.10.1: icmp_seq=0 ttl=64 time=0.580 ms

1580 bytes from 192.168.10.1: icmp_seq=1 ttl=64 time=0.135 ms

1580 bytes from 192.168.10.1: icmp_seq=2 ttl=64 time=0.378 ms

— 192.168.10.1 ping statistics —

3 packets transmitted, 3 packets received, 0% packet loss

round-trip min/avg/max = 0.135/0.364/0.580 ms

esxcli network diag ping –netstack=vxlan –host <vmknic IP> –df –size=<packet size>

Set Packet Size >= 1572

Trace:

Received Bytes: 1580

Host: 192.168.10.1

ICMP Seq: 0

TTL: 64

Round-trip Time: 43 us

Dup: false

Detail:

Received Bytes: 1580

Host: 192.168.10.1

ICMP Seq: 1

TTL: 64

Round-trip Time: 503 us

Dup: false

Detail:

Received Bytes: 1580

Host: 192.168.10.1

ICMP Seq: 2

TTL: 64

Round-trip Time: 87 us

Dup: false

Detail:

Summary:

Host Addr: 192.168.10.1

Transmitted: 3

Received: 3

Duplicated: 0

Packet Lost: 0

Round-trip Min: 43 us

Round-trip Avg: 211 us

Round-trip Max: 503 us

GUI: NSX -> Logical Switches -> Monitor -> Hosts-Ping