Objective 2.2 – Determine Physical Infrastructure Requirements for a VMware NSX Implementation

Principles

  • Discern management and edge cluster requirements
  • Differentiate minimum/optimal physical infrastructure requirements for a VMware NSX

Implementation

  • Determine how traffic types are handled in a physical infrastructure
  • Determine use cases for available virtual architectures
  • Describe ESXi host vmnic requirements
  • Differentiate virtual to physical switch connection methods
  • Compare and contrast VMkernel networking scenarios

References

  • vmw-nsx-network-virtualization-design-guide.pdf

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/nsx/vmw-nsx-network-virtualization-design-guide.pdf

  • vsphere-esxi-vcenter-server-651-networking-guide.pdf

https://docs.vmware.com/en/VMware-vSphere/6.5/vsphere-esxi-vcenter-server-651-networking-guide.pdf

  • nsx_62_install.pdf

https://pubs.vmware.com/NSX-62/topic/com.vmware.ICbase/PDF/nsx_62_install.pdf

Discern management and edge cluster requirements

  • Cluster requirements depend on the number of hosts in a deployment:
    • Small
    • Medium
    • Large
  • NSX Controllers uually go on same cluster as NSX Manager:
    • On Management cluster for Single vCenter deployments
    • On Edge cluster for Multi-vCenter deployments where separate vCenter is used to manage Compute/Edge resources

Small

  • 3 – 10 Hosts
  • Low growth expected

  • Single Cluster with all management and compute components co-located
  • vSphere standard edition license
  • Low bandwidth requirement (10Gbps), therefore ECMP not required
  • Use Resource Reservations for various components

Medium

  • 10 – 100 Hosts
  • Designed to be grown over time

  • 2+ clusters: Management/Edge + Compute
    • Management/Edge cluster can be separated if needed
  • Standard vSphere license
  • Bandwidth:
    • Initially 10Gbps,therefore ECMP not required
    • Switch to ECMP is > 10Gbps
  • Ensure DLR control VM not placed on same host as an ESG

Large

  • 100+ Hosts
  • Separate Compute, Management and Edge Clusters
  • Multiple vCenter
  • Multi-Site deployments => vSphere Enterprise License
  • >10 Gbps bandwidth => ECMP required
    • Max: 80 Gbps

Differentiate minimum/optimal physical infrastructure requirements for a VMware NSX Implementation

  • Cluster Design
    • Small/Medium
      • Minimum of 1 cluster (Small only)
      • Recommended 2: Management + Edge/Compute
    • Large
      • Minimum 3: Management, Edge, Compute
      • Recommended 4: Management, Edge, Compute 1 + Compute 2
  • Management Cluster
    • Minimum number of hosts: 2
    • Recommended minimum hosts for vSphere HA: 3
    • Recommended minimum hosts for vSAN: 4
  • Edge Cluster
    • Minimum number of hosts: 3
    • Recommended minimum for resilience: 4
  • Compute Cluster
    • Minimum 1
    • Recommended 2+ (for large deployments). Should be striped across two racks for resilience
  • Network
    • Uplinks
      • Small/Medium: Minimum 2, Recommended 4 (up to 8 for full ECMP)
      • Large: Minimum 4, Recommended 8 for full ECMP
  • ESXi Host Configuration: Host Boot Storage: Minimum 5GB
  • Edge Cluster Hosts:
    • High CPU clock speed
    • Enable VXLAN TSO and RSS

Determine how traffic types are handled in a physical infrastructure

Types of traffic generated in an NSX environment:

  • VXLAN
  • Management
  • vMotion
  • Storage

VXLAN Traffic

VXLAN traffic is generated as by hypervisors as UDP frames on the transport network on NSX enabled clusters only – typically this is the Compute and Edge clusters. VXLAN traffic emanates from VTEP endpoints on specific VMKernel ports assigned for that purpose.

VXLAN traffic is carried in VLAN on the uplink to the ToR switch, which usually provides the L3 gateway in the form of an SVI as shown below. In this case VLAN 103 is used to carry VXLAN traffic and is the “Transport Network” presented to participating host clusters. Note: the minimum recommended MTU for VXLAN traffic is 1600 bytes.

Management Traffic

Management traffic is handled through a VMkernel port on the each host. Typically the management kernel port is configured on a vSphere standard switch – even when a distributed switch is in place. The management network is shared with vCenter, NSX Manager Etc.

In a Spine/Leaf architecture VLANs do not extend beyond a single Leaf Switch. This means that the Management IP for ESXi hosts is constrained by the Leaf switch (and hence VLAN) to which it is connected. Ironic perhaps, considering that the VXLAN data plane extends the Layer 2 boundary across datacentres, but the Spine/Leaf architecture has the opposite effect unless inter-rack Layer 2 trunks are configured – thereby circumventing the Spine/Leaf architecture.

vMotion Traffic

vMotion traffic uses a dedicated VMKernel port on each host with its own IP Address. There is usually a dedicated VLAN for vMotion traffic.

  • Number of simultaneous vMotions on 10Gbps Interface = 8

Storage Traffic

Each host can have a dedicated VMkernel port configured for access to IP storage such as NAS or iSCSI. As with other services, in a Spine/Leaf architecture the VLAN and hence IP subnet assigned to each host is determined by the Leaf switch to which a host is connected.

Determine use cases for available virtual architectures

NSX programmatically creates, snapshots, deletes, and restores software-based virtual networks.

Main Use Cases:

  • Security
    • Zero trust security model i.e. all traffic is blocked by default and specific firewall policies must be created to permit flows
    • Granular firewall policies based on MAC, IP, Port, vCenter Objects, AD Groups (Identify Firewall)
    • 3rd party integration e.g. Palo Alto, Fortinet
    • Virtual Desktop security
  • Automation
    • REST API for Networking, Security and Services
    • Pre-Integrated with vRA
    • OpenStack Neutron plugin
  • Application Continuity
    • Extend networking and security up to 8 vCenters (vSphere 6.0)
    • Long distance vMotion with vSphere 6.0 and NSX
    • Cross-vCenter NSX for active-active data centers

Describe ESXi host vmnic requirements

  • Hosts must have at least 1 spare physical adapter (vmnic) available for NSX enabled DVS configuration.
  • vmnics carry trunked VLANs with a minimum recommended MTU of 1600 bytes
  • Support for 1Gbps, 10Gbps and 40Gbps network interface cards

Differentiate virtual to physical switch connection methods

The DVS dvUplink port group is used for uplink connectivity. The teaming option used for the VXLAN port group is specified when a cluster is NSX enabled. The following teaming options are supported.

Note: Route Based on Physical NIC Load (LBT) is not supported in NSX

Teaming and Failover Mode NSX Support Multi-VTEP

Support

 Uplink Behaviour

(2 x 10G ports)
Route Based on Originating Port ✓ ✓ Both Active
Route Based on Source MAC Hash ✓ ✓ Both Active
LACP (Discouraged) ✓ × Flow based
Route Based on IP Hash (Static EtherChannel) ✓ × Flow based
Explicit Failover Order ✓ × Only one link is active
Route Based on Physical NIC Load (LBT) × × ×
  • Teaming policy must be the same for all hosts on a VDS across all clusters e.g. all compute clusters on a given VDS should have the same teaming policy configured
  • If using LCAP or Static EtherChannel for VXLAN, that same policy must be used for all other port groups on that switch
    • Etherchannel configuration required on physical switches
    • Impacts all other port groups on a DVS
    • Discouraged because of this restriction
  • If <10Gps per ESXi host required, Explicit Failover can be used
    • Allocate 2 x vmnics specifically for VXLAN traffic
    • Other vmnics can be used for remaining traffic types

Teaming options shown below.

Recommended Teaming Option

  • “Route Based on Originating Port” is the recommended teaming mode for the Edge and Compute Clusters
  • LACP/Static EtherChannel discouraged because all port groups have to use the same option
    • This causes a problem when peering over a VPC which is not supported

Compare and contrast VMkernel networking scenarios

The diagram below shows single and multiple VTEP options.

  • Multi-VTEP where port aggregation is not used
  • Single-VTEP where port aggregation is in place because uplink is seen as a single logical link
  • Where port channels are not used:
    • A VTEP is associated with a single uplink port
    • To increase bandwidth utilisation, multiple VTEPs are provisioned
  • Number of provisioned VTEPS always matches number of vmnics available
    • Automatic when “SRCID” or “SRCMAC” selected for “VMKNic Teaming Policy” as shown below