Objective 4.1 – Configure Environment for Network Virtualisation

Principles

  • Comprehend physical infrastructure configuration for NSX Compute, Edge and Management
    clusters (MTU, Dynamic Routing for Edge, etc.)
  • Prepare a Greenfield vSphere Infrastructure for NSX Deployment
    • Configure Quality of Service (QoS)
    • Configure Link Aggregation Control Protocol (LACP)
  • Configure a Brownfield vSphere Infrastructure for NSX
  • Determine how IP address assignments work in VMware NSX
  • Determine minimum permissions required to perform an NSX deployment task in a vSphere
    implementation

References

  • vsphere-esxi-vcenter-server-651-networking-guide.pdf

https://docs.vmware.com/en/VMware-vSphere/6.5/vsphere-esxi-vcenter-server-651-networking-guide.pdf

  • NSX Administration Guide

http://pubs.vmware.com/NSX-62/topic/com.vmware.ICbase/PDF/nsx_62_admin.pdf

  • NSX Installation Guide

https://pubs.vmware.com/NSX-62/topic/com.vmware.ICbase/PDF/nsx_62_install.pdf

  • NSX Brownfield Deployment Guide

https://www.vmware.com/files/pdf/products/nsx/VMware-NSX-Brownfield-Design-and-Deployment-Guide.pdf

Comprehend physical infrastructure configuration for NSX Compute, Edge and Management clusters (MTU, Dynamic Routing for Edge, etc.)

Three types of rack may be used in a typical NSX deployment.

  1. Management
  2. Edge
  3. Compute

This example connects to a Leaf/Spine network whereby each rack connects to an Edge switch.

Network Infrastructure Requirements

The following pre-requisites must be met to deploy an NSX infrastructure:

  1. The MTU for Ethernet frames on the transport network must be set to 1600 or greater on the NSX Transport VLAN. (NSX introduces an extra header into Ethernet frames). This includes any transit links between Datacenters
  2. Enable L2 IGMP Snooping on the NSX Transport VLAN to allow local Multicast forwarding of VTEP traffic. An IGMP Querier should be configured on the Transport VLAN
  3. Assign a Transport Network subnet for VTEP configuration

Compute Requirements

NSX Manager NSX

Controller

 NSX Edge Guest Introspection Data Security
RAM 16GB (24GB for large env.) 4GB Compact: 512MB

Large: 1GB

Quad Large: 1GB

X-Large: 1GB

 4GB 512MB
Disk 60GB 20GB Compact: 512MB

Large: 512MB

Quad Large: 512MB

X-Large: 4.5GB (+4GB Swap)

 4GB 6GB per ESXi
CPU 4 (8 for large env.) 4 Compact: 1

Large: 2

Quad Large: 4

X-Large: 6

 2 1

“Large” environments are considered thus:

  • 100 hypervisors
  • 100 NSX Edges
  • 1,000 universal distributed firewall rules
  • 10,000 distributed firewall rules (non-universal)

Network Edge

Northbound traffic from the NSX environment traverses the NSX Edge Cluster. NSX supports OSPF and BGP routing. If using BGP it is preferable to place NSX into a different AS than the Core network. This will allow routes to be propagated directly into the Core from the NSX Edges. Using iBGP would require a full mesh of peers to each NSX Edge/DLR.

Prepare a Greenfield vSphere Infrastructure for NSX Deployment

The following pre-requisites must be met to deploy an NSX infrastructure:

  1. At least one Distributed virtual switch (dVS)
  2. Minimum MTU of 1572 for the transport network (1600 recommended)

Licencing:

Legacy: Per ESXi host

NSX for vSphere 6.2.x, 6.3.x and 6.4.x licensing

Feature Standard Advanced Enterprise
Hypervisors Supported
Platform
ESXi Yes Yes Yes
vCenter Yes Yes Yes
Cross vCenter Networking & Security No No Yes
Controller Architecture
NSX Controller Yes Yes Yes
Universal Controller for X-VC No No Yes
Optimized ARP Learning, BCAST suppression Yes Yes Yes
Switching
Encapsulation Format
VXLAN Yes Yes Yes
Replication Mode for VXLAN
Multicast Yes Yes Yes
Hybrid Yes Yes Yes
Unicast Yes Yes Yes
Overlay to VLAN bridging
SW Bridge (ESXi-based) Yes Yes Yes
Hardware VTEP (OVSDB) with L2 Bridging No No Yes
Universal Distributed Logical Switching (X-VC) No No Yes
Multiple VTEP Support Yes Yes Yes
Routing
Distributed Routing (IPv4 Only)
Distributed Routing – Static Yes Yes Yes
Distributed Routing – Dynamic Routing with BGP Yes Yes Yes
Distributed Routing – Dynamic Routing with OSPF Yes Yes Yes
Equal Cost Multi-Pathing with Distributed Routing Yes Yes Yes
Universal Distributed Logical Router (X-VC) No No Yes
Dynamic Routing without Control VM (Static Only) Yes Yes Yes
Active-standby Router Control VM Yes Yes Yes
Edge Routing (N-S)
Edge Routing Static – IPv4 Yes Yes Yes
Edge Routing Static – IPv6 Yes Yes Yes
Dynamic Routing with NSX Edge (BGP) IPv4 Yes Yes Yes
Dynamic Routing with NSX Edge (OSPFv2) IPv4 Yes Yes Yes
Equal Cost Multi-Pathing with NSX Edge Yes Yes Yes
Egress Routing Optimization in X-VC No No Yes
DHCP Relay Yes Yes Yes
Active-Standby NSX Edge Routing Yes Yes Yes
VLAN Trunk (sub-interface) support Yes Yes Yes
VXLAN Trunk (sub-interface) support Yes Yes Yes
Per Interface RPF check on NSX Edge Yes Yes Yes
Services
NAT Support for NSX Edge
NAT Support for NSX Edge Yes Yes Yes
Source NAT Yes Yes Yes
Destination NAT Yes Yes Yes
Stateless NAT
ALG Support for NAT Yes Yes Yes
DDI
DHCP Server Yes Yes Yes
DHCP Relay Yes Yes Yes
DNS Relay Yes Yes Yes
VPN
IPSEC VPN No No Yes
SSL VPN No No Yes
L2 VPN (L2 extension with SSL VPN) No No Yes
802.1Q Trunks over L2 VPN No No Yes
Security
Firewall – General
Single UI for Firewall Rule Enforcement – NS+ EW No Yes Yes
Spoofguard No Yes Yes
Firewall Logging Yes Yes Yes
Rule Export No Yes Yes
Auto-save & Rollback of Firewall rules No Yes Yes
Granular Sections of Firewall rule table No Yes Yes
Distributed Firewall
DFW – L2, L3 Rules No Yes Yes
DFW – vCenter Object Based Rules No Yes Yes
Identity Firewall Rules (AD Integration) No Yes Yes
IPFix Support for DFW No Yes Yes
Context-based control of FW enforcement No Yes Yes
(applied to objects)
DFW – L7** No No Yes
Edge Firewall
Edge Firewall Yes Yes Yes
Edge High-Availability Yes Yes Yes
Service Composer
Security Policy Yes Yes Yes
Security Tags Yes Yes Yes
vCenter Object based security groups Yes Yes Yes
IPSet, MACset based security groups Yes Yes Yes
Data Security
Scan Guest VMs for Sensitive Data No Yes Yes
Third Party Integration
Endpoint Service Insertion – Guest Introspection Yes Yes Yes
Network Service Insertion No Yes Yes
Public API based Integration Yes Yes Yes
Load-Balancing
Edge Load-Balancing
Protocols
TCP (L4 – L7) No Yes Yes
UDP No Yes Yes
FTP No Yes Yes
HTTP No Yes Yes
HTTPS (Pass-through) No Yes Yes
HTTPS (SSL Termination) No Yes Yes
LB Methods No Yes Yes
Round Robin No Yes Yes
Src IP Hash No Yes Yes
Least Connection No Yes Yes
URI, URL, HTTP (L7 engine) No Yes Yes
vCenter Context-aware LB No Yes Yes
L7 Application Rules No Yes Yes
Health Checks
TCP No Yes Yes
ICMP No Yes Yes
UDP No Yes Yes
HTTP No Yes Yes
HTTPS No Yes Yes
Connection Throttling No Yes Yes
High-Availability No Yes Yes
Monitoring
View VIP/Pool/Server Objects No Yes Yes
View VIP/Pool/Server Stats No Yes Yes
Global Stats VIP Sessions No Yes Yes
Distributed Load-Balancing
L4 Load-balancing No No Yes (tech-preview)
Health checks No No Yes (tech-preview)
Operations
Tools
Tunnel Health Monitoring No No No
TraceFlow Yes Yes Yes
Port-Connections Tool No No No
Server Activity Monitoring No Yes Yes
Flow Monitoring No Yes Yes
IPFix (VDS Feature) Yes Yes Yes
Endpoint Monitoring No No Yes
Application Rule Manager No Yes Yes
VMware Tools
vR Operations Manager Yes Yes Yes
vR Log Insight Yes Yes Yes
Cloud Management Platform
vRealize Automation
Logical Switch Creation Yes Yes Yes
Distributed router creation Yes Yes Yes
Distributed firewall security consumption No Yes Yes
Load-balancing consumption No Yes Yes
App Isolation No Yes Yes
VMware Integrated OpenStack (Neutron Plugin)
VLAN Provider Networks Yes Yes Yes
Overlay Provider Networks Yes Yes Yes
Overlay Tenant Networks Yes Yes Yes
Metadata Proxy Service Yes Yes Yes
DHCP Server Yes Yes Yes
Neutron Router – Centralized – Shared Yes Yes Yes
Neutron Router – Centralized – Exclusive Yes Yes Yes
Neutron Router – Distributed Yes Yes Yes
Static Routes on Neutron Router Yes Yes Yes
Floating IP Support Yes Yes Yes
No-NAT Neutron Routers Yes Yes Yes
Neutron Security Groups using Stateful Firewall No Yes Yes
Port Security Yes Yes Yes
Neutron L2 Gateway Yes Yes Yes
Load Balancing (LBaaS) Yes Yes Yes
Admin Utility ( Consistency Check, Cleanup) Yes Yes Yes
Cross VC Logical Networking and Security No No No

Bottom of Form

Configure Quality of Service (QoS)

QoS is supported from vSphere Distributed Switch 5.5.

NSX supports QoS through the vSphere Distributed Switch. The ESXi Host represents the trusted boundary for Cos/QoS markings. Traffic can be marked and filtered at either the Distributed/Uplink Port Group or individual Port level.

  • Configure Traffic filtering and marking on a Distributed Port Group for VM Guest Tagging i.e. the VM applies VLAN Tags
  • Configure Traffic filtering and marking on an Uplink Port Group for Virtual Switch Tagging i.e. the Distributed Switch applies VLAN Tags at the Port Group level

Traffic can be marked either at L2 or L3:

  • Layer 2 Class of Service (CoS) markings in the CoS field of the 802.1q header (3 bits: 0-7)
  • Layer 3 Differentiated Services Code Point (DSCP or DiffServ) in the DSCP field of the IP header (6 bits: 0-63)

The first three bits of the DiffServ header usually match the CoS values from the L2 header. This provides inter-operability with L2 CoS mappings.

Distributed and Uplink Port Group

To enable Traffic Filtering and Marking on a Distributed Port or Uplink Port Group, select the appropriate object in vSphere web client, click “Policies -> Edit -> Traffic filtering and marking” and change the Status to “Enabled” as shown below.

Figure -Enable Traffic filtering and marking on a Distributed Port Group

Select the appropriate Action to be performed on traffic flowing through this Port Group: Tag, Allow or Drop. Tagging allows the CoS or DSCP value to be set based on traffic qualifier rules.

The direction of traffic determines where a rule is applied – Ingress, Egress or both.

Traffic Qualifier rules allows for traffic to be classified based on IP, MAC or other system qualifier.

The following Policy contains a single rule to simply set a CoS value of 5 to all TCP traffic flowing through the Port Group. Additional rules can be added to complete a policy for the port group in question.

Individual Distributed and Uplink Ports

In order to configure a specific Traffic filtering and marking policy on a Distributed or Uplink Port, the Port Group must first be configured to allow policy override. Edit the Port Group settings and click Advanced -> Over port policies -> Traffic filtering and marking = Allowed

Figure – Port Group Level QoS Policy Override

The next step is to identify the specific distributed or uplink Port to configure it to override the Traffic filtering and marking policy. In the following example Port 37 of Port Group has it’s QoS policy configured differently to that of the Port Group. This requires the “Override” checkbox is selected. The Policy in this instance sets DSCP = 60 for specific MAC addresses, unlike the parent policy that sets CoS=5 for all TCP traffic flowing through the Port Group.

Configure Link Aggregation Control Protocol (LACP)

LACP support was introduced in vSphere 5.1 and later enhanced in vSphere 5.5. See https://kb.vmware.com/kb/2051316 for details.

Feature vSphere 4.x/5.0 vSphere 5.1 vSphere 5.5
LACP Support No Yes Yes
Multiple LAGs per uplink port group N/A No Yes
Multiple LACP load balancing algorithms N/A No Yes

Enhanced LACP support was introduced in vSphere 5.5 which supports up to 64 LAGs per Host or Distributed Switch. Likewise, a Distributed Switch supports up to 64 LAGs. The maximum number of ports allocated to a LAG is determined by the physical switch to which the ESXi host is connected. E.g. Cisco Nexus 9000 switches support up to 32 ports in a LAG. In the diagram below two LAGs configured on the Distributed Switch with 4 ports each. LAG1 is allocated vmnic0 and 1 on each host and LAG2 is allocated vmnic2 and 3 on each host.

Although the hosts in this example are connected to a single switch, they could just as easily be connected to separate switches for resilience i.e. Host 1 could have LAG1 on Switch A and LAG2 Switch B. Likewise Host 2 could have LAG1 on B and LAG2 on Switch A.

LAG Configuration

Teaming and Failover

  • Only one LAG can be active at a time
  • Multiple LAGs in an Active/Active or Active/Standby configuration are not supported
  • The standby port list must always be empty

In the example below a single LAG is configured for a Distributed Port Group with two vmnics. The DVUplinks have no assigned interfaces.

LAG Configuration

There are four steps to LAG configuration:

  1. Create LAG on the Distributed Switch
  2. Set LAG as Standby on Distributed Port Groups
  3. Assign Physical Ports to the LAG
  4. Set LAG as the Active uplink

Configure a Brownfield vSphere Infrastructure for NSX

Assuming traditional 3-layer datacentre layout:

  • Core Layer
  • Aggregation Layer
  • Access Layer

Main characteristics:

  • L2/3 Boundary at aggregation layer. FW = gateway for workload VLANs
  • Access Layer = Pure L2

Two migration use cases:

  1. Micro-Segmentation without overlays
  2. Micro-Segmentation with overlays

Micro-Segmentation without overlays

In a traditional datacentre layout, the Firewall provides routing and centralised security policies to VLANs. Note: there is not E-W security policy applied i.e. traffic on the same VLAN is not subject to Firewall Policies.

NSX Distributed Firewall provides E-W security policies with the option of adopting a zero-trust model i.e. all traffic between VMs on the same network is blocked by default without the need to change any of the existing network infrastructure.

Pre-Migration Considerations

  1. NSX requires a vDS to be deployed
  2. Hosts connected to Nexus 1000v need to be migrated to the vDS
  3. DFW does not require NSX Controllers – just NSX Manager and NSX VIBs to be installed on the managed clusters (Host Preparation)
  4. Adding new hosts to an existing prepared cluster automatically installs VIBs
  5. Add a new cluster requires it to undergo “Host Preparation” from the NSX vCenter plugin

Migration Goals:

  1. Adopt NSX DFW to E-W Security Policies aka “micro-segmentation”
  2. Move VM gateway from physical FW to Aggregation Layer switches (L3 switches)
  3. Offer several options for N-S Security Policy enforcement

Detailed Migration Procedure

  1. Verify all hosts are connected to a VDS
  2. Deploy NSX Manager
  3. Perform Host Preparation (NSX VIB Installation)

DFW Policy enforcement is applied as soon as the VIBs are installed. Default Rule = Permit

  1. Deploy DFW policies. 3rd party applications can be used to collect existing firewall policies from physical firewalls and apply directly to the DFW – typically on a per application basis
  2. At this stage security policies are applied both at the DFW and physical FW level
  3. Traffic not permitted is dropped at the hypervisor level thereby reducing the amount of traffic sent to the physical firewall

  1. Once DFW policies have been verified, the physical FW policies can be disabled. At this stage the DFW applies all E-W policies and the physical FW the N-S policies only.

  1. Configure SVIs on Aggregation Switches to replace the physical FW Gateway

Note: Interfaces are kept in a shutdown stage until cutover

The Gateway is in the process of being moved from the physical FW to the Aggregation Switches

  1. Switch Default Gateway from physical firewall to aggregation switches. This is a disruptive operation that should be conducted during a maintenance window.

In-Line Physical Firewall for North-South Traffic

In this scenario the physical firewall is connected north of the aggregation switch towards the external networks and is simple to adopt. Routing between the physical firewall and external network is most likely already in place so the only change required is to establish routing with the aggregation switches.

DFW for North-South Traffic

DFW replaces the N-S security policies

(RJ: not sure about this as traffic is managed by the DFW on a first match basis)

Will most likely still require another physical firewall.

Micro-Segmentation with Overlays

  1. Introduces VXLANs
  2. De-couples logical from physical connectivity
  3. May require connect to bare metal servers that cannot be virtualised (P-V communication)

Pre-Deployment Considerations

  1. NSX requires a vDS to be deployed
  2. Hosts connected to Nexus 1000v need to be migrated to the vDS
  3. Hosts need to have VIBs installed
  4. Edge Cluster: separate dedicated cluster recommended for NSX Edge devices
  5. NSX Controllers can be deployed either on the management cluster (recommended) or the Edge cluster

Migration Goals

  1. Migrate workload VMs to VXLANs
  2. Relieve physical firewall from routing and security policy duties
  3. Preserver L2 communication between VXLANs and bare-metal servers
  4. Simplify physical network configuration by removing VLANs migrated to VXLANs
  5. Deploy Transport VLAN for VXLAN overlay traffic
  6. Deploy uplink VLANs to connect ESGs to physical network

Detailed Migration Procedure

  1. Verify host clusters are on a VDS
  2. Deploy NSX Manager and connect to vCenter (NSX-M:vCenter = 1:1)
  3. Deploy NSX Controllers on Management Cluster or optionally on the Edge Cluster
  4. Perform Host Preparation (VIB Installation)
  5. Deploy DFW Policies (physical policies remain in place for the time being)
  6. Optionally remove E-W policies from physical firewall once DFW policies have been tested

  1. Configure VXLANs and Transport VLAN. The latter must span all Compute and Edge Clusters

  1. Increase physical network MTU (min 1600)
    1. Host -> Switch Interfaces
    2. Transport VLAN SVIs

  1. Configure required VXLANs
  2. Deploy L2 Bridges (on either Compute or Edge clusters)
    1. Each L2 Bridge supports up-to 512 VXLAN-VLAN Pairs

  1. Migrate VMS to VXLANs by changing Port Groups as necessary
    1. Default gateway remains on the physical firewall
    2. Inter-subnet routing still performed by physical firewall

  1. Deploy DLRs and build out NSX Edge network
  2. Migrate VMs from L2 Bridge to DLR and decommission physical firewall VLAN for the migrated VXLAN

  1. Remove physical firewall and deploy an NSX Edge in Active/Standby mode if physical firewall is to be removed i.e. no ECMP

  1. Clean-up: Remove physical firewall, unused VLANs, Port Groups etc.

Deploying NSX on a Greenfield Network and Migration from Brownfield

In this scenario a completely new deployment of NSX takes place on new hardware separate from the existing brownfield deployment. Subsequently workloads must be migrated from the old to new environment.

Deployment Considerations

  1. Assumes that NSX has been deployed in line with recommended best practice with dedicated Management, Compute and Edge clusters
  2. No dependency of hardware/software versions deployed at brownfield site
  3. A L2 Bridge is deployed on the brownfield site that is managed from the greenfield vCenter thereby allowing VLAN-VXLAN communication without VLAN cross DC VLAN extension.

  1. Alternatively the L2 Bridging function can optionally be provided by a Hardware VTEP (required NSX 6.2)

Migration Goals

  1. Migrate application to greenfield deployment
  2. Provide network security policies for migration applications
  3. Maintain connectivity to bare metal servers that remain in brownfield deployment
  4. Ensure optimal access to migrated applications avoiding hair-pinning traffic back to the brownfield site

Detailed Migration Procedure

The initial assumption before starting the migration procedure is that NSX with all its functional components has
been deployed in the greenfield network. This implies also the creation of logical switches that will host the
migrated VMs and of the logical routing components (DLR and NSX Edges) for establishing local communication
to the external network. The logical switches can also be already connected to the DLR instance, but the DLR
interfaces should be initially kept in disabled state, as shown in Figure 24

  1. Deploy greenfield NSX and fully configure with DLR and NSX Edges

  1. Deploy L2 Bridges on brownfield site and manage from greenfield vCenter

  1. Configure E-W firewall policies on greenfield DFW for workloads to be migrated
  2. Configure N-S firewall rules on greenfield NSX Edges
  3. Migrate VMS:
    1. Shutdown on brownfield
    2. Copy VM folder to greenfield datacentre and import to vCenter

  1. Physical Firewall on brownfield site continues to act as gateway for workloads and provide N-S security policies
  2. DFW enforces E-W policies for migrated workloads
  3. Workloads yet to be migrated reach migrated workloads over the L2 bridge via VXLAN
  4. Move gateway for migrated workload VLANs to the greenfield NSX DLR (disruptive)

  1. Complete migration and decommission L2 bridges on brownfield site except those required for communication to bare metal servers

Determine how IP address assignments work in VMware NSX

IP Addresses for Host preparation (VTEP) may be assigned either from an IP Pool or DHCP.

In the case where clusters are “striped” across racks e.g. 2 hosts per rack and all inter-rack connectivity is routed e.g. Spine/Leaf, then the recommended way to assign VTEPs is through DHCP. This is because the IP Allocation method is defined at the Cluster level in NSX. In using the DHCP option, each host in the cluster will get an IP from the DHCP server configured for that rack (typically through the use of an IP helper address on the top of rack switches). An alternative approach is to allow DHCP to timeout and subsequently manually apply the VTEP IPs through the vSphere Web Client or some scripting approach e.g. powershell.

For the more common topology where the ToR switch operates at L2 only, then a single transport VLAN can be presented to all racks, thereby allowing all hosts within a cluster to operate of the same subnet. In this case IP Address assignment may be done through either DHCP or IP Pool

Determine minimum permissions required to perform an NSX deployment task in a vSphere implementation

User Roles:

  1. Enterprise Administrator: NSX Operations and Security
  2. NSX Administrator: NSX operations (NSX operations only)
  3. Security Administrator: NSX security only e.g. Security Policies, Port Groups
  4. Audior: Read-only

A vCenter use must have at least Security Administrator privileges in NSX to perform deployments.