- Published on
NSX-T
- Authors
- Name
- Jackson Chen
VMware NSX-T Data Center Documentation
https://docs.vmware.com/en/VMware-NSX-T-Data-Center/index.html
It includes the references to the following resources, such as:
- NSX-T Data Center Migration Coordinator Guide
- NSX-T Data Center Reference Design Guide
- NSX-T Data Center Blog
- NSX-T Data Center Training and Demo videos
- NSX-T Data Center Load Balancing Encyclopedia
- NSX-T Data Center Multi-Location Design Guide
NSX-T
NSX-T Design Reference Guide NSX-T Design Reference Guide
Deploy NSX-T
Implementing NSX-T Data Center in vSphere steps
- Deploy an NSX Manager node from an OVF template
- Access the NSX UI
- Register vCenter Server with NSX Manager
- Deploy additional NSX Manager nodes to form an NSX management cluster
- Preconfigure transport nodes, including transport zones, IP pools, and unlink profiles
- Prepare hypervisor host transport nodes
- Deploy the NSX Edge nodes, and deploy edge cluster
Transport zone name can not have space, and should be using "-". Transport zone name could not be changed once configured.
NSX-T 3.1 Administration Guide
NSX-T 3.1 Administration Guide
NSX-T 3.1 Administration Guide
NSX-T 3.1 Installation Guide
NSX-T 3.1 Installation Guide
NSX-T 3.1 Upgrade Guide
NSX-T 3.1 Upgrade Guide
NSX-T 3.1 Administration Guide
NSX-T 3.0 Operational Guide
NSX-T Operations and Troubleshooting Guide
Key Concepts
NSX-T Data Center works by implementing three separate but integrated planes: management, control, and data. These planes are implemented as a set of processes, modules, and agents residing on two types of nodes
1. NSX Manager
2. Transport nodes
a. Every node hosts a management plane agent.
b. NSX Manager nodes host API services and the management plane cluster daemons.
c. NSX Controller nodes host the central control plane cluster daemons.
d. Transport nodes host local control plane daemons and forwarding engines.
NSX Manager supports a cluster with three node, which merges policy manager, management, and central control services on a cluster of nodes. NSX Manager clustering provides high availability of the user interface and API. The convergence of management and control plane nodes reduces the number of virtual appliances that must be deployed and managed by the NSX-T Data Center administrator.
Important: You must configure anti-affinity rules to prevent NSX Managers from being hosted at the same common host.
Temopary NSX Manager Nodes Usage
The normal production operating state is a three-node cluster of the NSX Manager (Local Manager in an NSX Federation environment) or Global Manager. However, you can add additional, temporary nodes to allow for IP address changes.
Key Concepts
Compute Manager
A compute manager is an application that manages resources such as hosts and VMs. One example is vCenter Server.
Management Plane
Provides single API entry point to the system, persists user configuration, handles user queries, and performs operational tasks on all of the management, control, and data plane nodes in the system. Management plane is also responsible for querying, modifying, and persisting user configuration.
Control Plane
Computes runtime state based on configuration from the management plane. Control plane disseminates topology information reported by the data plane elements, and pushes stateless configuration to forwarding engines.
Data Plane
Performs stateless forwarding or transformation of packets based on tables populated by the control plane. Data plane reports topology information to the control plane and maintains packet level statistics.
NSX Managed Virtual Distributed Switch or KVM Open vSwitch
The NSX managed virtual distributed switch (N-VDS, previously known as hostswitch) or OVS is used for shared NSX Edge and compute cluster.
N-VDS is required for overlay traffic configuration.
An N-VDS has two modes:
standard datapath
enhanced datapath.
* An enhanced datapath N-VDS has the performance capabilities to support NFV (Network Functions Virtualization) workloads.
* The N-VDS switch can be configured in the enhanced data path mode only on an ESXi host.
NSX Manager
Node that hosts the API services, the management plane, and the agent services. NSX Manager is an appliance included in the NSX-T Data Center installation package. You can deploy the appliance in the role of NSX Manager or nsx-cloud-service-manager.
Open vSwitch (OVS)
Open source software switch that acts as a virtual switch within XenServer, Xen, KVM, and other Linux-based hypervisors.
Overlay Logical Network
Logical network implemented using Layer 2-in-Layer 3 tunneling such that the topology seen by VMs is decoupled from that of the physical network.
Tier-0 Logical Router
A Tier-0 Logical Router provides north-south connectivity and connects to the physical routers. It can be configured as an active-active or active-standby cluster. The Tier-0 gateway runs BGP and peers with physical routers. In active-standby mode the gateway can also provide stateful services.
Tier-1 Logical Router
A Tier-1 logical router connects to one Tier-0 logical router for northbound connectivity to the subnetworks attached to it. It connects to one or more overlay networks for southbound connectivity to its subnetworks. A Tier-1 logical router can be configured as an active-standby cluster.
Transport Zone
Collection of transport nodes that defines the maximum span for logical switches. A transport zone represents a set of similarly provisioned hypervisors and the logical switches that connect VMs on those hypervisors. It also has been registered with the NSX-T Data Center management plane and has NSX-T Data Center modules installed. For a hypervisor host or NSX Edge to be part of the NSX-T Data Center overlay, it must be added to the NSX-T Data Center transport zone.
Transport Node
A fabric node is prepared as a transport node so that it becomes capable of participating in an NSX-T Data Center overlay or NSX-T Data Center VLAN networking. For a KVM host, you can preconfigure the N-VDS or you can have NSX Manager perform the configuration. For an ESXi host, NSX Manager always configures the N-VDS.
Uplink Profile
Defines policies for the links from hypervisor hosts to NSX-T Data Center logical switches or from NSX Edge nodes to top-of-rack switches. The settings defined by uplink profiles might include teaming policies, active/standby links, the transport VLAN ID, and the MTU setting. The transport VLAN set in the uplink profile tags overlay traffic only and the VLAN ID is used by the TEP endpoint.
Virtual Tunnel Endpoint
Each hypervisor has a Virtual Tunnel Endpoint (VTEP) responsible for encapsulating the VM traffic inside a VLAN header and routing the packet to a destination VTEP for further processing. Traffic can be routed to another VTEP on a different host or the NSX Edge gateway to access the physical network.
Install NSX Manager and Available Appliances
You can use the vSphere Client to deploy NSX Manager virtual appliances. The same OVF file can used to deploy three different types of appliances: NSX Manager, NSX Cloud Service Manager for NSX Cloud, and Global Manager for NSX Federation.
Cloud Service Manager (CSM) is a virtual appliance that uses NSX-T Data Center components and integrates them with your public cloud.
Important Logs
get cluster status get services get log-file login as root
(nsxcli) (nsxcli) (linux)
-----------------------------------------------------------------------------------------------------
DATASTORE datastore /var/log/corfu/corfu.9000.log
CLUSTER_BOOT_MANAGER cluster_manager /var/log/cbm/cbm.log
CONTROLLER controller /var/log/cloudnet/nsx-ccp.log
MANAGER manager manager.log /var/log/proton/nsxapi.log
POLICY policy policy.log /var/log/policy/policy.log
HTTPS http http.log /var/log/proxy/reserve-proxy.log
------------------------------------------------------------------------------------------------------
NSX audit log /var/log/nsx-audit.log
Overall log /var/log/syslog
NSX-T CLI References
https://www.simongreaves.co.uk/nsx-t-cli-reference-guide/
Reference Sites
Good NSX-T installation guide
https://docs.pivotal.io/tkgi/1-8/nsxt-3-0-install.html
Tier-0 Gateway and BGP
Configure logical routinng
Settingup Postman API
https://rutgerblom.com/2019/06/16/getting-started-with-the-nsx-t-api-and-postman/
Deploy NSX-T Using PowerShell
Deploy NSX-T using powerShell
https://www.virten.net/vmware/powershell-ovf-helper/vmware-nsx-t-3-1/
FileName: Deploy-VMware-NSX-T-3.1-ovf.ps1
$ovf = 'Z:\images\VMware\NSX-T\3.1\nsx-unified-appliance-3.1.0.0.0.17107171.ova' # Path to OVA File
$ovfConfig = Get-OvfConfiguration $ovf
$ovfConfig.DeploymentOption.Value = "small" # Deployment Size (extra_small | small| medium| large)
# ExtraSmall: 2 vCPU / 8GB RAM / 300GB Storage - This configuration is only supported for the nsx-cloud-service-manager role.
# Small: 4 vCPU / 16GB RAM / 300GB Storage - This configuration is supported for Global Manager Production deployment.
# Medium: 6 vCPU / 24GB RAM / 300GB Storage - This configuration is supported for Local Manager Production deployment.
# Large: 12 vCPU / 48GB RAM / 300GB Storage - This configuration is supported for Local Manager Production deployment.
$ovfConfig.NetworkMapping.Network_1.Value = "" # Destination network (Portgroup)
$ovfConfig.IpAssignment.IpProtocol.Value = "IPv4" # IP protocol for Management Network (IPv4|IPv6)
$ovfConfig.Common.nsx_passwd_0.Value = "" # System Root User Password
$ovfConfig.Common.nsx_cli_passwd_0.Value = "" # CLI "admin" User Password
$ovfConfig.Common.nsx_cli_audit_passwd_0.Value = "" # CLI "audit" User Password
# Minimum of 12 characters in length, complex password
# Default password complexity rules as enforced by the Linux PAM module.
# NOTE: Password strength validation will occur during VM boot.
# If the password does not meet the above criteria then login as root user for the change password prompt to appear.
$ovfConfig.Common.nsx_cli_username.Value = "admin" # CLI "admin" username (default: admin)
$ovfConfig.Common.nsx_cli_audit_username.Value = "audit" # CLI "audit" username (default: audit)
$ovfConfig.Common.nsx_hostname.Value = "" # The hostname for this VM.
$ovfConfig.Common.nsx_role.Value = "NSX Manager" # The role for this VM. (NSX Manager|nsx-cloud-service-manager|NSX Global Manager)
$ovfConfig.Common.nsx_ip_0.Value = "" # Management Network IPv4 Address
$ovfConfig.Common.nsx_netmask_0.Value = "" # Management Network Netmask
$ovfConfig.Common.nsx_gateway_0.Value = "" # Default IPv4 Gateway
$ovfConfig.Common.nsx_dns1_0.Value = "" # Space separated DNS server list for this VM.
$ovfConfig.Common.nsx_domain_0.Value = "" # Space separated domain search list for this VM.
$ovfConfig.Common.nsx_ntp_0.Value = "" # Space separated NTP server list for this VM.
$ovfConfig.Common.nsx_isSSHEnabled.Value = $true # Enable SSH ($true or $false)
$ovfConfig.Common.nsx_allowSSHRootLogin.Value = $true # Allow root SSH logins ($true or $false)
$ovfConfig.Common.nsx_swIntegrityCheck.Value = $false # Software Integrity Checker is required only for NDcPP 2.2
$VMName = "" # Virtual Machine Display Name
$vmhost = "" # ESXi Host to deploy the VM
$datastore = "" # Datastore to deploy the VM
$diskStorageFormat = "Thick" # Thin or Thick provisionig of virtual disks
$vm = Import-VApp -Source $ovf -OvfConfiguration $ovfconfig -Name $VMName -VMHost (Get-VMHost -Name $VMHost) -Datastore $datastore -DiskStorageFormat $diskStorageFormat
#$vm | Start-VM # Uncomment to power on the VM after creation.
NSX-T Concepts
Segment
Segment, also known as logical switch, provides switching functionality in an NSX-T Data Center virtual environment. Segments are similar to VLANs. Each segment has a virtual network identifier (VNI), similar to a VLAN ID. VNI scale well beyond the limits of VLAN IDs. A segment is a representation of a layer 2 broadcast domain across transport nodes. Segments segregate network from each other. The VMs connected to a segment can communicate with each other through tunnels between hosts. A segment is created either in an overlay or VLAN based transport zone.
Segment profile include layer 2 network configuration details for logical switches adn logical ports, supports QoS, port mirroring, IP Discovery, SpoofGuard, segment security, MAC management, and Network I/O control.
The NSX VNI range is 5000 thrugh 1677216, and must use MTU of 1600 to account for the encapsulation header.
Segment Command Lines
# Verify segments from NSX CLI
get logical-switches # list all logical switches
get logical-switches <segment-uuid> ports
get logical-switches <segment-uuid> arp-table
get logical-switches <segment-uuid> mac-table
get logical-switches <segment-uuid> vtep
# Verify segments from ESXi host CLI
nsxcli # Enter nsxcli command mode
get logical-switch <segment-VNI>
get logical-switch <segment-VNI> arp-table
get logical-switch <segment-VNI> mac-table
get logical-switch <segment-VNI> vtep-table
# Verify segments from KVM
nsxcli # Enter nsxcli command mode
get logical-switch <segment-VNI>
get logical-switch <segment-VNI> ports
KVM useful commands for troubleshooting
sudo -i # Enter root
virsh list --all # list all VMs
virsh dumpxml <VM-name> | grep interfaceid # Find VM interfaceid
Tier-0 and Tier-1 Gateway
Distributed Router (DR) and Servie Router (SR)
A DR is always created when creating a gateway. The DR component is distributed among all hypervisors and provides basic packet forwarding. The SR component is only located in the NSX Edge nodes and provides service. A SR is automatically created on the edge node when you configure the gateway with an edge cluster.
RouterLink
Routerlink is a type of interface that connects Tier-0 and Tier-1 gateways. The interface is created automatically when Tier-0 and Tier-1 gateways are connected. It uses a subnet assigned from the 100.64.0.0/10 IPv4 subnet.
Intratier Transit Link
The intratier transit link connection is automatically created when a service router (SR) is created. It is an internal link between the SR and DR on a gateway. It has an IP address in 169.254.0.0/28 subnet.
NSX Edge
When you first deploy an NSX Edge, you can think of it as an empty container. The NSX Edge does not do anything until you create logical routers. The NSX Edge provides the compute backing for tier-0 and tier-1 logical routers. Each logical router contains a services router (SR) and a distributed router (DR). When we say that a router is distributed, we mean that it is replicated on all transport nodes that belong to the same transport zone.
An NSX Edge can belong to one overlay transport zone and multiple VLAN transport zones. An NSX Edge belongs to at least one VLAN transport zone to provide the uplink access.
Virtual-Appliance or VM NSX Edge Networking
When you install NSX Edge as a virtual appliance or VM, internal interfaces are created, called fp-ethX, where X is 0, 1, 2, and 3. These interfaces are allocated for uplinks to a top-of-rack (ToR) switches and for NSX-T Data Center overlay tunneling.
When you create the NSX Edge transport node, you can select fp-ethX interfaces to associate with the uplinks and the overlay tunnel. You can decide how to use the fp-ethX interfaces.
On the vSphere distributed switch or vSphere Standard switch, you must allocate at least two vmnics to the NSX Edge: One for NSX Edge management and one for uplinks and tunnels.
Edge Cluster
An NSX edge cluster ensures that at least one NSX edge node is always available. The following guideline are required:
- Maximum 10 edge nodes in a cluster
- An edge transport node can be added to only one edge cluster.
- Maximum of 160 clusters can be configured. One cluster can provide 8 way ECMP path northbound and another cluster can provide centralised services.
NSX edge VM sizing options
Size Memory (GB) vCPU Diskspace (GB)
Small 4 2 200
Medium 8 4 200 # Support up to 64 hypervisors
Large 32 8 200 # Deploy with more than 64 hypervisors
Extra Large 64 16 200
NSX edge node can be deployed as VM on ESXi host or bare-metal node.
Note: NSX edge not can NOT be deployed in KVM.
Edge Node VM Interfaces
The first interface must be defined for management access (eth0) by using one vNIC.
The other interfaces must be assigned to the datapath process.
Edge Node Installation
Install edge node on bare-metal using the ISO file, or from NSX Manager, or from vCenter. By default, the root login password is vmware, and the admin login password is default.
Configure the NSX Edge node with a DNS Server
set name-servers <DNS-Server-IP>
Join NSX Edge Node to Management Plan
Install the NSX edge node by any method other than NSX UI does not automatically join the NSX edge node to the management plane.
Procedure to join the NSX edge node to the management plane:
1. Login to NSX manager appliance via SSH
2. get certificate api thumbprint # Run command to retrieve the SSL thumbprint
3. Login to NSX Edge node via SSH
4. Run the following command
join management-plane <NSX-Manager-IP> username admin password <required password> thumbprint <thumbprint obtained in step 2>
Notes from the field
1. After the Edge node first installed and reboot, login to the CLI with admin credentials
Note: After NSX Edge node starts, if you do not log in with admin credentials for the first time,
the data plane service does not automatically start on the NSX Edge node.
2. Run the get interface eth0 (without VLAN) or get interface eth0.<vlan_ID>
Verify the IP address was applied as expected
3. Run the get managers command to verify that the NSX Edge node is registered.
4. Verify that the NSX Edge node has the required connectivity.
If you enabled SSH, make sure that you can SSH to your NSX Edge node and verify the following:
a. You can ping your NSX Edge node management interface.
b. From the NSX Edge node, you can ping the node's default gateway.
c. From the NSX Edge node, you can ping the hypervisor hosts that are either in the same network or a network reachable through routing.
d. From the NSX Edge node, you can ping the DNS server and NTP Server IP or FQDN list.
5. By default, the NSX Edge node datapath claims all virtual machine NICs except the management NIC (the one that has an IP address and a default route).
If you incorrectly assigned a NIC as the management interface, follow these steps to use DHCP to assign management IP address to the correct NIC.
a. Log in to the NSX Edge CLI and type the stop service dataplane command.
b. Type the set interface interface dhcp plane mgmt command.
c. Place interface into the DHCP network and wait for an IP address to be assigned to that interface.
d. Type the start service dataplane command.
The datapath fp-ethX ports used for the VLAN uplink and the tunnel overlay are shown in the get interfaces and get physical-port commands on the NSX Edge node.
Note: Set static IP address
set interface eth0 ip <CIDR> gateway <gateway-ip> plane mgmt
Edge Cluster
Create an edge cluster for the following reasons:
- Having a multinode cluster of NSX edge nodes ensures that at least one NSX edge node is always available.
- We must associate an edge node with an NSX edge node cluster if we want to create stateful services, such as NAT, load balancer, etc.
Edge Commands Lines
get configuration # Verify configuration
get node-uuid # Verify the node uuid
get interface # list the interfaces
get managers # Verify the NSX manager that manages the manage the edge node
get host-switches
get tunnel-ports # check tunnel ports
get vteps # check VTEP
get high-availability status # check Edge high availability status
NSX-T Troubleshooting
The NSX-T Data Center kernel modules are packaged in VIB files and downloaded to transport nodes. The kernel modules provide services such as distributed routing, distributed firewall, and so on.
The functions of the VIBs are defined as follows:
nsx-esx-datapth # Provides NSX-T Data Center data plane packet-processing functionality
nsx-exporter # Provides host agents that report runtime state to the aggregation service
nsx-host # Provides metadata for the VIB bundle that is installed on the host
nsx-mpa # Provides communication between NSX manager and ESXi hosts
nsx-python-protobuf # Provides Python bindings for the protocol buffers
nsx-sfhc # Service fabric host components provides a host agent for managing the life cycle of the ESXi host as a fabric host
nsxcli # Provides the NSX CLI on ESXi host
Verify KVM Transport Node by CLI
dpkg -l | grep nsx # Ubuntu host
rpm / yum | grep nsx # RHEL and Centos
zypper / YaST | grep nsx # SLES
NSX-T Command Line
set cli-timeout 0 # Disable command line timeout
get cluster status # Verify NSX management cluster status
openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha256 -noout # Get the transport node SSL fingerprint
ESXi Host Command Lines
# verify transport node installation
esxcli software vib list | grep nsx
esxcli software vib list | grep vsip
# Verify kernel module installation
esxcli system module list | grep nsx
# Check VMKernel IPv4 addresses
esxcli network ip interface ipv4 address list
esxcli network ip netstack list # Check vxlan and hyperbus
esxcfg-vswitch -l # check vswitch installation
/etc/init.d/nsx-proxy status # Check NSX-proxy agent running status
esxcli network ip connection list | grep 1234 # NSX manager connection
esxcli network ip connection list | grep 1235 # NSX control plane connection
KVM Host Transport Node Command Lines
sudo -i # Enter root
dpkg --list | grep nsx # Check NSX packages installation
service nsx-proxy status # check nsx-proxy service status
netstat -nap | grep 1234
netstat -nap | grep 1235
You can prepare ESXi, KVM hosts and physical servers as NSX-T Data Center transport nodes. After adding an ESXi host to the NSX-T Data Center fabric, the following VIBs get installed on the host.
# Some of the VIBs
nsx-cfgagent
Provides communication between the central control plane and hypervisors.
Receives logical networking state from the central control plane and programs this state in the data plane.
nsx-mpa
Provides communication between NSX Manager and hypervisor hosts.
nsx-opsagent
Communicates operations agent executions
such as transport node realization, Link Layer Discovery Protocol - LLDP,traceflow, packet capture, etc with the management plane.
nsx-proxy
Provides the only northbound contact point agent, which talks to the central control plane and management plane.
nsx-exporter
Provides host agents that report runtime state to the aggregation service running in the management plane.
nsx-sfhc
Service fabric host component (SFHC).
Provides a host agent for managing the lifecycle of the hypervisor as a fabric host in the management plane's inventory.
This provides a channel for operations such as NSX-T Data Center upgrade and uninstall and monitoring of NSX-T Data Center modules on hypervisors.
nsx-platform-client
Provides a common CLI execution agent, for centralized CLI and audit log collecting.
nsxcli
Provides the NSX-T Data Center CLI on hypervisor hosts.
Prepare Standalone Hosts as Transport Nodes
A transport node is a node that participates in an NSX-T Data Center overlay or NSX-T Data Center VLAN networking.
# Note
If you plan to create transport nodes from a template VM, make sure that there are no certificates on the host in /etc/vmware/nsx/.
nsx-proxy does not create a certificate if a certificate exists.
The following prerequisites need to be met
Prerequisites
1. The host must be joined with the management plane, and connectivity must be Up.
2. he reverse proxy service on all nodes of the NSX Manager cluster must be Up and running.
To verify, run get service http. If the service is down, restart the service by running restart service http on each NSX Manager node.
3. transport zone must be configured.
4. An uplink profile must be configured, or you can use the default uplink profile.
5. An IP pool must be configured, or DHCP must be available in the network deployment.
6. At least one unused physical NIC must be available on the host node.
7. Hostname
8. Management IP address
9. User name
10. Password
11. (Optional) (KVM) SHA-256 SSL thumbprint
12. (Optional) (ESXi) SHA-256 SSL thumbprint
13. Verify that the required third-party packages are installed.
Precedure
1. Retrieve the hypervisor thumbprint so that you can provide it when adding the host to the fabric.
a. ESXi Host
i) Lunix shell
echo -n | openssl s_client -connect <esxi-ip-address>:443 2>/dev/null | openssl x509 -noout -fingerprint -sha256
ii) ESXi CLI in the host
[root@host:~] openssl x509 -in /etc/vmware/ssl/rui.crt -fingerprint -sha256 -nooutSHA256
b. KVM Hypervisor
# awk '{print $2}' /etc/ssh/ssh_host_rsa_key.pub | base64 -d | sha256sum -b | sed 's/ .*$//' | xxd -r -p | base64
2. Select System > Fabric > Nodes > Host Transport Nodes.
3. From the Managed by field, select Standalone Hosts and click + Add Host Node.
4. On the Host Details page, enter details information
5. (Required) For a KVM host, select the N-VDS type.
For an ESXi host, the N-VDS type is always set to NSX Created.
6. On the Configure NSX page, enter details. You can configure multiple N-VDS switches on a single host.
7. For a preconfigured N-VDS, provide N-VDS External ID, and VTEP
8. View the connection status on the Host Transport Nodes page.
9. Alternatively, view the connection status using CLI commands.
esxcli network ip connection list | grep 1234 # ESXi host
netstat -anp --tcp | grep 1234 # KVM host
10. Verify that the NSX-T Data Center modules are installed on the host.
esxcli software vib list | grep nsx # ESXi host
yum list installed or rpm -qa # RHEL, Centos, or Oracle Linux
dpkg --get-selections # Ubuntu
rpm -qa | grep nsx # SUSE
Verify the Transport Node Status
Make sure that the transport node creation process is working correctly.
After creating a host transport node, the N-VDS gets installed on the host.
Procedure
1. Log in to the NSX-T Data Center.
2. Navigate to the Transport Node page and view the N-VDS status.
3. Alternatively
a. view the N-VDS on ESXi
esxcli network ip interface list
b. If you are using the vSphere Client
you can view the installed N-VDS in the UI by selecting host Configuration > Network Adapters.
c. Verify N-VDS on KMV
ovs-vsctl show
Note that on KVM, the N-VDS name is nsx-switch.0
It does not match the name in the transport node configuration. This is by design.
4. Check the transport node's assigned tunnel endpoint address.
a. In ESXi
esxcli network ip interface ipv4 get
Note: The vmk10 interface receives an IP address from the NSX-T Data Center IP pool or DHCP
b. In KVM
ifconfig
Verify nsx-vtep0.0 with inet addr
5. Check the API for transport node state information.
GET https://<nsx-mgr>/api/v1/transport-nodes/<transport-node-id>/state
Tier-0 and Tier-1 Gateway
Before configuring the Tier-0 and Tier-1 gateways, at least one NSX edge node is installed, and an NSX edge cluster is configured.
You must manually connect the Tier-0 and Tier-1 gateways. The management plane cannot determine which Tier-1 instance should connect to which Tier-0 instance.
If there is no stateful service is required for the Tier-1 gateway, must not select any edge cluter.
If no cluster is selected for the Tier-1 gateway, no service router (SR) is created, only distributed router (DR) component is created.
VMs on various subnets or segments attached to the Tier-1 gateway can communicate with each other.
Each Tier-0 gateway can have multiple uplink connections if required or configured. After Tier-0 gateway is created, we could setup the interfaces.
Gateway Command Lines
# Gateway / router are interchangable
get logical-routers # List the routers/gateways, such as vrf number, etc
vrf <number> # Enter the specific vrf number to verify its connection information
get bgp neighbor # List BGP neighbor
get route # List the routing
get route connected # List the direct connected routes
get route bgp # List the GBP learned routes
get forwarding # List the logical router forwarding table
Logical Switches Command Lines
nsxcli # Enter nsxcli command mode
get logical-switches
Configure Routing
After creating the Tier-0 gateway, static or dynamic routing to the remote networks can be setup by editing the Tier-0 gateway.
To propagate tenant networks from Tier-1 gateway to Tier-0 gateway, route advertisement need to enable on the Tier-1 gateway.
Route Advertisement options:
All Static Routes
All DNS Forwarder Routes
All Connected Segments & Service Ports
All IPSec Local Endpoints
All NAT Routes
All LB VIP Routes
All LB SNAT Routes
Set Route Advertisement Rules
Dynamic routing protocol categories
- Interior gateway protool (IGPs)
- Exterior gateway protocol (EGPs)
NSX implements iBGP and external BGP (eBGP)
External BGP is used to establish neighbour relationship between Tier-0 and upstream physical gateways with different AS network. The Tier-0 gateway BGP topology should be configured with redundancy and symmetry between the Tier-0 gateways and the external peers. BGP is enabled by default on Tier-0 gateways.
Logical Routing Command Lines
# Verify logical routers from NSX CLI
get logical-routers
get logical-routers | find <router-name>
# Verify logical routers from ESX CLI
nsxcli # Ener esxcli command monde
get logical-routers
get logical-router <router-uuid> # Find router information
net-vdr -I --brief -l # list logical router on the ESXi host
net-vdr --lif --brief -l <router-uuid> # list the detail LIF information about the specific logical router
NSX-T Virtual IP Address
If NSX-T is integrated with VMware Identity Manager (vIDM) and VMwaer Life Cycle Manager, when configure NSX-T VIP, the following process is required:
1. Create NSX-T appliances SSL, including NSX-T VIP FQDN, and all NSX-T managers FQDN.
2. Install NSX-T appliances SSL to all NSX-T managers
3. Update vIDM NSX environment with NSX-T VIP FQDN
Firewall
Firewall Security Terminology
Construct Definition
Policy # A security policy includes various security elements including firewall rules and service configurations. Policy was previously called a firewall section.
Rule # A set of parameters with which flows are evaluated against, and define which actions will be taken upon a match.
Rules include parameters such as source and destination, service, context profile , logging, and tags.
Group # Groups include different objects that are added both statically and dynamically, and can be used as the source and destination field of a firewall rule.
Groups can be configured to contain a combination of virtual machines, IP sets, MAC sets, logical ports, logical switches, AD user groups, and other nested groups.
Service # Defines a combination or port and protocol. Used to classify traffic based on port and protocol.
Pre-defined services and user-defined services can be used in firewall rules.
Context Profile # Defines context aware attributes including APP-ID and domain name.
Also includes sub attributes such as application version, or cipher set.
Firewall rules can include a context profile to enable Layer-7 firewall rules
Identity Firewall
With Identity Firewall (IDFW) features an NSX administrator can create Active Directory userbased Distributed Firewall (DFW) rules. IDFW can be used for Virtual Desktops (VDI) or Remote desktop sessions (RDSH support), enabling simultaneous log ins by multiple users, user application access based on requirements, and the ability to maintain independent user environments
Troubleshooting Distributed Firewall
Distributed firewall policies have the following categories:
Ethernet
Emergency
Infrastructure
Environment
Application
NSX-T distributed firewall rule processing is carried out:
VMware Internetworking Service Insertion Platform (VSIP) module is the main part of the distributed firewall kernel module that receives the firewall rules and downloads them on the VM’s vNIC.
# Verify distributed firewall from ESX CLI
esxcli software vib list | grep vsip # Check firewall VIB installation
/etc/init.d/nsx-proxy status # Check nsx-proxy service status
esxcli network ip connection list | grep '1234\|1235' # Check communication/connection between ESXi host and NSX manager (management and CCP)
/etc/init.d/ntpd status # Time based distributed firewall rules need ntp service
summarize-dvfilter # Restrieve the distributed firewall filters
summarirze-dvfilter | grep -A10 <VM> # get the distribute firewall rules associates with the <VM>
vsipioctl getrules -f <dvfilter> # List the distribute firewall rule associates with a dvfilter
vsipioctl getfwconfig -f <dvfilter>
# List the distribute firewall configuration associates with a dvfiler, detail information about addrset
vsipioctl getcontainers -f <dvfilter> # Check firewall scheduler for a given VMware firewall identifier
# Verify distribute firewall from KVM
sudo -i # enter root
ovs-appctl -t /var/run/openvswitch/nsxa-ctl dfw/vif
ovs-appctl -t /var/run/openvswitch/nsxa-ctl dfw/vif <VIF-id>
ovs-appctl -t /var/run/openvswitch/nsxa-ctl dfw/vif <addrset>
# Verifiy firewall from NSX Manager CLI
get firewall summary
get firewall status
get firewall puslished-entity # check firewall rules published to NSX manager CCP
Verify firewal from API
https://code.vmware.com/apis/329/nsx-for-vsphere
Gateway Firewall Troubleshooting
Gateway firewall policies have the following categories:
Emergency
System
Pre Rules
Local Gateway
Auto Service
Default
Gateway firewall rules are programmed into rule classifier.
# Verify firewall from NSX edge node
get firewall interfaces # check edge node interfaces that have Gateway firewall rule configured.
get firewall <interface-id> ruleset rules # Check rule set applies to the interface
get service ntp # Time based firewall rule needs to ntp service running
get ntp-server associations # Check gateway edge ntp service communication with NTP server
#**** Check log files when troubleshooting gateway firewall rules
# NSM Manager CLI
get log-file policy
get log-file manager
get log-file controller
# Edge Node
get log-file syslog
NSX-T Services
Load Balancer
NSX-T 3.0 supports layer 4 and layer 7 load balancer. Load balancer runs on Tier-1 gateway edge cluster in an active-standby mode.
It is commonly deployed in one-arm or inlined deployment.
- Layer 4 - L4 load balancer is connection-based, it supports TCP and UDP protocols
- Layer 7 - L7 load balancer is content-based, it supports HTTP and HTTPS, also support URL manipulation through user defined rules.
Distributed Load Balancer
A Distributed Load Balancer (DLB) configured in NSX-T Data Center can help you effectively load balance East-West traffic and scale traffic because it runs on each ESXi host.
# Important
Distributed Load Balancer is supported only for Kubernetes (K8s) cluster IPs managed by vSphere with Kubernetes. Distributed Load Balancer is not supported for any other
workload types.
As an administrator, you cannot use NSX Manager GUI to create or modify Distributed Load Balancer objects. These objects are pushed by vCenter Server through NSX-T
API when K8 cluster IPs are created in vCenter Server.
# Note
Do not enable Distributed Intrusion Detection and Prevention Service (IDS/IPS) in an environment that is using Distributed Load Balancer. NSX-T Data Center does not support using IDS/IPS with a Distributed Load Balancer.
In traditional networks, a central load balancer deployed on an NSX Edge node is configured to distribute traffic load managed by virtual servers that are configured on the load balancer.
If you are using a central balancer, increasing the number of virtual servers in the load balancer pool might not always meet scale or performance criteria for a multi-tier distributed application. A distributed load balancer is realized on each hypervisor where load balancing workloads, such as clients and servers are deployed, ensuring traffic is load balanced on each hypervisor in a distribued way.
A distributed load balancer can be configured on the NSX-T network along with a central load balancer.
# NSX Edge CLI commands - Load Balancer
get load-balancers # List the load balancers information, such as UUID and deployment size
get load-balancer <lb-uuid> # Verify load balancer running state, HA state
get load-balancer <lb-uuid> virtual-server
get load-balancer <lb-uuid> virtual-server <virtual-server-uuid> # verify virtual server profile
get load-balancer <lb-uuid> virtual-server <virtual-server-uuid> status # verify VIP status
get load-balancer <lb-uuid> virtual-server <virtual-server-uuid> stats # check statistics
get load-balancer <lb-uuid> pools # list all pools
get load-balancer <lb-uuid> health-check-table # show health of members
get load-balancer <lb-uuid> pool <pool_uuid> status
get load-balancer <lb-uuid> error-log # check errror log
get load-balancer <lb-uuid> virtual-server <virtualserver-uuid> access-log # Verify access log
# API Verification
GET https://<nsx-ip>/api/v1/loadbalancer/services/<Load-balancer-uuid>/debug-info # List full configuration
Gateway Firewall
get firewall interfaces # list all edge interfaces that have firewall rules configured
# show uuid, type, name, vrf ID, context name
get firewall <interface-uuid> ruleset rules # Query firewall rules associates with the interface
VPN Service
IPSec VPN
IPSec VPN secures traffic flowing between two networks connected over a public network through IPSec gateways called endpoints.
IPSec VPN uses the IKE protocol to negotiate security paramters. The default UDP port is set to 500. If NAT is detected in the gatewway, the port is set to UDP 4500.
NSX Edge supports a policy-based or a route-based IPSec VPN.
The Internet Key Exchange (IKE) profiles provide information about the algorithms that are used to authenticate, encrypt, and establish a shared secret between network sites when you establish an IKE tunnel.
Policy Based IPSec VPN
Policy-based IPSec VPN requires a VPN policy to be applied to packets to determined which traffic is to be protected by IPSec before being passed through the VPN tunnle.
Note: DNAT is not supported on a tier-1 gateway where policy-based IPSec VPN is configured.
Route Based IPSec VPN
Route-based IPSec VPN provides tunneling on traffic based on the static routes or routes learned dynamically over a special interface called virtual tunnel interface (VTI) using, for example, BGP as the protocol. IPSec secures all the traffic flowing through the VTI.
Note:
a. OSPF dynamic routing is not supported for routing through IPSec VPN tunnels.
b. Dynamic routing for VTI is not supported on VPN that is based on Tier-1 gateways.
Route-based IPSec VPN is similar to Generic Routing Encapsulation (GRE) over IPSec, with the exception that no additional encapsulation is added to the packet before applying IPSec processing.
Layer 2 VPN
With Layer 2 VPN (L2 VPN), you can extend Layer 2 networks (VNIs or VLANs) across multiple sites on the same broadcast domain. This connection is secured with a route-based IPSec tunnel between the L2 VPN server and the L2 VPN client. The extended network is a single subnet with a single broadcast domain, which means the VMs remain on the same subnet when they are moved between sites.
L2 VPN services are supported on both Tier-0 and Tier-1 gateways. Only one L2 VPN service (either client or server) can be configured for either Tier-0 or Tier-1 gateway.
Each L2 VPN session has one Generic Routing Encapsulation (GRE) tunnel. Tunnel redundancy is not supported. An L2 VPN session can extend up to 4094 L2 segments.
VLAN-based and VNI-based segments can be extended using L2 VPN service on an NSX Edge node that is managed in an NSX-T Data Center environment. You can extend L2 networks from VLAN to VNI, VLAN to VLAN, and VNI to VNI. Segments can be connected to either Tier-0 or Tier-1 gateways and use L2 VPN services.
Autonomous Edge as an L2 VPN Client
You can use L2 VPN to extend your Layer 2 networks to a site that is not managed by NSX-T Data Center. An autonomous NSX Edge can be deployed on the site, as an L2 VPN client. The autonomous NSX Edge is simple to deploy, easily programmable, and provides high-performance VPN. The autonomous NSX Edge is deployed using an OVF file on a host that is not managed by NSX-T Data Center. You can also enable high availability (HA) for VPN redundancy by deploying primary and secondary autonomous Edge L2 VPN clients.
Overly network over NSX-T VPN
https://rutgerblom.com/2020/12/07/vpn-your-way-to-nsx-t-overlay/
Site-to-Site VPN Between NSX-T Tier-1 And AWS VPC
https://rutgerblom.com/2020/01/07/site-to-site-vpn-between-nsx-t-tier-1-and-aws-vpc/
Hub and Spoke Layer 2 VPNs between multiple NSX-T enabled sites
https://nsx.ninja/index.php/Hub_and_Spoke_Layer_2_VPNs_between_multiple_NSX-T_enabled_sites
Add an IPSec VPN Service
NSX-T provides support for two types VPN services:
- IPSec VPN provides secure transport service between locations connected by public or non-secured IP netowrks.
It supports two types of VPN
a. Policy based VPN
VPN policy is used to determine traffic and network topology configuration
b. Route based VPN
VPN gateways use GBP protocol, and routes are learned by using GBP.
Routes are learned at a specific interface (Virtual Tunnel Interface VTI)
It supports the following connections:
a. Provides interconnect remote IP networks between different locations
b. Uses a routing based connection between different, nonoverlappping IP subnets
c. Provides secure commmunication for nonsecure protocols, such as GRE (Generic Routing Encapsulation)
- Layer 2 VPN
It enables company to extend layer 2 network across two different locations, data centres securely.
a. It uses IPSec VPN (route based) as the transport tunnel for layer 2 traffic
b. The VLAN or overlay segments can stretch across two locations
# Command Line Verifications
get l2vpn session
get l2vpn services config
get l2vpn session <session-uuid> stats
get l2vpn session <session-uuid> logical-switches
IPSec Service Verification
# NSX Edge CLI commands
get ipsecvpn session summary # Provides information, such as session id, IPSec version, Authentication, local IP and remote IP
get ipsecvpn session sessionid <session-id>
# session-id is the output from session summary
# Example get ipsecvpn session sessionid 2963
# List UUID of the <session-id>, local ip and peer ip, type, SR ID (Service Router ID)
get ipsecvpn ikesa <session-id> # Retrieve IKE security associations in detail, such as Service Parameter Index (SPI)
get ipsecvpn sad <policy-id> # Retrieve IPSec policy UUID, Encapsulating Security Payload (ESP)
get ipsecvpn ipsecsa session-id <session-id>
get ipsecvpn tunnel stats
IPSec Logging
IPSec vPN syslog locates at /var/log/syslog
set service ike logging-level <level> # configure logging level
Ethernet VPN (EVPN)
EVPN (Ethernet VPN) is a standards-based BGP control plane that provides the ability to extend Layer 2 and Layer 3 connectivity between different data centers.
Configure an EVPN Tenant
If you configure EVPN Route Server mode, you must configure an EVPN tenant. To configure an EVPN tenant, you must specify one or more VLAN-VNI mappings. The VLANs identify the tenant. Each VNI identifies a VRF gateway.
Forwarding Policies
This feature pertains to NSX Cloud.
Forwarding Policies or Policy-Based Routing (PBR) rules define how NSX-T handles traffic from an NSX-managed VM. This traffic can be steered to NSX-T overlay or it can be routed through the cloud provider's (underlay) network.
Three default forwarding policies are set up automatically after you either deploy a PCG on a Transit VPC/VNet or link a Compute VPC/VNet to the Transit.
a. Route to Underlay # for all traffic that is addressed within the Transit/Compute VPC/VNet
b. Route from Underlay # for all traffic destined to the metadata services of the public cloud.
c. Route to Overlay # for all other traffic, for example, traffic that is headed outside the Transit/Compute VPC/VNet.
Such traffic is routed over the NSX-T overlay tunnel to the PCG and further to its destination.
Network Settings
vSphere Distributed Switch for NSX-T
# Important
To create a VDS switch supporting NSX-T networking, the following conditions must be met:
1. vCenter Server 7.0 or a later version
2. ESXi 7.0 or a later version
N-VDS support mode: Standard, Enhanced Datapath
Multicast
You can configure multicast on a tier-0 gateway and optionally on a tier-1 gateway for an IPv4 network to send the same multicast data to a group of recipients. In a multicast environment, any host, regardless of whether it is a member of a group, can send to a group. However, only the members of a group will receive packets sent to that group.
# The multicast feature has the following capabilities and limitations:
a. PIM Sparse Mode with IGMPv2.
# Protocol Independent Multicast (PIM)
# Internet Group Management Protocol (IGMP)
b. No Rendezvous Point (RP) or Bootstrap Router (BSR) functionality on NSX-T. However, RP information can be learned via PIM Bootstrap Messages (BSMs). In addition, multiple Static RPs can be configured.
c. The Reverse Path Forwarding (RPF) check for all multicast-specific IPs (senders of data traffic,
BSRs, RPs) requires that a route to each of them exists.
d. The RPF check requires a route to each multicast-specific IP with an IP address as the next
hop. Reachability via device routes, where the next hop is an interface index, is not
supported.
e. Both tier-0 and tier-1 gateways are supported. To enable multicast on a tier-1 gateway, an Edge cluster must be selected and the tier-1 gateway must be linked to a tier-0 gateway that also has multicast enabled.
f. All uplinks on a tier-0 gateway are supported.
g. Multiple Static RPs with discontinuous group ranges are supported.
h. IGMP local groups on uplink interfaces are supported.
i. PIM Hello Interval and Hold Time are supported.
j. Active-Cold Standby only is supported.
k. The NSX Edge cluster can be in active-active or active-standby mode.
l. East-west multicast replication: up to 4 VTEP segments for maximum replication efficiency.
m. ESXi host and NSX Edge only (KVM not supported).
n. Layer 2 bridge attached to a downlink segment not supported.
o. Edge Firewall services are not supported for multicast. Distributed Firewall is supported.
p. Multi-site (NSX Federation) not supported.
q. Multi-VRF not supported.
Distributed IDS/IPS
Distributed Intrusion Detection and Prevention Service (IDS/IPS) monitors network traffic on the host for suspicious activity.
Signatures can be enabled based on severity. A higher severity score indicates an increased risk associated with the intrusion event. Severity is determined based on the following:
a. Severity specified in the signature itself
b. CVSS (Common Vulnerability Scoring System) score specified in the signature
c. Type-rating associated with the classification type IDS detects intrusion attempts based on already known malicious instruction sequences. The detected patterns in the IDS are known as signatures. You can set a specific signature alert/drop/reject actions globally, or by profile
# ssh to NSX virtual appliance
nsxcli # enter nsxcli command mode
get ids status
get ids profiles
get ids engine profilestats <profile-id>
Endpoint Protection
NSX-T Data Center allows you to insert third-party partner services as a separate service VM that provides Endpoint Protection services . A partner Service VM processes file, process, and registry events from the guest VM based on the endpoint protection policy rules applied by the NSX-T Data Center administrator.
NSX-T Data Center Multisite
NSX-T Data Center supports multisite deployments where you can manage all the sites from one NSX Manager cluster.
Two types of multisite deployments are supported:
1. Disaster recovery
2. Active-active
NSX Federation
With NSX Federation, you can manage multiple NSX-T Data Center environments with a single pane of glass view, create gateways and segments that span one or more locations, and configure and enforce firewall rules consistently across locations.
Once you have installed the Global Manager and have added locations, you can configure networking and security from Global Manager.
Networking in NSX Federation
Tier-0 gateways, tier-1 gateways, and segments can span one or more locations in the NSX Federation environment.
# When you plan your network topology, keep these requirements in mind:
1. Tier-0 and tier-1 gateways can have a span of one or more locations.
2. The span of a tier-1 gateway must be equal to, or a subset of, the span of the tier-0 gateway it is attached to.
3. A segment has the same span as the tier-0 or tier-1 gateway it is attached to. Isolated segments are not realized until they are connected to a gateway.
4. NSX Edge nodes in the Edge Cluster selected on the Global Manager for tier-0 and tier-1 gateways must be configured with the Default TZ Overlay.
Packet Capture Troubleshooting
Use the packet capture tools in NSX-T transport node and ESXi hypervisor for troubleshooting.
https://www.simongreaves.co.uk/nsx-t-cli-reference-guide/
Capturing a network trace in ESXi using Tech Support Mode or ESXi Shell (1031186)
https://kb.vmware.com/s/article/1031186
http://www.vmwarearena.com/capture-network-traffic-for-esxi-host/
http://www.vmwarearena.com/how-to-capture-network-trafficpacket-on-esxi-hosts/
https://doogleit.github.io/2019/04/capturing-network-traffic-on-esxi-hosts/#
Detail walk throught with tcpdump-uw and pktcap-uw
https://www.virten.net/2015/10/esxi-network-troubleshooting-with-tcpdump-uw-and-pktcap-uw/
start capture
pktcap-uw & tcpdump-uw # ESXi commands
tcpdump # KVM and Linux hypervisors
set capture session <session-number>
set capture session <session-number> [file <filename>] [count <packet-count>] [expression <expression>]
set capture session <session-number> direction <direction>
set capture session <session-number> interface <interface-name> direction <direction>
set capture session <session-number> interface <port-uuid-name> direction <direction>
How to Changne NSX-T Virutal IP Address
- Access NSX node (not VIP) IP address url and login locally
https://<NSX-node-ip>/login.jsp?local=true
# MUST access NSX manager node IP address, not FQDN or hostname
- Navigate to System -> Appliance
- In the Virtual IP field, select Edit virtual IP to update VIP
# Set NSX cluster with virtual IP address if NSX is integrated with vIDM
1. If NSX is integrated with VMware Identity Manager (vIDM), need to generate new NSX manager SSL certificate first,
2. Then update NSX VIP and manager node with new SSL certificate, finally
3. Set NSX manager with Virtual IP address
NSX Manager Cluster Integration with VMware Identity Manager
- Add new remote app access in vIDM for NSX access
a. Login to vIDM management console
https://<vIDM-fqdn>
b. Navigate to top right window, select or click the login name, and choose "Administration Console"
c. Select Catalog tab, then select Settings from the drop down list
Web Apps
Virtual apps
Settings
d. Select Remote App Access from the selection list
Global Settings
Remote App Access
User Portal Branding
New End User Portal UI
People Search
e. In Remote App Access, create or access the NSX Client
i. On Create Client window, on the Access Type drop down selection, choose "Service Client token"
ii. Enter the required Client ID, such as NSX-Client-Access
iii. Expand Advanced and fill out the other information
Token Type: Bearer
Keep other as default values
- Access Token Time-To-Live (TTL)
- Refresh Token Time-To-Live (TTL)
- Idle Token Time-To-Live (TTL)
f. click Generate Shared Secret, to generate the shared secret or type in the share secret value
Note: This shared secret will be used in NSX VMware vIDM configuration
- Configure NSX manager with vIDM
a. Verify NSX vIDM integration
i. Login to NSX management console
ii. Navigate to System -> Settings -> Users and Roles
iii. Select VMware Identity Manager tab
iv. Click Edit
It will show the following configuration
Item/Component Value/Configuration
--------------------------------------------------------------------
External Load Balancer Enabled/Disabled
VMware Identity Manager Integration Enabled/Disabled # Select Enabled
VMware Identity Manager Appliance <vIDM-fqdn>
oAuth client Secret The shared secret configured in above vIDM shared secret
SSL Thumbprint vIDM SSL thumbprint
NSX Appliance <nsx-fqdn> # This will be the manager cluster url value
Manaually join the new NSM manager node to NSX manager cluster
1. SSH to NSX manager cluster node
2. Get NSX manager API certifiicate thumbpring
get certificate api thumbprint
3. Get cluster ID
get cluster config # note down the cluster ID from the output
4. SSH to the new NSX manager node
join <nsx-cluster-node-ip> cluster-id <cluster-id> thumbprint <cluster thumbprint>
5. get cluster status # verify cluster status
Authentication and Authorization
In NSX-T Data Center 3.1, you can log in to NSX Manager using a local user account, a user account managed by VMware Identity Manager (vIDM), or a user account managed by a directory service such as Active Directory over LDAP or OpenLDAP. You can also assign roles to user accounts managed by vIDM or a directory service to implement role-based access control.
Local user passwords on NSX appliances are secured using the default Linux/PAM libraries which store the hashed and salted representation in /etc/shadow.
## On NSX manager (without vIDM integration)
set user <username> password # change password
get user admin password-expriration # Get password expiration information
set user admin password-expiration <number-of-days> # set passsword expiration days
clear user <admin/audit> password-expiration # disable password expiry so the password never expires
## On NSX Edge node
passwd admin # Reset admin password
How to Reset the Forgotten Password of an Appliance
The following procedure applies to NSX Manager, NSX Edge, Cloud Service Manager, and NSX Intelligence appliances.
Note: When you reboot an appliance, the GRUB boot menu does not appear by default. The following procedure requires that you have configured GRUB to display the GRUB boot menu.
## Procedure
1 Connect to the console of the appliance.
2 Reboot the system.
3 When the GRUB boot menu appears, press the left SHIFT or ESC key quickly. If you wait too long and the boot sequence does not pause, you must reboot the system again.
4 Press e to edit the menu.
Enter the user name root and the GRUB password for root (not the same as the appliance's user root).
# Default GRUB password VMware1
5 Press e to edit the selected option.
6 Search for the line starting with linux and add systemd.wants=PasswordRecovery.service to the end of the line.
7 Press Ctrl-X to boot.
8 When the log messages stop, enter the new password for root.
9 Enter the password again.
The boot process continues.
10 After the reboot, you can verify the password change by logging in as root with the new password
How to Integrate vIDM with NSX-T
Before you configure the integration of vIDM with NSX-T, you must get the certificate thumbprint from the vIDM host.
You must use OpenSSL version 1.x or higher for the thumbprint. In the vIDM host, the command openssl runs an older OpenSSL version and therefore you must use the command openssl1 in the vIDM host. This command is only available from the vIDM host.
In a server that is not the vIDM host, you can use the openssl command that is running OpenSSL version 1.x or higher.
Procedure
1 Log in to the vIDM host's console or by using SSH or log in to any server that can ping thevIDM host.
2 Use OpenSSL version 1.x or higher to get the thumbprint of the vIDM host.
a) openssl1: If you are logged in to the vIDM host in a console or using SSH, run the following command to get the thumbprint:
openssl1 s_client -connect <FQDN of vIDM host>:443 < /dev/null 2> /dev/null | openssl x509 -sha256 -fingerprint -noout -in /dev/stdin
b) openssl: If you are logged in to a server that can ping the vIDM host but is not the vIDM host, run the following command to get the thumbprint:
openssl s_client -connect <FQDN of vIDM host>:443 < /dev/null 2> /dev/null | openssl x509 -sha256 -fingerprint -noout -in /dev/stdin
You can integrate NSX-T Data Center with VMware Identity Manager (vIDM), which provides identity management services. The vIDM deployment can be a standalone vIDM host or a vIDM cluster.
Note: The new product name for VMware Identity Manager is VMware Workspace ONE Access.
# NSX Login after integration with vIDM
When you register NSX Manager with vIDM, you specify a redirect URI that points to NSX Manager. You can provide either the fully qualified domain name (FQDN) or the IP address.
** It is important to remember whether you use the FQDN or the IP address.
When you try to log in to NSX Manager through vIDM, you must specify the host name in the URL the same way, that is, if you use the FQDN when registering the manager with vIDM, you must use the FQDN in the URL, and if you use the IP address when registering the manager with vIDM, you must use the IP address in the URL. Otherwise, login will fail.
Note NSX Managers and vIDM must be in the same time zone. The recommended way is to use UTC.
With vIDM enabled, you can still log in to NSX Manager with a local user account if you use the URL
https://<nsx-manager-ip-address>/login.jsp?local=true.
NSX-T Certificates
There are three categories of self-signed certificates in NSX-T Data Center.
1. Platform Certificates
2. NSX Services Certificates
3. Principal Identity Certificates
Platform Certificates
After installing NSX-T Data Center, navigate to System > Certificates to view the platform certificates created by the system. By default these are self-signed X.509 RSA 2048/SHA256 certificates for internal communication within NSX-T Data Center and for external authentication when NSX Manager is accessed using APIs or the UI.
NSX Service Certificates
NSX service certificates are used for services such as load balancer and VPN.
NSX service certificates cannot be self signed. You must import them.
Principal Identity (PI) Certificates
PI certificates can be for services or for platform.
PI for Cloud Management Platforms (CMP), such as Openstack, uses X.509 certificates that are uploaded when onboarding a CMP as a client.
Certificates for NSX Federation
The system creates certificates required for communication between NSX Federation appliances as well as for external communication.
By default, the Global Manager uses self-signed certificates for communicating with internal components and registered Local Managers, as well as for authentication for NSX Manager UI or APIs.
You can view the external (UI/API) and inter-site certificates in NSX Manager. The internal certificates are not viewable or editable.
Backing Up and Restoring NSX Manager or Global Manager
While the appliance is inoperable, the data plane is not affected, but you cannot make configuration changes.
# Requirement
1. You must restore to new appliances running the same version of NSX-T Data Center as the appliances that were backed up.
2. If you are using an NSX Manager or Global Manager IP address to restore, you must use the same IP address as in the backup.
3. If you are using an NSX Manager or Global Manager FQDN to restore, you must use the same FQDN as in the backup. Note that only lowercase FQDN is supported for backup and restore.
Create the SFTP server, verify that the SFTP server is ready for use and is running SSH and SFTP, using the following commands:
ssh backup_user@sftp_server
sftp backup_user@sftp_server
Ensure that the directory path exists where you want to store your backups. You cannot use the root directory (/).
If you have multiple NSX-T Data Center deployments, you must use a different directory for storing the backup of each deployment.
You can take backups using either the IP address or the FQDN of the NSX Manager or Global Manager appliance:
a) If you are using the IP address for backup and restore, do not publish the appliance's FQDN.
b) If you are using FQDN for backup and restore, you must configure and publish the FQDN before starting the backup. Backup and restore only support lowercase FQDN. Use this API to publish the NSX Manager or Global Manager FQDN.
Example request:
PUT https://<nsx-mgr OR global-mgr>/api/v1/configs/management
{
"publish_fqdns": true,
"_revision": 0
}
Backup Procedure
## Procedure
1 From a browser, log in with admin privileges to the NSX Manager or Global Manager at
https://<manager-ip-address>.
2 Select System > Backup & Restore.
3 Click Edit under the SFTP Server label to configure your SFTP server.
4 Enter the IP address or FQDN of the backup file server.
5 Change the default port if necessary. The default port is 22.
6 The protocol text box is already filled in.
SFTP is the only supported protocol.
7 In the Directory Path text box, enter the absolute directory path where the backups will be stored.
The directory must already exist and cannot be the root directory (/). Avoid using path drive letters or spaces in directory names; they are not supported.
If the backup file server is a Windows machine, you must use the forward slash when you specify the destination directory.
For example, if the backup directory on the Windows machine is c:\SFTP_Root\backup, specify /SFTP_Root/backup as the destination directory.
8. Enter the user name and password required to log in to the backup file server.
The first time you configure a file server, you must provide a password. Subsequently, if you reconfigure the file server, and the server IP or FQDN, port, and user name are the same, you do not need to enter the password again.
9. You can leave the SSH Fingerprint blank and accept or reject the fingerprint provided by the server after you click Save in a later step. If necessary, you can retrieve the SSH fingerprint by using this API:
POST /api/v1/cluster/backups?action=retrieve_ssh_fingerprint
Note that only SHA256 hashed ECDSA (256 bit) host key is accepted as a fingerprint. .
10 Enter a passphrase.
Important You will need this passphrase to restore a backup. If you forget the passphrase, you cannot restore any backups.
11 Click Edit under the Schedule label.
12 Click Save.
Restore a Backup
Restoring a backup restores the state of the network at the time of the backup. In addition, the configurations maintained by NSX Manager or Global Manager appliances are also restored. For NSX Manager, any changes, such as adding or deleting nodes, that were made to the fabric since the backup was taken, are reconciled. Note DNS entries (name servers and search domains) are not retained when you restore from a backup.
1. You must restore the backup to a new NSX Manager or Global Manager appliance.
2. If you had a cluster of the NSX Manager appliance when the backup was taken, the restore process restores one node first and then prompts you to add the other nodes. You can add the other nodes during the restore process or after the first node is restored.
3. If you had a cluster of Global Manager appliances, you can only restore one node using the restore process. You must create the cluster after the restore of the first node completes.
#****** Important *******
If any nodes in the appliance cluster are still available, you must power them off before you start the restore.
Prerequisites
1. Verify that you have the login credentials for the backup file server.
2. Verify that you have the SSH fingerprint of the backup file server. Only SHA256 hashed ECDSA (256 bit) host key is accepted as a fingerprint.
3. Verify that you have the passphrase of the backup file.
4. Identify which backup you want to restore by following the procedure in Listing Available Backups. Take note of the IP or FQDN of the NSX-T Data Center appliance that took the backup.
Restore Procedure
# Restore Procedure
1. If any nodes in the appliance cluster that you are restoring are online, power them off.
2. Install one new appliance node on which to restore the backup.
a. If the backup listing for the backup you are restoring contains an IP address, you must deploy the new NSX Manager or Global Manager node with the same IP address. Do not configure the node to publish its FQDN.
b. If the backup listing for the backup you are restoring contains an FQDN, you must configure the new appliance node with this FQDN and publish the FQDN. Only lowercase
FQDN is supported for backup and restore.
Note: Until the FQDN is configured and published, the Restore button for the backup is disabled in the newly deployed NSX Manager or Global Manager UI.
Use this API to publish the NSX Manager or Global Manager FQDN.
Example request:
PUT https://<nsx-mgr OR global-mgr>/api/v1/configs/management
{
"publish_fqdns": true,
"_revision": 0
}
In addition, if the new manager node has a different IP address than the original one, you must update the DNS server's forward and reverse lookup entries for the manager node with the new IP address.
After the new manager node is running and online, you can proceed with the restore
3. From a browser, log in with admin privileges to the NSX Manager or Global Manager at
https://<manager-ip-address>.
4. Select System > Backup & Restore.
5. To configure the backup file server, click Edit.
Do not configure automatic backup if you are going to perform a restore.
6. Enter the IP address or FQDN.
7. Change the port number, if necessary.
The default is 22.
8. To log in to the server, enter the user name and password.
9. In the Destination Directory text box, enter the absolute directory path where the backups are stored.
10. Enter the passphrase that was used to encrypt the backup data.
11. You can leave the SSH Fingerprint blank and accept or reject the fingerprint provided by the server after you click Save in a later step. If necessary, you can retrieve the SSH fingerprint by using this API:
POST /api/v1/cluster/backups?action=retrieve_ssh_fingerprint.
12. Click Save.
13. Select a backup.
14. Click Restore.
15. The restore process prompts you to take action, if necessary, as it progresses.
Note: If you are restoring a Global Manager appliance, the following steps do not appear.
After restoring the first Global Manager node, you must manually join the other nodes to form the cluster
A progress bar displays the status of the restore operation noting the step the restore process is on. During the restore process, services on the manager appliance get restarted and the control plane becomes unavailable until restore completes. After the restore operation is finished, the Restore Complete screen shows the result of the restore, the timestamp of the backup file, and the start and end time of the restore operation. Any segments created after the backup was taken are not restored.
You can also determine if there was a cluster restore or node restore failure by selecting the og files. Run get log-file syslog to view the system log file and search for the strings Cluster restore failed and Node restore failed.
To restart the manager, run the restart service manager command.
To reboot the manager, run the reboot command.
16. If you have only one node deployed, after the restored manager node is up and functional, you can deploy additional nodes to form a cluster.
17. If you had other manager cluster VMs that you powered down in Step 1, delete them after the new manager cluster is deployed.
Certificate Management after Restore
After restoring your NSX Manager appliances, certificates in the system get into an inconsistent state and you must update all self-signed or CA-signed certificates.
How to Change the IP Address of an NSX Manager
You can change the IP address of an NSX Manager in an NSX Manager cluster. This section describes several approaches. For example, if you have a cluster consisting of Manager A, Manager B, and Manager C, you can change the IP address of one or more of the managers in the following ways:
Scenario A:
Manager A has IP address 172.16.1.11.
Manager B has IP address 172.16.1.12.
Manager C has IP address 172.16.1.13.
1. Add Manager D with a new IP address, for example, 192.168.55.11.
2. Remove Manager A.
3. Add Manager E with a new IP address, for example, 192.168.55.12.
4. Remove Manager B.
5. Add Manager F with a new IP address, for example, 192.168.55.13.
6. Remove Manager C.
How to Resize an NSX Manager Node
Procedure
1 Deploy a new manager node with the new size.
2 Add the new manager node to the cluster.
3 Remove an old manager node.
4 Repeat steps 1 to 3 to replace the other two old manager nodes.
Replacing an NSX Edge Transport Node in an NSX Edge Cluster
You can replace an NSX Edge transport node in an NSX Edge cluster using the NSX Manager UI or the API.
# Procedure
1 If you want the new Edge transport node to have the same configurations as the Edge transport node to be replaced, make the following API call to find the configurations:
GET https://<nsx-manager-IP>/api/v1/transport-nodes/<tn-id>
2 Follow the procedures in the NSX-T Data Center Installation guide to install and configure an Edge transport node.
If you want this Edge transport node to have the same configurations as the Edge transport node to be replaced, use the configurations obtained in step 1.
3 In NSX Manager, select System > Fabric > Nodes > Edge Clusters.
4 Select an Edge cluster by clicking the checkbox in the first column.
5 Click Actions > Replace Edge Cluster Member.
It is recommended that you place the transport node being replaced in maintenance mode.
If the transport node is not running, you can safely ignore this recommendation.
6 Select the node to be replaced from the dropdown list.
7 Select the replacement node from the dropdown list.
8 Click Save.
Log Messages and Error Codes
NSX-T Data Center components write to log files in the directory /var/log.
Viewing Logs
On NSX-T appliances syslog messages are in /var/log/syslog. On KVM hosts, syslog messages re in /var/log/vmware/nsx-syslog.
On NSX-T appliances, you can run the following NSX-T CLI command to view the logs:
get log-file <auth.log | controller | controller-error | http.log | kern.log | manager.log | nodemgmt.log | policy.log | syslog> [follow]
The log files are:
Name Description
auth.log Authorization log
controller Controller log
controller-error Controller error log
http.log HTTP service log
kern.log Kernel log
manager.log Manager service log
node-mgmt.log Node management log
nsx-audit-write.log NSX audit write log
nsx-audit.log NSX audit log
policy.log Policy service log
syslog System log
On hypervisors, you can use Linux commands such as tac, tail, grep, and more to view the logs.
Configure Remote Logging
Procedure
1 To configure remote logging on an NSX-T Data Center appliance, run the following command to configure a log server and the types of messages to send to the log server. Multiple facilities or message IDs can be specified as a comma delimited list, without spaces.
set logging-server <hostname-or-ip-address[:port]> proto <proto> level <level> [facility <facility>] [messageid <messageid>]
[serverca <filename>] [clientca <filename>] [certificate <filename>] [key <filename>] [structured-data <structured-data>]
You can run the command multiple times to add multiple configurations. For example:
nsx> set logging-server 192.168.110.60 proto udp level info facility syslog messageid SYSTEM,FABRIC
nsx> set logging-server 192.168.110.60 proto udp level info facility auth,user
To forward only audit logs to the remote server, specify audit="true" in the structureddata parameter. For example:
set logging-server <server-ip> proto udp level info structured-data audit="true"
2 To configure secure remote logging using the protocol LI-TLS, specify the parameter proto li-tls. For example:
set logging-server vrli.prome.local proto li-tls level info messageid
SWITCHING,ROUTING,FABRIC,SYSTEM,POLICY,HEALTHCHECK,SHA,MONITORING serverca intermed-ca-fullchain.crt
If the configuration is sucessful, you will get a prompt without any text.
3 To configure secure remote logging using the protocol TLS, specify the parameter proto tls. For example:
set logging-server vrli.prome.local proto tls level info serverca Orange-CA.crt.pem clientca
Orange-CA.crt.pem certificate gc-nsxt-mgr-full.crt.pem key gc-nsxt-mgr.key.pem
4 To view the logging configuration, run the get logging-server command. For example,
nsx> get logging-servers
5 To clear the remote logging configuration, run the following command:
nsx> clear logging-servers
6 To configure remote logging on an ESXi host:
a) Run the following commands to configure syslog and send a test message:
esxcli network firewall ruleset set -r syslog -e true
esxcli system syslog config set --loghost=udp://<log server IP>:<port>
esxcli system syslog reload
esxcli system syslog mark -s "This is a test message"
b) You can run the following command to display the configuration:
esxcli system syslog config get
7 To configure remote logging on a KVM host:
a) Edit the file /etc/rsyslog.d/00-vmware-remote-logging.conf for your environment.
Create the file if it does not exist. The typical default permissions granting read permission to the group and others are sufficient.
b) Add the following line to the file:
*.* @<ip>:514;NsxLogfmt
c)Run the following command:
service rsyslog restart
Log Message IDs
In a log message, the message ID field identifies the type of message. You can use the messageid parameter in the set logging-server command to filter which log messages are sent to a logging server.
Message ID Examples
FABRIC Host node
Host preparation
Edge node
Transport zone
Transport node
Uplink profiles
Cluster profiles
Edge cluster
SWITCHING Logical switch
Logical switch ports
Switching profiles
switch security features
ROUTING Logical router
Logical router ports
Static routing
Dynamic routing
NAT
FIREWALL Firewall rules
Firewall rule sections
FIREWALL-PKTLOG Firewall connection logs
Firewall packet logs
GROUPING IP sets
Mac sets
NSGroups
NSServices
NSService groups
VNI Pool
IP Pool
DHCP DHCP relay
SYSTEM Appliance management (remote syslog, ntp, etc)
Cluster management
Trust management
Licensing
User and roles
Task management
Install
Upgrade (NSX Manager, NSX Edge and host-packages upgrades )
Realization
Tags
MONITORING SNMP
Port connection
Traceflow
- All other log messages.
Troubleshooting Syslog Issues
If logs are not received by the remote log server, perform the following steps.
1. Verify the remote log server's IP address.
2. Verify that the level parameter is configured correctly.
3. Verify that the facility parameter is configured correctly.
4. f the protocol is TLS, set the protocol to UDP to see if there is a certificate mismatch.
5. If the protocol is TLS, verify that port 6514 is open on both ends.
6. Remove the message ID filter and see if logs are received by the server.
7. Restart the rsyslog service with the command restart service rsyslogd.
Find the SSH Fingerprint of a Remote Server
Some tasks that involve communication with a remote server require that you provide the SSH fingerprint for the remote server. The SSH fingerprint is derived from a host key on the remote server.
To connect using SSH, the NSX Manager and the remote server must have a host key type in common.
NSX Manager supports the ECDSA (256 bit) key. The default location of this key is
/etc/ssh/ssh_host_ecdsa_key.pub.
## Procedure
1 Log in to the remote server as root.
2 Locate the ECDSA (256 bit) key. The default location of the key is
/etc/ssh/ssh_host_ecdsa_key.pub.
$ ls -al /etc/ssh/*pub
-rw-r--r-- 1 root root 93 Apr 8 18:10 ssh_host_ecdsa_key.pub
-rw-r--r-- 1 root root 393 Apr 8 18:10 ssh_host_rsa_key.pub
3 Get the fingerprint of the key.
ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub | awk '{print $2}'
Using NSX Cloud
NSX Cloud enables you to manage and secure your public cloud inventory using NSX-T Data Center.
VPCs or VNets
Navigate to Clouds -> <your public cloud> -> VPCs orVNets to view the VPCs or VNets in your public cloud account or subscription.
CSM Icons
CSM displays the state and health of your public cloud constructs using descriptive icons.
Join CSM with NSX Manager
You must connect the CSM appliance with NSX Manager to allow these components to communicate with each other.
NSX Maintenance Mode
If you want to avoid vMotion of VMs to a transport node that is not functional, place that transport node in NSX Maintenance Mode.
To put a transport node in NSX Maintenance Mode, select the node, click Actions → NSX Maintenance Mode.
When you put a host in NSX Maintenance Mode, the transport node cannot participate in networking. Also, VMs running on other transport nodes that have N-VDS or vSphere Distributed Switch as the host switch cannot be vMotioned to this transport node. In addition, logical network cannot be configured on ESXi or KVM hosts.
Scenarios to put the transport node in NSX Maintenance Mode:
- A transport node is not functional.
- If a host has hardware or software issues that are unrelated to NSX-T, but you want to retain the node and its configurations in NSX-T, place the host in NSX Maintenance Mode.
- A transport node is automatically put in NSX Maintenance Mode when an upgrade on that transport node fails.
Note: Any transport node put in the NSX Maintenance Mode is not upgraded.
How to install/configure new NSX-T transport node
1. Login to NSX-T management console
2. Access Systems -> Fabric -> Nodes -> Host Transport Nodes
3. From the Managed by drop down menu, select vCenter server (fqdn)
4. Navigate to the new ESXi host and select an ESXi host
5. Click Configure NSX, to start NSX installation
a. Name # It will automatically populate with ESXi host fqdn
6. Configure NSX
a. Type
i. N-VDS
ii. VDS # For NSX-T version, recommend to use "VDS"
b. Mode
i. Standard # Select standard
ii. ENS Interrupt
iii. Enhanced Datapath
Note:
Enhanced Datapath: Is the mode that provides accelerated networking performance.
This mode requires nodes to use VMXNET3 vNIC enabled network cards.
It is not supported on KVM, NSX Edge nodes and Public Gateways.
The supported hypervisor is ESXi.
It is recommended to run ESXi v6.7 U2 and later versions.
c. Name Enter a name for the N-VDS switch
# Enter name, such as N-VDS-VLAN-PDC-Servers
# Ensure NO space, or "_"
d. Transport Zone # select the configured transport zone
e. NIOC Profile # select default "nsx-default-nioc-hostswitch-profile
f. Unplink Profile # Select configured or default uplink profile
g. Teaming Policy Uplink Mapping
# Enter and select the uplink and physical NICs
Uplinks Physical NICS
--------------------------------
uplink-1 vmnic4
uplink-2 vmnic5 # It teaming has been configured in the switch
h. PNIC only Migration
# toggle to choose PNIC only migration if required
# Keep default "Disable"
Note:
To migrate a management network NIC, configure its associated VMkernel network mapping and keep PNIC only Migration disabled.
If you only migrate the management NIC, the host loses connectivity.
i. Network Mappings for Install
Add Mappings # Migrate vmkernel to N-VDS during installation
Note:
To migrate VMkernels to N-VDS switch during installation, map VMkernels to an existing logical switch.
The NSX Manager migrates the VMkernel to the mapped logical switch on N-VDS.
j. Network Mappings for Uninstall
Add Mappings
7. Click Finish
How to migrate vmkernel adapters to N-VDS method 2
Beside using configure NSX/Install NSX and specify the network mappings to migrate vmk0, vmkx to be migrated from standard switch or distributed switch to N-VDS. It can be migrated from Actions -> Migrate ESX VMkernel and Physical Adapters when selecting the ESX transport node.
# Prerequisite
The ESX host has been successfully installed with NSX.
Important: Only migrate one vmkernel adapter at a time.
# Process
1. Login to NSX-T management console
2. Access Systems -> Fabric -> Nodes -> Host Transport Nodes
3. From the Managed by drop down menu, select vCenter server (fqdn)
4. Navigate to the new ESXi host and select an ESXi host
5. From menu option, select ACTIONS -> Migrate NSX VMkernel and Physical Adapters
6. Direction
a. Migrate to Logical Switches # Select logical switches
b. Migrate to Port Groups
7. Select N-VDS # select the required N-VDS from the drop down menu
8. Select VMkernel Adapters to Migrate
a. Click + ADD
b. Under VMkernel Adapter, type vmk0 # migrate managment vmkernel adapter
c. Under Logical Switch, select the pre-configured N-VDS logical switch
9. Under Edit Physical Adapters In the N-VDS, click + ADD # Add the vmnic
a. Physical Adapter
vmnic(x)
# type and select the available vmnic as configured when install NSX on the transport node
b. Uplink Name
uplink-1
# type and select the uplink-(x) as configured when install NSX on the transport node
10. click SAVE
Important:
If the following error appears, try the following process
a. The uplink name is not found in the active or standby list...
Solution: Try again, and select the correct uplink
b. You cannot edit Node/xxxxxxx it is already modified through another operation and its revision number has changed.
To successfully edit the object, cancel this operation, refresh the object and retry the edit operation (Error code: 604)
Solution: Press F5 on the NSX management console to refresh the page and the objects, then try again.
How to manually remvoing faulty NSX-T Transport Nodes
This should be your last resort option if the regular procedures to remove ESXi based Transport Nodes from the NSX Manager do not work anymore.
Important:
If possible put the node to be removed in “Maintenance Mode”, move the VM’s and reboot after performing one of the solutions.
There are 3 methods depends on the transport node is reachable from NSX manager
- Using NSX Manager
- Using NSXCLI
- Using ESXi native tools
If NSX Intelligence is also deployed on the host, uninstallation of NSX-T Data Center will fail because all transport nodes become part of a default network security group. To successfully uninstall NSX-T Data Center, you also need to select the Force Delete option before proceeding with uninstallation.
Remove NSX from the transport node Using NSX Manager
1. From NSX Manager -> System -> Fabric -> Ndoes -> Host Transport Nodes menu
2. From Managed by drop down, select <vCenter-FQDN>
3. Select the required transport node, then click REMOVE NSX
Note:
when the node cannot be recovered anymore and / or is not reachable over the management interface.
Selecting “Force Delete” only, it will forcefully remove the Transport Node configuration in NSX Manager.
Transport Node Profiles
When a “Transport Node Profile” is attached to the cluster the faulty node resides in, the “Remove NSX” option is not available for that specific node, but only for the whole cluster. In this case use the “Detach Transport Node Profile” option in the “Actions” menu. When detaching the “Transport Node Profile” from a cluster it has no impact on data plane traffic within the cluster.
Remove NSX from transport node using NSXCLI
If all the NSX-T modules are still present on the node, this method should be your preferred one over using the ESXi native tools.
# On NSX Manager - get the thumbprint
manager> get certificate api thumbprint
# On Transport Node
1. Detach the Transport Node from NSX Manager if possible
[root@esxi-01:~] nsxcli
host> detach management-plane <MANAGER> username <ADMIN-USER> password <ADMIN-PASSWORD> thumbprint <MANAGER-THUMBPRINT
2. Remove NSX-T filters - need to run with "-Override" switch
[root@esxi-01:~] vsipioctl clearallfilters -Override
3. Stop the netcpa service (NSX-T 2.x)
[root@esxi-01:~] /etc/init.d/netcpad stop
4. Finally, remove all NSX-T VIBs
[root@esxi-01:~] nsxcli
esxi-01> del nsx # This will remove all NSX-T VIBs
Important:
a. The host must be in maintenance mode.
b. All resources attached to NSXPGs must be moved out.
When prompt
Are you sure you want to remove NSX-T on this host? (yes/no) yes # Type yes
Remove NSX from transport node using ESXCLI
Only when removing from NSX GUI and NSXCLI is not possbile due to whatever reason, a manual cleanup could be needed.
The config that is normally removed by NSX Manager or NSXCLI (besides the VIBs) is:
a. The NSX VMkernel ports (vxlan and hyperbus)
b. The NSX network IO filters
c. The NSX IP stacks (vxlan and hyperbus)
How to do manual cleanup
1. List the kernel ports
esxcli network ip interface list # verify the output configuration of vmk10, vmk50
2. Remove the NSX VMkernel ports
esxcli network ip interface remove --interface-name=vmk10
esxcli network ip interface remove --interface-name=vmk50
3. Now list the netstacks
esxcli network ip netstack list # It will lists vxlan, hyperbus
4. Remove the NSX related netstacks
esxcli network ip netstack remove --netstack=vxlan
esxcli network ip netstack remove --netstack=hyperbus
5. Verify if the N-VDS still exists, If so, the it has impact during the removal step of the actual VIBs
esxcfg-vswitch -l # verify N-VDS exists
6. List the install NSX VIBs
esxcli software vib list | grep -E 'nsx|vsipfwlib' # This will be used when uninstall VIBs
Now remove the VIBs in the correct order from the node. The last VIB to be removed, “nsx-esx-datapath” cannot be removed if the N-VDS is still present on the node. Only for that VIB the “--no-live-install” switch must be added. Run the command for every VIB to be removed.
# Finally, run the command for every VIB to be removed
esxcli software vib remove -n <vib name> # The VIB name is from step 6 above
Note:
esxcli software vib remove -n nsx-esx-datapath --no-live-install
If "--no-live-install" is omited,
an error is thrown if a "opaque" portgroup connected to a N-VDS is present on the node
VMware process
Prerequisites
- If there are VMkernel adapters on the host that must be migrated to another switch during uninstallation, ensure that the network uninstall mapping is configured. See Verify Host Network Mappings for Uninstall.
- n vCenter Server, put the hosts in maintenance mode and power off VMs running on the hosts if you want to migrate VMkernel adapters during uninstallation.
# Procedure
1. From a browser, log in with admin privileges to an NSX Manager at https://<nsx-manager-ip-address>.
2. Select System > Fabric > Nodes > Host Transport Nodes.
3. From the Managed by drop-down menu, select the vCenter Server.
4. Select the cluster you want to uninstall, and select Remove NSX.
Note:
If NSX Intelligence is also deployed on the host,
uninstallation of NSX-T Data Center will fail because all transport nodes become part of a default network security group.
To successfully uninstall NSX-T Data Center, you also need to select the Force Delete option before proceeding with uninstallation.
5. Verify that the NSX-T Data Center software is removed from the host.
a. Log into the host's command-line interface as root.
b. Run this command to check for NSX-T Data Center VIBs
esxcli software vib list | grep -E 'nsx|vsipfwlib'
6. (Host on an N-VDS switch) If the host goes into a failed state and NSX-T Data Center VIBs cannot be removed,
then run the command to remove NSX-T Data Center from the host.
del nsx
a. Before running the del nsx command, put the ESXi host in maintenance mode.
The vCenter Server does not allow the host to be put in maintenance mode unless all running VMs on the host are in powered off state or moved to a different host.
b. Log in to the ESXi CLI terminal, run
nsxcli -c del nsx.
c. Read the warning message. Enter Yes if you want to go ahead with NSX-T Data Center uninstallation.
d. Verify that the existing VMkernel and physical NICs on the N-VDS switch are migrated to a new vSwitch.
If there are more than one N-VDS switches on the host, each N-VDS switch is migrated to a separate vSwitch.
e. In NSX Manager, if a transport node profile (TNP) is attached to the cluster, detach the profile.
f. Select each host and click Remove NSX.
g. In the popup window, select Force Delete and begin uninstallation.
h. On the ESXi host, verify that system message displayed is
Terminated
Note: This message indicates that NSX-T Data Center is completely removed from the host.
7. If the host goes into failed state and NSX-T Data Center VIBs cannot be removed, then run the del nsx command to remove NSX from the host.
a. Before running the del nsx command, put the ESXi host in maintenance mode.
The vCenter Server does not allow the host to be put in maintenance mode ,
unless all running VMs on the host are in powered off state or moved to a different host.
b. If there are VMkernel adapters on NSX port groups on the VDS switch,
you must manually migrate or remove vmks from NSX port group to DV port groups on the VDS switch.
If there are any vmks available on the NSX port groups, del nsx command execution fails.
c. Log in to the ESXi CLI terminal, run
nsxcli -c del nsx
d. Read the warning message. Enter Yes if you want to go ahead with NSX-T Data Center uninstallation.
e. In the NSX Manager UI, if a Transport Node Profile is attached to the cluster, detach the profile.
f. Select each host and click Remove NSX.
g. In the popup window, select Force Delete and begin uninstallation.
h. On the ESXi host, verify that system message displayed is Terminated.
https://kontainers.in/2019/09/03/how-to-cleanup-an-orphaned-esxi-host-that-is-nsx-t-prepared/
https://patrik.kernstock.net/2020/07/quick-tip-nsx-t-3-0-removing-vibs-manually-from-esxi-host/
Reference sites
VMware VCDX
https://cloudmaniac.net/about/
Open Source Router and Firewall -vyos
The universal router and open source software https://vyos.io/