This is an old revision of the document!
NX-OS
Nexus 5620 and 2k FEX
CLI commands ; Models ; Troubleshooting
; VPC
; VDC
; User roles
; VLANs
; VXLANs`
; Fabric Path
; OTV
; FC
; CoS (Cisco Fabric Services)
; MDS devices
MODELS
- FEX support, N9K-C93360YC-FX2
- EOL: 5K (56128P); 5696Q (vertical slots)
- 2K (aka FEX or Fabric Extensor)
- 7K (Only >7k support MPLS)
- 9K : 9500 platform modular, 9300 platform fixed-configuration.
- 9300 is EOLed
- N9K-C93180-EX/FX (25G generation): SoC switch (1U). Note 180 comes from 1.8Tbps line rate
- 9336C-FX2-E (100G generation) 7.2 Tbps
- N9K-C93240-FX2
- 9504 : Now is not strictly needed for ACI, we can use for general purpose.
- 9516
- UCS (rack of blades)
- CSR 1000V
More accessible: C200 (this is just a Cisco server) and run Nexus 1000v, VSM
ARCHITECTURE TERMS
- UPC: unified port controllers (“behind the ports”). Data plane. unified crossbar fabric (UCF) ross-connects the UPCs
- SUP
- SFM: Switch fabric mode determines the speed of a single fabric link between UPC and Fabric.
- PFC: priority flow control
LINE CARDS
F2 line cards in nx7k have only 16k mac addresses. Nerd comment from ipspace. Apparently need to install the same thing in multiple places. In order to identify the vm flows in the fex cards we can use the vm fex technology
I/O Modules
M-Series modules: old ones. Don’t support FEX or LISP or FCoE or PTP (see lisp definition in SDN document)
F-Series modules : New ones, they support everything
Mode F (F port) is specific for storage ; mode NP (F is server NP is external interfaces)
For the nexus 7k There are 2 classes of I/O Modules: M-Series and F-Series. This is a list of the modules and their capabilities. 7600 Series supervisors and linecards: These are derived from 6500 but, are they applicable to fex?
Interface naming: three numbers, as when using a fabric extender, quite similar to juniper
Fex/slot/interface
FEX (Fabric Extender Technology): See this link. Encapsulation mechanism to transport frames from the FEX to the controlling bridge. Remember that the 2k doesn’t forward but the main 5k instead:
sh module fex ← To see the FEX modules (model and status) show inventory switchport mode fex-fabric
Under the command definition: fex associate 101 To check the FEX:
show interface fex-fabric
On the output, we can see the ‘Fex uplink’ which is meant to be the link between the 5k/7k and the 2k FEX
Take the name of the FEX (e.g.: 100), and do :
show fex 100 detail
Topologies:
ToR: less cable but more difficult to dimension/provision the switches
EoR: more cable needed but we can always use virtual chassis with members on each of the racks
Cisco UFT (Unified Fabric Technologies):
CHASSIS TECHNOLOGIES
See this link
show environment [ all | last | leds | location { all | node-id } | table | temperatures | voltages ] [node-id]
ingress fabric, crosswire fabric, and egress fabric chassis fully loaded with fabric modules Nexus service modules – ASA, ACE, and NAM VXLAN termination
vPC (Virtual Port Channel)
Configuration example can be founf in this blogpost
Similar to Juniper MC-LAG or Arista MLAG.
Another useful analogy is Catalyst VSS is like juniper VC (remember VSS cannot be used in Nexus)
Devices stay as separate entities, therefore managed separately and with different control planes. vPC is just a technology to present a unified LAG to other devices
Use cases: one fex with two 5k both acting as one (vpc) ;v one 5k connected to two 7k
- Tcam link ~vPC peer link
- 2 x 5020 Nexus switches and 12 x 2248s
* We need to inform this link is in the vpc * Remember the fex-associate keyword is in the normal interface configuration * The configuration is analogous to the one above
show port-channel summary
vpc related feature:
feature vpc feature lacp vpc domain 1 ← this to match in both devices peer-keepalive ← for the vpc peer link vpc peer-linkW show vpc brief ← show us if peer adjacency is ok. Note that we convey many active vlans in the vpc link
When configuring the port channels:
int et16 channel-group 16 int po 16 vpc ← This is a bit different from the classical port channels as we need to inform this link is in the vpc
We can also apply vpc to the fabric extenders
int et16 channel-group 16 int po 16
VDC
VDC(virtual domain contexts)
There is a default virtual context.
Similar to contexts in ASA devices
virtual switches. Isolated inside the same (chassis). Different kernels and failure domains. Only a few common resources as ntp.
By default, CPU is equally shared among the VDCs. However we can use priority to control the process allocation to a vdc
Commands (see link)
allocate interface ethernet
limit resource …. switchto vdc
switch# configure terminal switch(config)# vdc MyVDC \\
Note: Creating VDC, one moment please …
switch(config-vdc)# switch(config-vdc)# allocate interface ethernet 2/11-1
Until here, we are not inside the vdc, to switch to it, we must use the command:
switchto vdc MyVDC
And the prompt will become:
MyVDC# switchback to go back to the original vdc
QoS:
switch(config)# policy-map type network-qos Policy-buffer switch(config-pmap-nq)# class type network-qos class-default switch(config-pmap-nq-c)# queue-limit 400000 bytes switch(config-pmap-nq-c)# system qos switch(config-sys-qos)# service-policy type network-qos Policy-buffer
VLANs: To completely remove the danger of a native vlan being used, we can just tag the native vlan:
(cfg)# vlan dot1q tag native show interface trunk \\
show mac address-table
2-1005 normal vlans
1006-4094 extended vlans
sh vlan internal usage ← lists those one internally being used by the system (e.g.: for mcast)
VXLAN:
https://sites.google.com/site/amitsciscozone/home/data-center/vxlan
To provide Layer 2 extension beyond the layer 3 boundaries.normally in different pods in the same datacentre.
Not using spanning tree therefore better link utilisation (all is them). Leather 3 balancing technologies like ecmp
MACinUDP: External hacer is common one,just VXLAN tag in the outer l2 header. Then VXLAN header (with the VNID) + original L2 frame in the payload.
VXLAN uses VXLAN tunnel endpoint (VTEP) devices to map tenants’ end devices to VXLAN segments. One switch interface on the local LANs, the other is an IP interface to the transport IP network.
VXLAN uses stateless tunnels between VTEPs to transmit traffic of the overlay Layer 2 network through the Layer 3 transport network.
Uses existing Layer 2 mechanisms - flooding, and dynamic MAC address learning. IP multicast is used to reduce the flooding scope of the set of hosts that are participating in the VXLAN segment.
Each VXLAN segment, or VNID, is mapped to an IP multicast group in the transport IP network. Each VTEP device is independently configured and joins this multicast group as an IP host through the Internet Group Management Protocol (IGMP).
The IGMP joins trigger Protocol Independent Multicast (PIM) joins and signaling through the transport network for the particular multicast group. The multicast distribution tree for this group is built through the transport network based on the locations of participating VTEP.
MCAST limits Layer 2 flooding to those devices that have end systems participating in the same VXLAN segment.
Cisco ACI: See document called SDN
Fabric Path: ISIS works behind the scenes in L2 FabP. It uses a control plane which is used in FabP unicast, mcast and anycast.
TCN are used to pass the topology from the adjacent STP domains
Classic VLANs vs FP VLANs
show fabricpath isis route show fabricpath route
Encapsulated packets: ODA, OSA FP TAG (Etype, FTag, TTL)
OTV
Overlay Transport Virtualization (~ tunneling)
ISIS is also used.
Layer 2 over Layer 3
show otv adjacency show otv overlay vpn state (should be up) , control group (matches the one on the mcast group) show otv route show otv isis internal event-history adj
FHRP (First-hop redundancy protocol) filtering. Implementing ACL in the edge devices: VACL, OTV MAC route filtering
User Roles
If we create a user and assign password without assigning a role, it goes straight to admin role
lg2prdswi-5k-access1:show role name ? network-admin System configured role network-operator System configured role san-admin System configured role vdc-admin System configured role vdc-operator System configured role
To check my own role we use where
Functionalities can be enable/disabled with:
feature feature telnet feature Interface-vlan show interface xxx capabilities
- Different users can be assigned different roles
PROCESS RECOVERY Each process (e. g. :hsrp) had checkpoints or periodically writes its state to a file called PSS
CONFIGURATION CHECKPOINT AND CONFIGURATION ROLLBACK
Similar to junos but we also need to specify a checkpoint where we want to go back to.
We can have rollback in the default vdc and also in specific vdcs
! Create Configuration Checkpoint n7000# checkpoint before-remove-vlans description remove vlan 10 and 20 ......................Done ! Now modify the Running-Configuration: n7000# config t n7000(config)# no vlan 10,20 n7000(config)# exit Perform the Rollback Procedure n7000# rollback running-config checkpoint remove-vlans verbose
Management over vrf mgmt 0
EVPN (OFFICIAL):
- Is as evolution from OTV (aee above). We have EVPN (standard) wich uses BGP
- VXLAN EVPN for the dc: https://www.youtube.com/watch?v=O8wU1qNlsyI
NEXUS UPGRADE
To upgrade it, we need two files:
kickstart and system
This is why we need a kickstart file!
Things to check before upgrade
show version show interface brief # to see eth and fc interfaces show vpc brief show fex # If any show fcns database # This is like the MACssion show flogi database show interface description # better to view UCS, fc .. show vpc orphan-ports copy startup-config bootflash:///config.cgf # show tech-support
Verify installer compatibility:
show install all impact kickstart bootflash:n5000-uk9-kickstart.4.2.1.N1.1a.bin # Alo Check integrity with md5sum
Install:
install all kickstart bootflash:n5000-uk9-kickstart.7.0.7.N1.1.bin system 5000-uk9.7.0.7.N1.1.bin
Things to check after upgrade:
- show interface status
- show vpc
- show fex
- show fcns database
- shpw flogi database
- show zoneset active
- show vpc consistency-parameters global
If we lose the fc channels, we can convert eth back to fc with:
slot 1 port 17-32 type fc
REMEMBER to SAVE and RELOAD after this!
check ; copy bootflash ; copy bin
Nexus 5k. See link
Nexus 7k. See link
Nexus 9k See link
ISSU UPGRADE:
! Copy images to flash show fle <image> md5sum ! verify md5 install all kickstartbootflash:n7000-s1-kickstart.6.2.12.bin system bootflash:n7000-s1-dk9.6.2.12.bin
LINE CARD UPGRADE:
out-of-service module <module-number> ! shut down the card ! Double check line card support matrix
FC - Fibre Channel F ports, M ports MDS devices VSAN
Zoning
Collection of ports that can communicate between them over the SAN. Recommended to define a zone per initiator and target, and deploy multiple small zones, rather than having larger zones defined as they consume more resources
Soft zoning: (Software) Name server (FCNSto allow devices connect to FC) to reply with all devices registered to that zone.
Hard zoning: (Hardware): Access through access list ACL
Zone membership
Concept of VSAN
How to setup
sh int brief OR show interface fc1/5
Besides the classical up, down we have:
- init: initializing, the interface can be stuck in this state
- inactive: vlan suspended/deleted
- isolated: genrally due to a parameter mismatch
- link failure: phy down
show flogi database vsan
show fcns database vsan {~dns but for the interface names. For a switch to join the fabric)
show fc-timer
NPiv as a technology allows to assign several End point id / fc ids to a host port
E-port troubleshooting
MDS -FC→Cisco port analyzer (take FC traffic and encapsulates the traffic in ethernet to be sent to the wireshark machine)
FCoE:
Jumbo frames Map FC ids to MAC Special (unique) vdc FCForwarders: Encap and decap of FC traffic for FCoE FCoE 0x8906 STP type: MSTP
CoS (Cisco Fabric Services) Basically a way to propagates and synchronise the configurations show cfs application/peers/lock
See also::
Cisco ACI: Cisco’s SDN
Cisco ACS: Access Control Server
Cisco UCS: Unified Computing System. Cloud director. Rack of blades in the UCS (B2). Virtual data centres technology. To reduce the number of boxes required. Improve air flow, cabling and power consumption. One cab ~4.5kw)
Cisco CRS (Cisco Carrier Routing System
Cisco ASR (Aggregation Service Router
mds: int fc1/12 switchport mode sd switchport speed 1000
Troubleshooting:
show interface status show interface transceiver show system system/lacp internal
Nexus remarkable process names:
- ethpc–ethernet port client: responsible for talking to the mac and phy
- ethpm–ethernetport manager: responsible for translating between configuration and ethpc. ethpcwould inform ethpmthat link is up, and then ethpmwill proceed to give instructions on what the configuration is for the port
- eort-channel –port-channeling process responsible for aggregating physical links
lacp–802.3ad standard for aggregating links
Message and Transaction Service(MTS) to communicate between processes. To see if there are messages hanging in the inter process communication queue:
show system internal mtsbuffers
Is a modular OS. SUPERVISOR MODULE (>7k it live in a standalone supervisor card): Check its status CMP Ethernet port: For sup management only
# show module Mod Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 1 0 Supervisor module-1X N7K-SUP1 active * 2 0 Supervisor module-1X N7K-SUP1 ha-standby # system switchover ! to failback to the standby supervisor
show logging onboard diagnostic bootup level ? GOLD diagnostics locator-led : to make the blue beacon flash in the card, module (useful for the remote hands)
show debug logfile myfile show ip eigrp internal event history
Configure session (the equivalent to commit in junos):
verify commit
Ethanalyzer Cisco NX-OS Software Built-In Packet Capture Utility ethanalyzer local interface inband limit-captured-frames 5 ethanalyzer local interface inband write bootflash:xxx.pcap
pong : uses ptp to measure latency
NAPALM (python)
https://github.com/jotasantos/ansible-eveng/tree/main/mgmt-dhcp
! in nx
feature scp-server
feature nxapi
! in nms
driver_nxos = get_network_driver('nxos')
device_nxos = driver_nxos('10.8.11.28', 'admin', 'admin')
device_nxos.open()
device_nxos
—-
ANSIBLE
TODO
