**__NX-OS__** ** Nexus 5620 and 2k FEX ** CLI commands ; Models ; Troubleshooting ; VPC ; VDC ; User roles ; VLANs ; VXLANs` ; Fabric Path ; OTV ; FC ; CoS (Cisco Fabric Services) ; MDS devices \\ \\ **__MODELS__** * FEX support, N9K-C93360YC-FX2 * EOL: 5K (56128P); 5696Q (vertical slots) * 2K (aka FEX or Fabric Extensor) * 7K (Only >7k support MPLS) * [[https://www.cisco.com/c/en/us/products/switches/nexus-9000-series-switches/datasheet-listing.html|9K]] : 9500 platform modular, 9300 platform fixed-configuration. * 9300 is EOLed * N9K-C93180-EX/FX (25G generation): SoC switch (1U). Note 180 comes from 1.8Tbps line rate * 9336C-FX2-E (100G generation) 7.2 Tbps * N9K-C93240-FX2 * 9504 : Now is not strictly needed for ACI, we can use for general purpose. * 9516 (x21 RUs ) * UCS (rack of blades) * CSR 1000V Tables to check in the specs sheet:\\ **MAC (dynamic Mac learning) ; ARP-ND-ICMP ; Forwarding ; RIB (unicast/multicast)**. Check these [[https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/92x/scalability/guide_923/b_Cisco_Nexus_9000_Series_NX-OS_Verified_Scalability_Guide_923.html|verified-limits]]\\ **Layer-2** , Layer-3(host routes + mcast) , LPM(longest prefix match, for ipv4 and ipv6 Entries) \\ \\ More accessible: C200 (this is just a Cisco server) and run Nexus 1000v, VSM \\ **__ARCHITECTURE TERMS__** * UPC: unified port controllers ("behind the ports"). Data plane. unified crossbar fabric (UCF) cross-connects the UPCs * SUP * SFM: Switch fabric mode determines the speed of a single fabric link between UPC and Fabric. * PFC: priority flow control \\ __**LINE CARDS**__\\ __F2 line cards__ in nx7k have only 16k mac addresses. Nerd comment from ipspace. Apparently need to install the same thing in multiple places. In order to identify the vm flows in the fex cards we can use the [[http://www.cisco.com/c/en/us/solutions/data-center-virtualization/data-center-virtual-machine-fabric-extender-vm-fex/index.html|vm fex technology]] __I/O Modules__\\ __M-Series__ modules: old ones. Don’t support FEX or LISP or FCoE or PTP (see lisp definition in SDN document) \\ [[http://www.cisco.com/c/en/us/products/switches/nexus-7000-series-switches/models-comparison.html#~tab-c|F-Series]] modules : New ones, they support everything __Mode F (F port)__ is specific for storage ; mode NP (F is server NP is external interfaces) For the nexus 7k There are 2 classes of I/O Modules: M-Series and F-Series. [[https://jeremywaldrop.wordpress.com/2011/06/30/cisco-nexus-7000-io-module-cheat-sheet/|This]] is a list of the modules and their capabilities. 7600 Series supervisors and linecards: These are derived from 6500 but, are they applicable to fex? Interface naming: three numbers, as when using a fabric extender, quite similar to juniper Fex/slot/interface \\ FEX (Fabric Extender Technology): See this [[http://routing-bits.com/2012/05/16/what-is-a-fabric-extender/|link]]. Encapsulation mechanism to transport frames from the FEX to the controlling bridge. Remember that the 2k doesn’t forward but the main 5k instead: sh module fex ← To see the FEX modules (model and status) show inventory switchport mode fex-fabric Under the command definition: fex associate 101 To check the FEX: show interface fex-fabric On the output, we can see the ‘Fex uplink’ which is meant to be the link between the 5k/7k and the 2k FEX Take the name of the FEX (e.g.: 100), and do : show fex 100 detail \\ **Topologies:** \\ ToR: less cable but more difficult to dimension/provision the switches EoR: more cable needed but we can always use virtual chassis with members on each of the racks \\ Cisco UFT (Unified Fabric Technologies): ---- MAINTENANCE:\\ Eg: for a line card replacement. graceful insertion and removal (GIR) [[https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/system_management/configuration/guide/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x/b_Cisco_Nexus_9000_Series_NX-OS_System_Management_Configuration_Guide_7x_chapter_011101.html|External Link]] NET-5587 snapshot create PRE-WORKS linecard-replacement conf system mode maintenance ! works conf no system mode maintenance snapshot create POST-WORKS linecard-replacement ---- **__CHASSIS TECHNOLOGIES__**\\ See this [[http://www.cisco.com/c/en/us/td/docs/routers/crs/software/crs_r4-2/system_management/command/reference/b_sysman_cr42crs/b_sysman_cr42crs_chapter_01000.html|link]] show environment [ all | last | leds | location { all | node-id } | table | temperatures | voltages ] [node-id] ingress fabric, crosswire fabric, and egress fabric chassis fully loaded with fabric modules Nexus service modules – ASA, ACE, and NAM VXLAN termination ---- **__vPC (Virtual Port Channel)__**\\ Configuration example can be founf in this [[https://www.packetcoders.io/what-is-cisco-vpc-virtual-port-channel/|blogpost]] **Similar to Juniper MC-LAG or Arista MLAG.** \\ Another useful analogy is Catalyst VSS is like juniper VC (remember VSS cannot be used in Nexus) \\ Devices stay as separate entities, therefore managed **separately** and with different **control planes**. vPC is just a technology to present a unified LAG to other devices\\ Use cases: one fex with two 5k both acting as one (vpc) ;v one 5k connected to two 7k Read: [[http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/configuration_guide_c07-543563.htm]]l \\ * Tcam link ~vPC peer link * 2 x 5020 Nexus switches and 12 x 2248s * Topology: [[http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/design_guide_c07-625857.pdf]]\\ * We need to inform this link is in the vpc * Remember the fex-associate keyword is in the normal interface configuration * The configuration is analogous to the one above show port-channel summary \\ vpc related //feature//: feature vpc feature lacp vpc domain 1 ← this to match in both devices peer-keepalive ← for the vpc peer link vpc peer-linkW show vpc brief ← show us if peer adjacency is ok. Note that we convey many active vlans in the vpc link \\ When configuring the port channels: int et16 channel-group 16 int po 16 vpc ← This is a bit different from the classical port channels as we need to inform this link is in the vpc We can also apply vpc to the fabric extenders int et16 channel-group 16 int po 16 ---- __**VDC**__ __VDC(virtual domain contexts)__ There is a default virtual context. Similar to contexts in ASA devices virtual switches. Isolated inside the same (chassis). Different kernels and failure domains. Only a few common resources as ntp. By default, CPU is equally shared among the VDCs. However we can use priority to control the process allocation to a vdc Commands (see [[http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/virtual_device_context/command/reference/vdc_commands.html|link]]) \\ [[http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/virtual_device_context/command/reference/vdc_commands.html#wp1029390|allocate interface ethernet]] \\ limit resource …. switchto vdc switch# configure terminal switch(config)# vdc MyVDC \\ Note: Creating VDC, one moment please ... switch(config-vdc)# switch(config-vdc)# allocate interface ethernet 2/11-1 Until here, we are not inside the vdc, to switch to it, we must use the command: switchto vdc MyVDC And the prompt will become: MyVDC# switchback to go back to the original vdc \\ **__QoS:__** switch(config)# policy-map type network-qos Policy-buffer switch(config-pmap-nq)# class type network-qos class-default switch(config-pmap-nq-c)# queue-limit 400000 bytes switch(config-pmap-nq-c)# system qos switch(config-sys-qos)# service-policy type network-qos Policy-buffer **__VLANs:__** To completely remove the danger of a native vlan being used, we can just tag the native vlan: (cfg)# vlan dot1q tag native show interface trunk \\ show mac address-table 2-1005 normal vlans \\ 1006-4094 __extended vlans__ \\ sh vlan internal usage ← lists those one internally being used by the system (e.g.: for mcast) \\ **__VXLAN__**: \\ [[https://sites.google.com/site/amitsciscozone/home/data-center/vxlan]] \\ To provide Layer 2 extension beyond the layer 3 boundaries.normally in different pods in the same datacentre.\\ Not using spanning tree therefore better link utilisation (all is them). Leather 3 balancing technologies like __ecmp__\\ MACinUDP: External hacer is common one,just VXLAN tag in the outer l2 header. Then VXLAN header (with the VNID) + original L2 frame in the payload. \\ VXLAN uses __VXLAN tunnel endpoint (VTEP)__ devices to map tenants’ end devices to VXLAN segments. One switch interface on the local LANs, the other is an IP interface to the transport IP network. \\ VXLAN uses stateless tunnels between VTEPs to transmit traffic of the overlay Layer 2 network through the Layer 3 transport network.\\ Uses existing Layer 2 mechanisms - flooding, and dynamic MAC address learning. IP multicast is used to reduce the flooding scope of the set of hosts that are participating in the VXLAN segment. \\ Each VXLAN segment, or VNID, is mapped to an IP multicast group in the transport IP network. Each VTEP device is independently configured and joins this multicast group as an IP host through the Internet Group Management Protocol (IGMP). \\ The IGMP joins trigger Protocol Independent Multicast (PIM) joins and signaling through the transport network for the particular multicast group. The multicast distribution tree for this group is built through the transport network based on the locations of participating VTEP. \\ MCAST limits Layer 2 flooding to those devices that have end systems participating in the same VXLAN segment. \\ Cisco ACI: See document called SDN \\ **__Fabric Path:__** **IS-IS** works behind the scenes in L2 FabP. It uses a control plane which is used in FabP unicast, mcast and anycast. TCN are used to pass the topology from the adjacent STP domains Classic VLANs vs FP VLANs show fabricpath isis route show fabricpath route Encapsulated packets: ODA, OSA FP TAG (Etype, FTag, TTL) **__OTV__** Overlay Transport Virtualization (~ tunneling) ISIS is also used. Layer 2 over Layer 3 \\ show otv adjacency show otv overlay vpn state (should be up) , control group (matches the one on the mcast group) show otv route show otv isis internal event-history adj \\ **FHRP** (First-hop redundancy protocol) filtering. Implementing ACL in the edge devices: VACL, OTV MAC route filtering \\ **__User Roles__** \\ If we create a user and assign password without assigning a role, it goes straight to admin role lg2prdswi-5k-access1:show role name ? network-admin System configured role network-operator System configured role san-admin System configured role vdc-admin System configured role vdc-operator System configured role To check my own role we use where \\ Functionalities can be enable/disabled with: feature feature telnet feature Interface-vlan show interface xxx capabilities - Different users can be assigned different **roles** **__PROCESS RECOVERY__** Each process (e. g. :hsrp) had checkpoints or periodically writes its state to a file called PSS ---- __CONFIGURATION CHECKPOINT AND CONFIGURATION ROLLBACK__ \\ Similar to junos but we also need to specify a checkpoint where we want to go back to.\\ We can have rollback in the default vdc and also in specific vdcs \\ ! Create Configuration Checkpoint n7000# checkpoint before-remove-vlans description remove vlan 10 and 20 ......................Done ! Now modify the Running-Configuration: n7000# config t n7000(config)# no vlan 10,20 n7000(config)# exit Perform the Rollback Procedure n7000# rollback running-config checkpoint remove-vlans verbose ---- **__Management over vrf mgmt 0__** \\ ---- EVPN (OFFICIAL):\\ * Is as evolution from OTV (aee above). We have EVPN (standard) wich uses BGP * VXLAN EVPN for the dc: [[https://www.youtube.com/watch?v=O8wU1qNlsyI]] * VNI (todo): see [[https://www.juniper.net/documentation/en_US/junos/topics/concept/evpn-vxlan-data-plane-encapsulation.html]] ---- __**NEXUS UPGRADE**__\\ To upgrade it, we need two files: kickstart and system This is [[https://learningnetwork.cisco.com/s/question/0D53i00000KspxJ/why-do-we-have-kickstart-image-for-nexus|why]] we need a kickstart file! \\ Things to check **before** upgrade show version show interface brief # to see eth and fc interfaces show vpc brief show fex # If any show fcns database # This is like the MACssion show flogi database show interface description # better to view UCS, fc .. show vpc orphan-ports copy startup-config bootflash:///config.cgf # show tech-support Verify installer compatibility: show install all impact kickstart bootflash:n5000-uk9-kickstart.4.2.1.N1.1a.bin # Alo Check integrity with md5sum \\ Install: install all kickstart bootflash:n5000-uk9-kickstart.7.0.7.N1.1.bin system 5000-uk9.7.0.7.N1.1.bin Things to check **after** upgrade: * show interface status * show vpc * show fex * show fcns database * shpw flogi database * show zoneset active * show vpc consistency-parameters global \\ If we lose the fc channels, we can convert eth back to fc with: slot 1 port 17-32 type fc REMEMBER to SAVE and RELOAD after this! check ; copy bootflash ; copy bin \\ Nexus 5k. See [[http://www.cisco.com/c/en/us/support/switches/nexus-5000-series-switches/products-installation-guides-list.html|link]] \\ Nexus 7k. See [[http://www.cisco.com/c/en/us/support/switches/nexus-7000-series-switches/products-installation-guides-list.html|link]] \\ Nexus 9k See [[http://www.cisco.com/c/en/us/support/switches/nexus-5000-series-switches/products-installation-guides-list.html|link]] \\ \\ ISSU UPGRADE:\\ ! Copy images to flash show fle md5sum ! verify md5 install all kickstartbootflash:n7000-s1-kickstart.6.2.12.bin system bootflash:n7000-s1-dk9.6.2.12.bin LINE CARD UPGRADE: out-of-service module ! shut down the card ! Double check line card support matrix ---- **See also:**:\\ Cisco ACI: Cisco’s SDN\\ Cisco ACS: Access Control Server\\ Cisco UCS: Unified Computing System. Cloud director. Rack of blades in the UCS (B2). Virtual data centres technology. To reduce the number of boxes required. Improve air flow, cabling and power consumption. One cab ~4.5kw)\\ Cisco CRS (Cisco Carrier Routing System\\ Cisco ASR (Aggregation Service Router\\ \\ mds: int fc1/12 switchport mode sd switchport speed 1000 ---- **Troubleshooting:** \\ show interface status show interface transceiver show system system/lacp internal Nexus remarkable process names: * ethpc–ethernet port client: responsible for talking to the mac and phy * ethpm–ethernetport manager: responsible for translating between configuration and ethpc. ethpcwould inform ethpmthat link is up, and then ethpmwill proceed to give instructions on what the configuration is for the port * eort-channel –port-channeling process responsible for aggregating physical links lacp–802.3ad standard for aggregating links \\ __Message and Transaction Service(MTS)__ to communicate between processes. To see if there are messages hanging in the inter process communication queue: show system internal mtsbuffers Is a modular OS. **SUPERVISOR MODULE** (>7k it live in a standalone supervisor card): Check its status CMP Ethernet port: For sup management only # show module Mod Ports Module-Type Model Status --- ----- ----------------------------------- ------------------ ---------- 1 0 Supervisor module-1X N7K-SUP1 active * 2 0 Supervisor module-1X N7K-SUP1 ha-standby # system switchover ! to failback to the standby supervisor \\ show logging onboard diagnostic bootup level ? GOLD diagnostics locator-led : to make the blue beacon flash in the card, module (useful for the remote hands) \\ show debug logfile myfile show ip eigrp internal event history Configure session (the equivalent to commit in junos): verify commit [[http://www.cisco.com/c/en/us/products/collateral/switches/nexus-3000-series-switches/white_paper_c11-673817.html|Ethanalyzer]] Cisco NX-OS Software Built-In Packet Capture Utility ethanalyzer local interface inband limit-captured-frames 5 ethanalyzer local interface inband write bootflash:xxx.pcap pong : uses ptp to measure latency \\ show forwarding route 10.50.200.0 ! instead of 'show ip cef' ---- **NAPALM (python)**\\ [[https://github.com/jotasantos/ansible-eveng/tree/main/mgmt-dhcp]] ! in nx feature scp-server feature nxapi ! in nms driver_nxos = get_network_driver('nxos') device_nxos = driver_nxos('10.8.11.28', 'admin', 'admin') device_nxos.open() device_nxos ---- **ANSIBLE**\\ TODO ---- **FC - Fibre Channel** F ports, M ports MDS devices **VSAN** [[https://supportforums.cisco.com/document/130761/san-zoning-guidelines-nexus|Zoning]]\\ Collection of ports that can communicate between them over the SAN. Recommended to define a zone per initiator and target, and deploy multiple small zones, rather than having larger zones defined as they consume more resources \\ \\ **Soft zoning:** (Software) Name server (FCNSto allow devices connect to FC) to reply with all devices registered to that zone. **Hard zoning:** (Hardware): Access through access list ACL \\ **Zone membership** \\ **Concept of VSAN** **How to setup** \\ {{ :network_stuff:cisco:vsanvszone_table.png?550|}} sh int brief OR show interface fc1/5 Besides the classical up, down we have: * init: initializing, the interface can be stuck in this state * inactive: vlan suspended/deleted * isolated: genrally due to a parameter mismatch * link failure: phy down show flogi database vsan show fcns database vsan {~dns but for the interface names. For a switch to join the fabric) show fc-timer **NPiv** as a technology allows to assign several End point id / fc ids to a host port E-port troubleshooting \\ MDS -FC->Cisco port analyzer (take FC traffic and encapsulates the traffic in ethernet to be sent to the wireshark machine) **FCoE:** Jumbo frames Map FC ids to MAC Special (unique) vdc FCForwarders: Encap and decap of FC traffic for FCoE FCoE 0x8906 STP type: MSTP **CoS (Cisco Fabric Services)** Basically a way to propagates and synchronise the configurations show cfs application/peers/lock ---- __IS-IS PROTOCOL__ \\ Sample configuration: interface loopback0 ip address 10.0.1.1 255.255.255.255 ip router isis ipv6 address FEC0::CCCC:1/128 ipv6 router isis ! interface et1/1 description Link to P router ip address 10.0.7.9 255.255.255.252 ip router isis ipv6 enable ipv6 router isis ! router isis net 49.0000.0000.cccc.0001.00 metric-style wide [transition] address-family ipv6 multi-topology [transition]