**__EX SWITCHES__**\\
My notes in googled : [[https://drive.google.com/open?id=1-mI2lDzbom1_NZXiDIQYGfo1QtH_G13a1GRIrnqwSLc|External Link]]\\
Password recovery procedure for EX switches: [[http://www.juniper.net/techpubs/en_US/junos15.1/topics/task/troubleshooting/ex-series-root-password-recovery.html|link1]] , [[http://kb.juniper.net/InfoCenter/index?page=content&id=KB14102|link2]]\\

\\
\\TFXPC0(vty)# show filter hw groups
For the mastership selection, remember higher wins [[https://www.juniper.net/documentation/en_US/junos/topics/concept/virtual-chassis-ex4200-master-switch-election.html|External Link]].  the members that are not selected as master or backup function as linecard members of the Virtual Chassis configuration. A switch with a mastership priority of 0 is always in the linecard role. The default value for mastership priority is 128. Normally we assign master and backup the Same Priority.
\\
succinctly: 1st is priority values, .... then lowest MAC address
  show virtual-chassis
To log to the virtual chassis member 
  request session member 2
This shows the VCP ports (back and front, used for signalling the vchasss). VCPs can be aggregated to form a LAG
  show virtual-chassis vc-port

To check the inserted SFPs:
  show version <-- This shows me the number of fpc (devices)
  show chassis pic fpc-slot 2 pic-slot 1

----
Example of md5 junos password Junos OS 12.3 and earlier.
  easy0n
  set system login user test authentication encrypted-password "$1$6Ub0uM5t$08QKpPT1ZO0GjwcVe6mTP1"

----
AGGREGATED INTERFACES 802.3ad (LAG)

To assign interface to an aggegate:
  set interfaces et-0/1/1 ether-options 802.3ad ae0
To see the members of a ae interface
  show interfaces ae0 extensive  # and check down in the section 'Bundle'


----
**IRB** - INTEGRATED ROUTING AND BRIDGING INTERFACES\\
See this [[https://www.juniper.net/documentation/en_US/junos/topics/concept/bridging-routed-vlan-interface.html|External Link]]

  set interfaces irb unit 0 family inet address 10.5.6.39/21
  set vlans Internalmock vlan-id 400
  set interfaces et-0/1/0 ether-options no-auto-negotiation
  set interfaces et-0/1/0 unit 0 family ethernet-switching interface-mode access
  set interfaces et-0/1/0 unit 0 family ethernet-switching vlan members Internalmock
  set vlans Internalmock l3-interface irb.0       # This binds the irb with the vlan (Internalmock)
{{ :network_stuff:juniper:diagram1.jpeg?600|}}


----
**ENABLE SFLOW**
  [edit protocols]
  +   sflow {
  +       polling-interval 20;
  +       sample-rate {
  +           ingress 2000;
  +           egress 2000;
  +       }
  +       collector 185.89.204.18 {
  +           udp-port 2055;
  +       }
  +       interfaces xe-0/0/0.0;
  +       interfaces xe-0/0/1.0;
  +       interfaces xe-0/0/2.0;
  +   }


  /var/tmp/sflowtool/sflowtool-3.22
  ss2# 
  ss2# sflowtool -p 2055 -t | tcpdump -r - -s0

[[https://blog.sflow.com/2011/12/sflowtool.html]]


----





ROUTED VLAN (OR **RVI**, Not to confuse with IRB):\\
To communicate outside the vlan realm (for those EX switches without ELS (//Enhanced// Layer 2 Software)) 
Create a layer 2 VLAN:
  set vlans <vlan-name> vlan-id <vlan-id> (1..4094)>
Create a logical layer 3 VLAN interface:
  set interfaces vlan unit <unit# (0..16385)> family inet address <ip address/mask>
Link the layer 2 VLAN to the layer 3 VLAN interface:
  set vlans <vlan-name> l3-interface vlan.<unit# mentioned above>
See: [https://kb.juniper.net/InfoCenter/index?page=content&id=KB11000]
\\
Switches normally don't accept untagged data in tagged port. If we want them to accept it we need to do something like this:
  set interface ae3 native-vlan-id 99
  

----


CONFIGURE MULTIPLE ENTITIES AT ONE (SIMILAR TO CISCO RANGE COMMAND):\\
  wildcard range set interfaces xe-[0,1]/0/[0,1] disable
  

----


**CHECK SFP/INTERFACE FLAPS**
To see light/laser level:
  show interfaces diagnostics optics
  show interfaces diagnostics optics xe-0/0/20:0 | except thresh | except Off    # for the qsfp-404-10g is enough with this (this shows everything)
  show chassis pic pic-slot 0 fpc-slot 1     # for more specific transceiver type


To see location of SFP:
  show chassis hardware
To find SFP information:
  show chassis pic fpc-slot 0 pic-slot 0
  
This is what needs to be seen in the logs when an interface flaps, result of the process mib2d generating a log:  \\
[[https://www.b00z.nl/blog/2016/04/juniper-srx-error-could-not-format-alternate-root-solution/|LINK]] 

----

UPGRADE JUNOS
- Standalone procedure:\\
  * Copy .tgz with winscp. If ssh is enabled, it will work.
Remember **before the upgrade**
  * Remember the upgrade **will fail if the local time is not correct (due to certificate validation)**. Either use ntp or set the time with:
  * Also Be Sure that we have a **backup of the running configuration**. In an interrupted upgrade, some config parts might be missing.
  * Be sure we have **CONSOLE ACCESS**
  * Copy output of this. Note than lacp bond interfaces in the linux end **can get in a down state after an upgrade even with NSSU**:

Checks1:

  request support information | no-more   # have this in a file case thorough checks are needed
  show interface terse | no-more # REVIEW ALL BONDS STAY UP AFTER UPGRADE
  show virtual-chassis | no-more
  show chassis hardware | no-more
  show ethernet-switching table | no-more
  show system alarms  # request system configuration rescue save # if rescue configuration not set yet

Checks2:
  * For multihome systems, be sure the member that will stay up is **sending and receiving** the correct routes via BGP  

  show route advertising-protocol bgp 206.126.236.37
  show route 0.0.0.0 detail
  
Below some helpful commands: 
  set date YYYYMMDDHHMM.ss
  show system uptime
  show task replication
  request session member <member-id>
  show system switchover # from the backup re
  request chassis routing-engine master switch    # case we need to force switchover , run this in the fpc that WANTS to become master (backup)

\\
Now the proper upgrade:\\
\\
And here the list of EEOL: [[http://www.juniper.net/support/eol/junos.html|External Link]]
\\
SINGLE DEVICE UPGRADE:
  request system software add /var/tmp/jinstall-host-qfx-5-17.2R1.n-signed.tgz force-host
  # Then reboot or, if we want to rollback 'request system software rollback'
\\
"You can upgrade or downgrade to the EEOL release that occurs directly before or after the currently installed EEOL release, or to two EEOL releases before or after. For example, Junos OS Releases 10.0, 10.4, and 11.4 are EEOL releases. You can upgrade from Junos OS Release 10.0 to Release 10.4 or even from Junos OS Release 10.0 to Release 11.4.". See [[http://www.juniper.net/support/eol/junos.html|External Link]]
\\  

----
**UPGRADE EX MIXED/NOT MIXED VC WITH NSSU**

BEFORE STARTING, DO THIS:
  # 1st thing verify console access is OK
  # Check rancid config backup is up to date
  # log the output of your ssh session and collect the info from the commands below:
  set cli timestamp
  delete system services ssh root-login deny # and copy the image
  show system alarms  # request system configuration rescue save
  request support information | no-more
  show chassis hardware | no-more
  show ethernet-switching table | no-more
  show interface terse | except down | no-more
  show interfaces descriptions | no-more
  show virtual-chassis | no-more
  show virtual-chassis vc-port 
  file checksum md5 /var/tmp/jinstall-host-ex-4600-16.1R6-S6-signed.tgz
  


CHECK THIS:
  show system uptime   # date time needs to be correct
  show task replication     # needs to be enabled
  ONLY IF THIS IS A TWO MEMBER VIRTUAL CHASSIS!!
  show configuration | display set | match split
  set virtual-chassis no-split-detection # << ONLY IF TWO MEMBER VC!:
  show configuration | display set | match nonstop
  set protocols layer2-control nonstop-bridging
  set routing-options nonstop-routing
  show configuration | display set | match switchover
  set chassis redundancy graceful-switchover
  show virtual-chassis vc-port    # verify all members are connected to each other (daily chain). For each of the fpcx sections, check Status (Up) and neighbour (needs be two and contiguous). This is important, otherwise upgrade will fail and VC with break 
  

UPGRADE COMMAND for mixed VC:

  request system software nonstop-upgrade set [/var/tmp/package-name.tgz /var/tmp/package-name.tgz] force-host # MIXED virtual 
  chassis # this will start the whole process, no manual reboot
UPGRADE COMMAND for NON-MIXED VC:\\
To see what hapens during the nssu see this [[https://www.juniper.net/documentation/en_US/junos/topics/concept/nssu-qfx-series.html|External Link]]
    request system software nonstop-upgrade  /var/tmp/package-name.tgz force-host   # this will start the whole process, no manual reboot
AFTER: 

  set system services ssh root-login deny

Useful commands
    set date YYYYMMDDHHMM.ss
    request session member <member-id>
    request chassis routing-engine master switch


----


**UPGRADE EX4600 OR QFX WITH __NSSU__**\\  
\\
**for minimal downtime see [[https://forums.juniper.net/t5/Ethernet-Switching/EX4200-virtual-chassis-upgrade-with-minimal-downtime-hitless/td-p/161262|External Link]]**  ***<<< BE VERY PATIENT, IT ENDS UP DOING ALL MEMBERS BUT TAKES TIME!!**

  set chassis redundancy graceful-switchover
  set protocols layer2-control nonstop-bridging
  set chassis redundancy graceful-switchover
  set routing-options nonstop-routing
  request system software nonstop-upgrade force-host /var/tmp/jinstall-ex-4200–12.1R5.5–domestic-signed.tgz  # issu, chassis same type
  request system software nonstop-upgrade reboot <package>
  request system software nonstop-upgrade set [/var/tmp/package-name.tgz /var/tmp/package-name.tgz] # MIXED virtual chassis
  # Then reboot or, if we want to rollback 'request system software rollback'
\\
__POST UPGRADE CHECKS__
  show interface terse  # REVIEW ALL BONDS STAY UP AFTER UPGRADE
  show virtual-chassis
  show system alarms
\\
\\
If we stop the upgrade (power failure or similar)
  * the system might fail over to the secondary partition. To fix this, follow this [[https://kb.juniper.net/InfoCenter/index?page=content&id=KB23180|External Link]]
  * members in different version, member inactive. To reactive the member or to upgrade it as a standalone, Break the VCP:
    * Manually (remote hands)
    * request virtual-chassis vc-port set interface vcp-0 disable # from the reachable one.
  * In an interrupted upgrade, its possible that some of the configuration might be missing. restore it.
  * If all members are reachable and we want to rollback: [[https://kb.juniper.net/InfoCenter/index?page=content&id=KB19500|External Link]]


----
UPGRADE VC MANUALLY (NON-NSSU)\\
BEFORE STARTING, DO THIS:
  # 1st thing verify console access is OK
  # log the output of your ssh session and collect the info from the commands below:
  delete system services ssh root-login deny # and copy the image
  request support information | no-more
  show chassis hardware | no-more
  show ethernet-switching table | no-more
  show interface terse | except down | no-more
  show interfaces descriptions | no-more
  show virtual-chassis | no-more
  show system alarms  # request system configuration rescue save 

CHECK THIS:
  show system uptime   # date time needs to be correct
  show task replication     # needs to be enabled
  ONLY IF THIS IS A TWO MEMBER VIRTUAL CHASSIS!!
  show configuration | display set | match split
  set virtual-chassis no-split-detection # << ONLY IF TWO MEMBER VC!:
  show configuration | display set | match nonstop
  set protocols layer2-control nonstop-bridging
  set routing-options nonstop-routing
  show configuration | display set | match switchover
  set chassis redundancy graceful-switchover
  show virtual-chassis vc-port    # verify all members are connected to each other (daily chain). For each of the fpcx sections, check Status (Up) and neighbour (needs be two and contiguous). This is important, otherwise upgrade will fail and VC with break 




Now the proper manual upgrade:

  file copy /tmp/jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz fpc1:/tmp/    # push packet to member to be upgraded (fpc1)
  request session member fpc1

  wildcard range set xe-1/0/[0-23] disable
  wildcard range set et-1/0/[24-27] disable

  request virtual-chassis vc-port set interface vcp-0 member 1 disable  # << example. disabled the VCP on the member 1 and member 0 and then console onto member 1

  request system software add /tmp/jinstall-ex-4200-13.2X51-D35.3-domestic-signed.tgz reboot validate reboot force-host # before check servers are fine. From console in the the isolated fpc, trigger the upgrade.
  # Then reboot or, if we want to rollback 'request system software rollback'

\\
TSHOOT ISSUES AFTER UPGRADE:

- If error like : "warning: Database header sequence numbers mismatch for file \n '/var/run/db/juniper.data'. If a package has just been" the apply the commands below (provided by jtac). More info [[https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1426341|here]]:

  rm -rf /var/db/scripts/translation/openconfig-*
  mgd -I




----

UPGRADE JUNOS WITH ANSIBLE:\\
[[http://anastarsha.com/install-and-upgrade-junos-software-packages-using-ansible/]]
----

VIRTUAL SWITCH MANAGEMENT:\\
Move roles in a running virtual chassis:\\

#1 initial configuration:

  set virtual-chassis preprovisioned
  set virtual-chassis member 1 role routing-engine
  set virtual-chassis member 1 serial-number BP0208369135
  set virtual-chassis member 0 role routing-engine
  set virtual-chassis member 0 serial-number BP0208369174  << currently master
  set virtual-chassis member 2 role line-card
  set virtual-chassis member 2 serial-number LX0213502924
  set virtual-chassis member 3 role line-card
  set virtual-chassis member 3 serial-number LX0213502917
\\

#2 apply this config. takes 2-3 minutes to make effect.
  set virtual-chassis preprovisioned
  set virtual-chassis member 1 role line-card   << becomes line-card
  set virtual-chassis member 1 serial-number BP0208369135
  set virtual-chassis member 0 role routing-engine 
  set virtual-chassis member 0 serial-number BP0208369174  
  set virtual-chassis member 2 role routing-engine  << becomes re backup!!
  set virtual-chassis member 2 serial-number LX0213502924
  set virtual-chassis member 3 role line-card
  set virtual-chassis member 3 serial-number LX0213502917

\\
#3  apply this config. takes 2-3 minutes to make effect
  set virtual-chassis preprovisioned
  set virtual-chassis member 1 role line-card
  set virtual-chassis member 1 serial-number BP0208369135
  set virtual-chassis member 0 role line-card  << becomes line-card
  set virtual-chassis member 0 serial-number BP0208369174  
  set virtual-chassis member 2 role routing-engine << moves to re master!!
  set virtual-chassis member 2 serial-number LX0213502924
  set virtual-chassis member 3 role routing-engine << becomes re backup!!
  set virtual-chassis member 3 serial-number LX0213502917



----
SANITY CHECKS IN GS

  * Test ftp/netapp:\\
  Go: https://filer01-mgt.dc.mycompany1.co.uk/sysmgr/SysMgr.html#
  ssh ftp01 and write in mount

  * Check internet reachability from any cc 1:1 outbound
  [root@cc05.dc.mycompany1.co.uk ~]# ping google.com
  * Check reachability from internet to internal hosts 1:1 inbound 
  telnet clarify.mycompany1.co.uk 443 # this is in asci pool
  * check crawling : **curl ipecho.net/plain**
TO check that nat44 is happening in eacg different CC
  show services inline nat pool _jinpool_0/18/src_r2_cc01
  show services inline nat pool _jinpool_0/19/src_r2_cc02
  show services inline nat pool _jinpool_0/20/src_r2_cc03
  show services inline nat pool _jinpool_0/21/src_r2_cc04
  show services inline nat pool _jinpool_0/22/src_r2_cc05
  show services inline nat pool _jinpool_0/23/src_r2_cc06
  show services inline nat pool _jinpool_0/24/src_r2_cc07
  show services inline nat pool _jinpool_0/25/src_r2_cc08
  show services inline nat pool _jinpool_0/26/src_r2_cc09
  show services inline nat pool _jinpool_0/27/src_r2_cc10
  show services inline nat pool _jinpool_0/28/src_r2_cc11
  show services inline nat pool _jinpool_0/29/src_r2_cc12
  show services inline nat pool _jinpool_0/30/src_r2_cc13
  show services inline nat pool _jinpool_0/31/src_r2_cc14
  show services inline nat pool _jinpool_0/32/src_r2_cc15
  show services inline nat pool _jinpool_0/33/src_r2_cc16

  * Check napt-44
  [root@titan26.dc.mycompany1.co.uk ~]# telnet google.com 80
  (fw)# sh nat translated 89.145.95.2 detail # there should be transalated hits
  # testing the below translation  
  object network obj-10.8.11.0
   nat (management,outside) dynamic 89.145.95.38

----
IMPLEMENTING RSTP: \\
  * We should implement bpdu-control (bpdu-block-on-edge) in the edge interfaces so they get disabled if they receive a bpdu. Otherwise they would send a TC when the port changed state
RSTP is configured on an interfaces basis:\\
[[https://www.juniper.net/documentation/en_US/junos/topics/topic-map/spanning-tree-configuring-rstp.html]]

  set protocols rstp bridge-priority 16k
  set protocols rstp interface xe-0/0/13.0 mode point-to-point 
  set protocols rstp interface ge-0/0/3.0 mode edge

To **quickly** add rstp to a port:
  del interfaces et-0/0/26 
  set protocols rstp interface et-0/0/26
  set interfaces et-0/0/26.0 family ethernet-switching

----

**__Upgrade phases/milestones:__**

  * The Virtual Chassis master verifies that: 
    * The backup is online and running the same software version.
    * Graceful Routing Engine switchover (GRES) and nonstop active routing (NSR) are enabled.
    * The Virtual Chassis has a preprovisioned configuration.
  * The master installs the new software image on the BACKUP fpc1 normally and reboots it.
    * issu: preparing daemons
    * issu: upgrade FRU
  * The master resynchronizes the backup.
  * Linecard upgrades (if present)
    * The master installs the new software image on member switches that are in the linecard role and reboots them, one at a time. The master waits for each member to become online and active before starting the software upgrade on the next member.
    * Any lacp members should be 'ready to carry traffic' : KERN_LACP_INTF_STATE_CHANGE: lacp_update_state_userspace: cifd xe-1/1/2 - CD state - ready to carry traffic
**until this point, the master still runs the old junos version, it is the last one to jump to the new version**
  * When all members that are in the linecard role have been upgraded, the master performs a graceful Routing Engine switchover, and the upgraded backup becomes the master.
  * The software on the original master is upgraded and the original master is automatically rebooted. 
  * After the original master has rejoined the Virtual Chassis, you can optionally return control to it by requesting a graceful Routing Engine switchover.
\\
\\
UPGRADE EX4600 OR QFX
  * Supports NSSU (better than ISSU)
  * request system software nonstop-upgrade **force-host** /var/tmp/package-name.tgz  # so the hypervisor gets upgraded too
  * request app-engine host-shell # to get into the proper OS (now junos is a guests of the Centos OS)
QFX
  request system software add /var/tmp/jinstall-host-qfx-5-17.2R1.n-signed.tgz
  # Then reboot or, if we want to rollback 'request system software rollback'


----


**HOW TO CREATE A NEW VIRTUAL CHASSIS MADE OF 4600 **\\

  * Do not connect the VC cables yet (QSFP port in front panel, in this case)
  * Power on both swicthes and note down the RE serial numbers (chassis_s-n).
  * Shut down one of the members and connect the VC cables.
  * Keep powered on only the member that will become master. Connect via console. Console settings here: [[https://www.juniper.net/documentation/en_US/release-independent/junos/topics/task/configuration/ex-series-initial-configuration-setting-up-cli.html]]

  set system root-authentication plain-text-password # then the usual root password
  set system host-name sw-Xyy
  commit
  @log out and verify hostname and root access OK
  set virtual-chassis preprovisioned
  set virtual-chassis member 0 serial-number <chassis_s-n-0> role routing-engine
  set virtual-chassis member 1 serial-number <chassis_s-n-1> role routing-engine
  # if 2 member chassis
  set virtual-chassis no-split-detection
  commit


  - NOW Power on the other member switche/s. 

  - Do this in each of the members

  request virtual-chassis vc-port set pic-slot 0 port 24 local
  request virtual-chassis vc-port set pic-slot 0 port 25 local
  request virtual-chassis vc-port set pic-slot 0 port 26 local
  request virtual-chassis vc-port set pic-slot 0 port 27 local
  
NOTE: if we want to remove a vc-port, we use the above but with 'delete'. **WARNING** Before converting a vc port to a normal port, best to have it disconnected/shutdown, otherwise, in absence of STP, we can create a **nasty layer 2 loop!**
  
- Verify VC is healthy, output similar to this

  jaime_santos@sw-d09> show virtual-chassis 
  
  Preprovisioned Virtual Chassis
    Virtual Chassis ID: XXXX.XXXX.XXXX
  Virtual Chassis Mode: Enabled
                                                  Mstr           Mixed Route Neighbor List
  Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
  0 (FPC 0)  Prsnt    <chassis_s-n-0> ex4600-40f     129   Backup       N  VC   1  vcp-255/0/24
                                                                             1  vcp-255/0/25
                                                                             1  vcp-255/0/26
                                                                             1  vcp-255/0/27
  1 (FPC 1)  Prsnt    <chassis_s-n-1> ex4600-40f     129   Master*      N  VC   0  vcp-255/0/24
                                                                             0  vcp-255/0/25
                                                                             0  vcp-255/0/26
                                                                           0  vcp-255/0/27


----
NEW MIXED VIRTUAL CHASSIS FROM EXISTING EX4200 (ADD EX4550):\\
Any of the members can be Master in this kind of mixed VC, in this case we will have the 4200 keeping their master role.\\
[[https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/virtual-chassis-ex4200-ex4500-cli.html]]
\\
In the EX4550:\\

  * Do not connect the VC cables yet
  * Power ON the new EX4550 and enable CON
  * Zeroize the devices to avoid issues
  * Note down the RE serial numbers (chassis_s-n).

  set system root-authentication plain-text-password # then the usual root password
  set system host-name sw-Xyy

  * Connect them to mgmt to the laptop and upgarde the firmware in the new members. to 15.1R7.9
  * Verify the PIC mode setting:

  show chassis pic-mode # If the PIC mode was not set to Virtual Chassis mod, set the PIC mode to Virtual Chassis mode:
  request chassis pic-mode virtual-chassis

  * Reboot each future member
  * shut them down
  * Interconnect the VC cables for the 4 members, daisy chain.

In the EX4200:\\

  set virtual-chassis preprovisioned
  del virtual-chassis no-split-detection
  set virtual-chassis member 0 role routing-engine
  set virtual-chassis member 0 serial-number BP0208369580
  set virtual-chassis member 1 role routing-engine
  set virtual-chassis member 1 serial-number BP0211193901
  set virtual-chassis member 2 role line-card
  set virtual-chassis member 2 serial-number XXXXXXXXXXXX # new SNs
  set virtual-chassis member 3 role line-card
  set virtual-chassis member 3 serial-number XXXXXXXXXXXX
  
Power ON the new EX4200, now connected to the VC lot

  show virtual-chassis # Verify VC is healthy, output similar to this
    Preprovisioned Virtual Chassis
  Virtual Chassis ID: 5ca4.23ce.6939
  Virtual Chassis Mode: Mixed
                                                  Mstr           Mixed Route Neighbor List
  Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
  0 (FPC 0)  Prsnt    XXXXXXXXXXXX ex4200-48t     129   Backup       Y  VC   1  vcp-0      
                                                                             2  vcp-1      
  1 (FPC 1)  Prsnt    XXXXXXXXXXXX ex4200-48t     129   Master*      Y  VC   3  vcp-0      
                                                                             0  vcp-1      
  2 (FPC 2)  Prsnt    XXXXXXXXXXXX ex4550-32f       0   Linecard     Y  VC   3  vcp-255/2/0
                                                                             3  vcp-255/2/1
                                                                             0  vcp-255/2/3
  3 (FPC 3)  Prsnt    XXXXXXXXXXXX ex4550-32f       0   Linecard     Y  VC   1  vcp-255/2/0
                                                                             2  vcp-255/2/2
                                                                             2  vcp-255/2/3
  show virtual-chassis vc-port


----
TROUBLESHOOT BROKEN VC ( VIRTUAL CHASSIS ):\\
  - in case of 2 devices, the backup will survive, the master will suspend in case of 3 the one who still sees a second device will survivE
  - split the master election is run again to determine master , if the pre-split master is still detecting more than half the VC members still connected to it it will assume mastership. the pre-split backup will assume master ship if it detects at least half the Virtual Chassis members still connected to it

  show chassis pic pic-slot 2 fpc-slot 4   # To check the statuc of a vc module
  request virtual-chassis device-reachability test-name member3-to-member4 source-fpc 3 destination-fpc 4
  request virtual-chassis reactivate <ENTER> # You need to console to the member in question. You can use this command to reactivate a device that was previously part of the Virtual Chassis or VCF but whose status is no longer Prsnt.


----


**DUPLEX SETTINGS IN JUNIPER:**\\
Note that is NOT ENOUGH with set the interface to full 100 (for instance). We need to expicitely  disable auto negotiation:
  set interfaces ge-0/0/4 ether-options link-mode full-duplex
  set interfaces ge-0/0/4 ether-options speed 100m
  set interfaces ge-0/0/4 ether-options no-auto-negotiation <<<<<<<<<<<

Result: 

  jaime_santos@cpe1.singapore> show interfaces ge-0/0/4 | match MTU 
  Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 100mbps, Duplex: Full-Duplex,
  

----

**RPM PROBES AND EVENT POLICIES (~ SLA AND EMM IN IOS)**\\

# RPM below generates an event:


  set services rpm probe icmp-ping-probe test ping-probe-test probe-type icmp-ping test-interval 2
  set services rpm probe icmp-ping-probe test ping-probe-test probe-type icmp-ping target address 10.2.2.1
  set services rpm probe icmp-ping-probe test ping-probe-test thresholds successive-loss 1
  set system syslog file syslog-event-daemon-info daemon info

Event logs location:

  show log syslog-event-daemon-info

Event (config change in response to the rpm):

  set event-options policy disable-on-ping-failure events ping_test_failed
  set event-options policy disable-on-ping-failure within 5 trigger on
  set event-options policy disable-on-ping-failure within 5 trigger 1
  set event-options policy disable-on-ping-failure attributes-match ping_test_failed.test-owner matches icmp-ping-probe
  set event-options policy disable-on-ping-failure attributes-match ping_test_failed.test-name matches ping-probe-test
  set event-options policy disable-on-ping-failure then change-configuration retry count 5
  set event-options policy disable-on-ping-failure then change-configuration retry interval 4
  set event-options policy disable-on-ping-failure then change-configuration commands "set interfaces ge-0/0/1 description "BBBBBB"
  set event-options policy disable-on-ping-failure then change-configuration commit-options log "updating configuration from event policy disable-on-ping-failure"
  
  set event-options policy enable-on-ping-completed events ping_test_completed
  set event-options policy enable-on-ping-completed within 5 trigger on
  set event-options policy enable-on-ping-completed within 5 trigger 1
  set event-options policy enable-on-ping-completed attributes-match ping_test_completed.test-owner matches icmp-ping-probe
  set event-options policy enable-on-ping-completed attributes-match ping_test_completed.test-name matches ping-probe-test
  set event-options policy enable-on-ping-completed then change-configuration retry count 5
  set event-options policy enable-on-ping-completed then change-configuration retry interval 4
  set event-options policy enable-on-ping-completed then change-configuration commands "set interfaces ge-0/0/1 description "AAAAAA"
  set event-options policy enable-on-ping-completed then change-configuration commit-options log "updating configuration from event policy enable-on-ping"
\\



----

POLICY TROUBLESHOOTING:\\
To see the policy hits:
  show services rpm probe-result
  > show policy statistics Default>>>OSPF    
  Policy Default>>>OSPF:
      [705] Term Inject:
          from [13 0]  proto BGP
               [13 0] route filter:
                   0.0.0.0/0 exact
          then [13 0] ospf-external-type 1 [13 0] accept
      [692] Term Reject:
          then [692 0] reject

To test a policy BEFORE APPLYING IT:
  > show policy statistics eBGP_OUT       
  Policy eBGP_OUT:
      [647] Term AdvertiseOut:
          from
               [5 0] route filter:
                   148.64.56.0/24 exact
          then [5 0] accept
      [642] Term Reject:
          then [642 0] reject


----

**CoS**:\\
  show interfaces queue xe-5/1/0
  show interfaces queue xe-5/1/0 forwarding-class <name>



----
TROUBLESHOOTING:\\
  set cli timestamp
  show snmp statistics extensive 
  show system statistics  [upd/arp/bridge/icmp] [extended]  # Note this is for **switch bound packets!**
  show system buffers  #  route engine's packet memory (mbuf). To diagnose fragmentation in the re
  show pfe statistics traffic  


---

__**PFE TROUBLESHOOTING / DEBUGGING "VTY" **__\\
**INTERNAL FRAME PATH**:\\
Check this session: {{ :network_stuff:juniper:vty_fpc.txt |}} ; [[https://packetpushers.net/junos-useful-show-commands-capture-data-verification-troubleshooting-part-2/]]\\
  run start shell        
  vty fpc5
  show nhdb id <index>    # to see what the forwarding table actually does with that route
  show shim
  halp-analyser
  

  request pfe execute command "show nhdb type unicast" target fpc3 | match xe
  
**PFE TROUBLESHOOTING / CPU USAGE**: Don't use sh chassis routing engine (deceiving as everything under 'CPU utilization' is time, not load based (eg kernel 15 means kernel has been doing things 15% of the last 10 seconds)\\
Do **this** instead:
  show system processes extensive | except 0.0 | refresh 1
  start shell
  vty fpc0
  show syslog messages
  show threads
  show threads cpu
  show threads verbose
 
\\
** PFE TROUBLESHOOTING/ CAPTURE PACKETS DESTINED TO RE:**\\
To capture packets going to the routing engine: 
  rtsockmon -t    # If it shows a lot of add/delete routes there might be an issue with exception traffic
  rtsockmon : to view the actual route replication process \\
\\

** PFE TROUBLESHOOTING / MC-LAG**

For the full troubleshooting check here {{ :network_stuff:juniper:pfe-tshoot-mclag.odt |}}

  request pfe execute target fpc0 command "set dcbcm bcmshell \"l3 l3table show\"" | grep "Entry|185.89.206.27" 
  Entry VRF IP address       Mac Address           INTF MOD PORT    CLASS HIT    H/W Index
  99    1    185.89.206.27    00:00:00:00:00:00  100154    0    0         0 y      141712
  # internal interface for .27 destination is 100154
  request pfe execute target fpc0 command "set dcbcm bcmshell \"l3 egress show 100154\""     
  HW (unit 0)
  Entry  Mac                 Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount L3MC
  100154  00:10:e0:bd:8e:0e    7    4     6t   0        -1   no   no    1   no
  # no drop. Uses internal vlan 7
  request pfe execute target fpc0 command "show bridge-dom"   
  SENT: Ukern command: show bridge-dom
  Bridging Domain                          BD-Index RTT-Index  BD-Type   BD-Hw-Token
  server-hosting+1                                6        4   Regular            7
  [..]
  # for vlan 7
  request pfe execute target fpc0 command "set dcbcm bcmshell \"vlan show\"" | grep "Vlan 7" 
  vlan 7	ports xe6-xe13,xe16-xe24,xe28,xe32,xe36 (0x0000000000000000000000000000000000000000000000000000002223fe7f80), untagged xe6-xe13,xe16-xe23 
  # to show all ifd
  request pfe execute target fpc0 command "show dcbcm ifd all"                                  
   ifd name     global-dev  local-dev   port-num   port-name
   xe-0/0/16             0          0         17      xe16
   xe-0/0/17             0          0         18      xe17
  [...]
  # this is cef information
  show route forwarding-table destination 185.89.206.27
  Routing table: default.inet
  Internet:
  Enabled protocols: Bridging, 
  Destination        Type RtRef Next hop           Type Index    NhRef Netif
  185.89.206.27/32   dest     0 0:10:e0:bd:8e:e    ucst     1805     1 ae5.0
  # this bounces the port physically (completely):
  request pfe execute target fpc0 command "set cmqfx xcvr remove/insert pic 0 port 17"
  # mclag filter creation seems to fail
  show log messages | last 20 
  Jun 15 14:24:12  csw1-coresite-la1 mib2d[2203]: SNMP_TRAP_LINK_UP: ifIndex 569, ifAdminStatus up(1), ifOper
  request pfe execute target fpc0 command "show filter hw all drop non_zero_only 0
  # THIS SHOWS THE FILTERS IN ACTION:
  F   9 U:  0 Pi: 0 G:33 E:   9216 A:IDR stat (id 7243 val 0x0000000000006A75) P:7FFFFD3F I7: protect-RE (IRACL)
  F  71 U:  0 Pi: 0 G:17 E:    127 H:0 A:CCD stat (id  127 val 0x000000000000004F) P:00000001 I5: CPU Code  69 -ipv6_linklocal
  F 131 U:  0 Pi: 0 G:17 E:    170 H:0 A:CCD stat (id  170 val 0x00000000002EB5DB) P:7FFFFFFB I5: COSQ 16 -ipv6-ns-na
  F 131 U:  0 Pi: 0 G:17 E:    172 H:0 A:CCD stat (id  172 val 0x000000000014014E) P:7FFFFFFB I5: COSQ 16 -ipv6-ns-na



INVESTIGATE QUEUE DEPTH FOR ARPs:\\
on 12.3R12.4 ARP is assigned to DSAIdx 5 and it goes to queue 2a which has 300pps bandwidth:\\
**lcdd , link card daemon**:connects you to various other parts of the switch (including the software forwarding infrastructure (sfid), chassis manager (chassism),
and the virtual chassis system (vccpd). \\
More info [[https://wtf.hijacked.us/wiki/index.php/Juniper|here]] and [[https://forums.juniper.net/t5/Ethernet-Switching/lcdd-0-chassism-on-QFX5100/td-p/273221|here]]:
  >start shell
  %lcdd 0 sfid [ 0 means fpc 0 ]  # Connects to the software forwarding infrastructure (sfid) process in fpc0
  sfid<1>#show stats ge-1/0/28 cpucodes
  Counter Type Rx Tx
  [...]\
  ARP 1027 0   # 1027 ARP packets get received on interface ge-1/0/28
  
  sfid<1>#show stats ge-1/0/28 hw-cpucodes
  DsaCode Rx Tx
  2 326657 0
  5 1027 0   # This dsacode hit with same number of packets (ARP is assigned to DSAIdx 5 and it goes to queue 2a. This queue has 300ppp bw
  [...]

  PFEM1(vty)# show shim ddos cpu-code 5
  Dev DSAIdx CpuCode Client State Q DP Trunc TgtCPU RLMode StatRLIdx RLIdx Rate
  0 005 00032 any notcare 2 green 0 0 1 0 20 Q2a 300     # from PFE you can check rate-limit for Dsacode/idx 5. (300)

----
FACTORY RESET 4200:\\
[[https://www.juniper.net/documentation/en_US/release-independent/junos/topics/task/configuration/ex-series-switch-default-factory-configuration-reverting.html]]
  request system zeroize


----
SNAPSHOTING PARITION IN EX2200 (NO EXTERNAL MEDIA):
  show system storage partitions  
  show system snapshot media internal
  request system snapshot slice alternate << if all good, snapshot from the current (healthy) to the other one)
If it doesn't work, it could be that the altroot is still mounted. see this: [[https://www.b00z.nl/blog/2016/04/juniper-srx-error-could-not-format-alternate-root-solution/]]
----
**TROUBLESHOOT MAC TABLE ISSUES:**\\

  show ethernet-switching table 
  show ethernet-switching flood
  show ethernet-switching statistics mac-learning interface xe-3/0/3 detail
  monitor interface ge-0/0/1




----


MAC-CHURNING (issue these commands every 5secs and compare):

  show ethernet-switching mac-learning-log
  show ethernet swithching table


----


**JUNOS NAMING CONVENTION:**

  * 14.1X53-D45.3
    * 14.1X53 << VERSION
    * D45 << RELEASE

  * JSA: Juniper Security Advisories