This is an old revision of the document!
NEW SWITCH DEPLOYMENT, BASIC ACCESSORIES, NEVER FORGET!!
- DATACENTER
- (1) RACK PDUs
- CISCO ORDER
- (2) REDUNDANT PSU
- (3) CORRECT POWER CORDS
- (4) MOUNTING RAILS!
- (5) STACKING CABLES (VSS) 5.1 9300LM HAVE SPECIAL EXPENSIVE T3 TYPES AND NO STACKPOWER
- (6) CORRECT TRANSCEIVERS AND CABLES
INTERCONNECTION TYPES:
- TRANSIT PEERING:
- (aka common internet access): you pay an isp and gives you access to internet
- IXP (PUBLIC PEERING)
- Equinix ibx
- route server and bilateral peering
- PNI (literally a cable, xconnect p2p)
- Bilateral peering
- settlement free peering or pay-per-peer
- SFI (Settlement Free Interconnect, normally between ISPs)
SP terms:
- UNI, User Network Interface, is the interface that faces the subscriber
- NNI, Network Node Interface, is the interface that faces the service provider network.
- SMB (service management boundary): demarcation point. Its location changes depending on whether the router is managed by the SP or not. If the former, the smb is normally in the LAN side of the cpe, if the former, it's usually in the WAN side.
EQUINIX ECX
- General overview
- Port details (QinQ) External Link
- Normally there's a limitation per session (100mb?)
Guidelines:
- Best Practices for Network Interconnections. External Link
- BGP Best Practices for ISPs External Link or https://www.yumpu.com/en/document/view/46371637/bgp-best-practices-for-isps-a
- Filter inbound BGP prefixes External Link
For inbound filtering (maintained by BOGON project):
IPV6 NOTES , IPV6 BEST PRACTICES:
https://blogs.infoblox.com/ipv6-coe/a-48-for-every-site-and-for-every-site-a-48/
ICANN/IANA recommendations (we have /32):
- /48 per site
- /48 is smallest Internet routable IPv6 prefix
- /64 link, hosts. smallest permissible p2p. note that, for instance slaac ips might be in this range
- link-local are stated as /8 but they are really used as /64 once we consider the IID External Link
- ULA (should be restrict to the network realm)
- we, normally, subnet IN THE NIBBLE BOUNDARIES. However sometimes we don't. Example: country is given /29 so it has eight /32 prefixes.
SLAAC+dynamic DNS registration vs RA+DHCPv6
- RA tells clients what to use of both
- DHCPV6 (sl or sf)>
- Stateful ( M=1,O=x,A=0) : exact porting of DHCP into IPv6 world
- Stateless(~pseudo-slaac) ( M=0,O=1,A=1): (Other and Autonomous)) meaning: Three players: 1) SLAAC to configure your IPv6 address 2) gateway given via RA and 3) DHCPV6 to give you the Other very important info:info like dns server and dns suffix list. @@' ; This is what i have at home with the openwrt . A flags stands for 'Autonomous Flag'
ipv6 dhcp pool LAN1 …iface) ipv6 nd other-config-flag
- Pure SLAAC £ ( M=0,O=0,A=0) : for more modern green field deplo. or modest systems (iot)
@ This seems a bit convoluted but this decoupling of default route from the dhcpv6 allows resiliency from any possible failure of the dhcp server
@@ Also the next hop you see in the rt (given by the RA) is the link local address!. There is not dhcpv6 support in Android ;
FLAGS
- @@ M flag “Managed address configuration” : Suggests the host to use dhcpv6 for address configuration
- @@ O flag “Other stateful configuration” : Suggests the host to use dhcpv6 for additional configuration like DNS and domains
- @@ A flag : Suggests autoconfiguration (SLAAC)
- Another way of 'forcing' the host to us dhcpv6 is for the router not advertise the link so the host doesn't see the prefix as an on-link prefix and therefore it thinks is talking directly to the gateway. ipv6 nd prefix FEC0:1:2300:1::/64 no-advertise' in PE towards the client. See example in pdf below.
sdjo
£ Global Unicast Allocation (GUA, /64) sent via a RA; node combines this prefix with its interface identifier resulting in the GUA address (eg: 2001:db8:66:6:536:e8ff:febb:14cb)
ABOVE https://blogs.infoblox.com/ipv6-coe/slaac-to-basics-part-2-of-2-configuring-slaac/
For linux: https://www.tldp.org/HOWTO/html_single/Linux+IPv6-HOWTO/
Sample configurations: large_ipv6_access_networks.pdf
- Slide 13 : simple static route, 14 routing (tag, bog no-export additive)
- Slide 31 : PE slaac configuration. “ nd router-preference High” to prevent client-side 'mistakes' ; 'no ipv6 nd ra suppress' (to be sure ra is on)
- Slide 32 : PE dhcpv6 configuration. 'ipv6 nd managed-config-flag'
FEC0:1:2300:1::1/64
BRANCH
BUYING TRANSIT SERVICES:
What to ask:
- We require at least 300Mbit on 1G bearer. Can you send us some quotes? If a site visit is required, we can organise it.
- We want to use BGP with multihoming, advertising our own PI /24 range. Please confirm.
- Bgp policy
- Maximum prefix length for advertised prefixes
- ISP advertises default route / full Internet table?
- maximum-prefix towards you
- Do you support IPv6?
- Do you accept 4 bytes Autonomous System numbers
- Can we use our PI range to the peer with you or you'll provide VM a /30?
- Do you accept bfd for fast bgp failover
- Related to the physical connectivity (understand this might require site visit):
- Physical connectivity to your PE. Optic fibre types, PE location.
- Can we provide (choose) and have access to the CE router?
- CE router
- Medium term we want to move to mpls. What options does VM offer?
- General Timelines
DC
BUYING TRANSIT SERVICES. What to ask:
- Capacity
- Fibers presented
- Peering with aws (provide information)
- BGP POLICY (Default Route Only - Full BGP Table - Default Route + Full BGP Table)
- Maximum prefix length for advertised prefixes
- ISP advertises default route / full Internet table?
- maximum-prefix towards you
- policy control
- Can we use our PI space?
- Support for 4 bytes ASN
- ipv6 support
DATACENTER EQUIPMENT
PDU:
- Monitoring capabilities: standard (not monitoring) ; metered/smart (monitored). Check this site: https://www.apc.com/shop/sg/en/products/Rack-PDU-2G-Metered-ZeroU-32A-230V-36-C13-6-C19/P-AP8853
- max input curent (~30A), max input power ~5kVA)
- Input plug:
- NEMA (American domestic standard)
- IEC
CONNECTOR TYPES, CHECK THIS LINK FIRST: https://www.showmecables.com/blog/post/power-cables
- C14 male (this connects to the pdus in gyron); PDUs have C13
- C19 female: typical of Dell or Sun servers
- C14-C19: for servers
- C20-C19: for servers (just 1 in each PDU in Gyron)
- C14-C13: for switches (low power)
- C14-C13: for switches (high power)
PDU TYPES:
- 3-phase:
- metered (the normal one): Receptacle type is L15-30
- switched switch rack pdu will allow you to turn outlets on or off :
USB TYPES:
Datacenter powering concepts
The datacenters sell you, for instance 4.99 kVA. but it should be used in a primary and redundant fashion.
That's per feed, so you can potentially load the rack up to x2 4.99 kVA. The problem is that, if one feed is disconnected, the other will not be able to absorb all the power and will trip out!
Good numbers per rack is:
- 10 kVA ; 50 kVA (by using the three phases to the full)
- High-density racks: 14kW total: “three-phase” power delivery (32A three-phase commando sockets) under the floor to those racks.
- three phase 30/208v (an example)
- Medium-density, based in NEMA_L6-30 , and you can draw 5.5kW (which is approx. 24A) from those. Single-phase power delivery (32A single-phase commando sockets).
- In datacenter power redundancy is given by: A-feed + B-feed + Generator + Bypass
dc 2 vs three phase disquisition:
- KVA is the unit of apparent power. Real power is always less due to inductive loads. kW/kVA is <1 power factor
- Three phase power circuits are often used in power transmission lines and large electric motors because they allow lower line voltages and provide a smoother flow of electricity.
- Power factor for 3-phase is square root of Three (1.73)
- Formula for Amps (in 3-phase) is : Amps = Watt/(Volt*1.73)
- see External Link
Use case:
- 32-amp 400v Three Phase Redundant AC Power
- Available power (because is 3 phase) = (32*400)*1.73=22.1 kVA
- Each phase will be 230V when taking the neutral as a reference. Because each phase is phased out by 120°, the voltage between any two phase is 400V.
The above is Wye (the other one is Delta configuration, more used for motors and equipment requiring more initial torque)
OPTIC TECHNOLOGIES
https://community.fs.com/blog/name-brands-vs-third-party-transceivers-which-do-you-prefer.html ; https://tmgmatrix.cisco.com
SFP , SFP+(1G or 10G) , SPF28, QSFP , QSFP28, QSFP-DD
- SFP28 (25G)
- QSFP+ 4x10G lanes. (4Gbps, not in use anymore)
- QSFP+ modules can break out into 4x10G lanes: 40Gbps (QSFP-PLR4)
- 100G QSFP28 4x(up to)28G : is 100G
- can break out into x4 or x2 lanes
- QSFP28-100-SR4 - MPO connector - MPT12 cable - 100m with OM4 cables (OM3 ~70m) - Be aware of mpo_rules when selecting cable
- QSFP28-100-LR4 - LCLC cable connector - 2km OM4
- QSFP28-100-LR4-S
- QSFP-DD form factor is is backward compatible with qsfp+ (40G), qsfp28 (100G) and qsfp56 (200G). Allows less power than the osfp
DR vs FR: difference is the distance they support. FR is 2km while DR is 500m. Different in the transmitter power and the receiver sensitivity
For single mode, nowadays tends to be OS2 UPC
Modulation:
- NRZ (low/high) for short distances
- Coherent optics (Multi-level Amplitud and/or phase modulation): for long (thousands) distances.
Silicon photonics: We 'couple' light into silicon (concept of impedance in silicon fotonics) so we can guide compine split modulate it
PHYSICAL LABEL JUNOS HARDWARE CLI:
Use this for juniper hardware: https://apps.juniper.net/hct/home/
- FS: 1GLX-31 > SFP-SX (1g LR)
- FS: LX (740-011614) > SFP-LX10 (1g SR)
- FS: 10GLR31 > 10g-LR (classical 10G SFP+)
- FS: QSFP-LR4-40G > QSFP+-40G-LR4 (40G, x1 SMF)
- FS: QSFP-PLR4-40G > 'UNKNOWN'(X4 10G OVER 40G connected to mpo dom harness/breakout cable)
100Gb/s (4 lanes so QSFP28 )
400GBPS:
- '400 qsfp dd with dr4 connector'
- MSA: Group of companies developing standars. Quicker than IEEE
- QSFP-DD (Double density/cage)“ : backward compatible with qsfp External Link
Manufacturers
- Lumentum Operations: Makes the optics themselves, based in usa.
- Finistar: Makes the optics themselves, based in usa
- FlexOptix is a German company with a good reputation.
- Luma Optics
- Fibertone
CAT5e-6
- PVC patch cords are very soft; ««
- LSZH patch cords are more rigid because they contain the flame retardant compound, and they are aesthetically more pleasing.
DWDM
- Head devices: (see this link
- ADVA
- CIENA
OADM (add drop multiplexer): An OADM generally consists of three parts: an optical demultiplexer, an optical multiplexer, a method of reconfiguring the paths between the optical demultiplexer and the optical multiplexer, as well as a set of ports for adding and dropping signals. The multiplexer is used to couple two or more wavelengths into the same fiber.
Branding
- Zayo wavelenght
- BT WaveStream
Setups:
- Same provider
- Less control on the protection method
- More opaque about resiliancy
- Different providers
- What if they do maintenance same time on a Saturaday night?
ISPs are confirmed to be on-net
EQUIPMENT AMORTIZATION DEPRECIATION FORMULA:
For servers: 3 year depreciation
Example if 20 months: (36-20)/36 = 44% of what they cost
MESSAGE BUS
- Apache kafka
- ZeroMQ Rabbit MQ
- Protobuf
SUBMARINE CABLES:
https://www.submarinecablemap.com/#/submarine-cable/apollo
- Europe to south Asia : https://en.wikipedia.org/wiki/SEA-ME-WE_4
CDN NOTES:
Eg: Akamai
- CDN must direct users to the best edge location
SP NAMING:
- CenturyLink (Level3) now Lumen
- alias: QWest
- gin.ntt.net : Gyron (same)
- Arelion (formerly Telia carrier)
- twelve99 is their backbone
LOCATION ACRONYMS:
-
- msp1.he.net : Minnesota, USA
- Los Angeles : los , lax
- torix.ca : Toronto Internet Exchange
- vanix.ca : Vancouver Internet Exchange
UK DATACENTERS:
- LINX
- LON1 (all three in Brick Lane London)
- Interxion LON 1
- Interxion LON 2
- Interxion LON 3
- ICE (Intercontinental Exchange) Basildon
CARRIER ETHERNET NOTES
It is mostly a marketing term to define what happens between the Carrier and User.
- P2P service > E-Line
- P2MP service > E-tree
- LAN service > E-LAN
ROUTE SERVER NOTES
route server
Is a special type of BGP participant, it MUST:
- Maintain a separated Routing Information Base (Loc-RIB) for each peer configured as RS-client, containing the routes selected as a result of the ‘Best Path Selection’ process that is performed on behalf of that RS-client.
- Whenever it receives an announcement from a RS-client, it must consider it for the Loc-RIBs of the other RS-clients.
- This means that for each of them the route server must pass the announcement through the appropriate Out filter of the announcer.
- Then through the appropriate In filter of the potential receiver.
- Only if the announcement is accepted by both filters it will be passed to the ‘Best Path Selection’ process.
It is also common to demand from a route server that it does not modify some BGP attributes (next-hop, as-path and MED) that are usually modified by standard BGP speakers before announcing a route.
Implementations:
- FRR
DPU Data Processing Unit (IPU if intel)
All governed from the nic as this is really the part that never goes.
DPU is small computer with pcibus and fast memory. Accelerator to do things like crypto or nvme access.
- NVIDIA bluefiled based in arm with even ib.
- aws glue
- pensando: arm p4 dpu
- pluribus
BUSINESS SOFTWARE / SERVICES
- ERP (Enterprise resource planning): eg: Netsuite(Oracle)
- CRM (Customer relationship management): eg: Salesforce (dominant), Netsuite(Oracle)
- PSA (Professional services automation): eg: Netsuite(Oracle)
COPPER ETHERNET CABLE TYPES
https://www.comparitech.com/net-admin/difference-between-straight-through-crossover-rollover-cables/
- On a curious note, the name RJ-45 is a discontinued format of telephone line connector. The proper name for these connectors is 8P8C
- All cables are distrubuted in four pairs
- PC-PC or PC-router : straight through cables
- cossover : not in use
- rollover: PC to serial: The wire that connects to pin 1 at one end goes to pin 8 at the other end, the wire that goes to pin 2 and the first end goes to pin 7 at the other end. In the middle of the connector pins 4 and 5 are reversed.
SERIAL ADAPTERS (DB9 (female) to RJ45
- Straight-Through: TDB9F - RJ45 Modular Serial Adapter Ethernet to Console Server (B090-A9F)
- Cross-over : DB9F - RJ45 Crossover Modular Serial Adapter Ethernet to Console (B090-A9F-X)
https://www.decisivetactics.com/support/view?article=crossover-or-null-modem-vs-straight-through-serial-cable
The PowerEdge R740 system supports one serial connector on the back panel:
- which is a 9-pin connector, Data Terminal Equipment (DTE), (pinout)
- 16550-compliant (~rs-232 protocol)
For more info : https://www.staub.ca/search?q=DB9
ISP customer/access layer
- DSL (ADSL): DSLAM External Link
- CMTS (cable): External Link
LATENCY / THROUGHPUT
- Inter-region latency rule of thumb (1): 1ms per 100km
- On air is 3ms = 1000 Km ; 1us = 300m
- In reality should be more 0.5 ms with 200km/s speed of light in fiber
- Low latency rule of thumb: 1m in 3.3 ns
- From pps to bps = multiply by 4000
LTE DONGLE
https://openwrt.org/docs/guide-user/network/wan/wwan/ltedongle
- Specs:
- Chipset brand: ZTE, Qualcom…
- Serial protocols: QMI or MBIM 1)
When connected to the rpi:
You won’t find a TTY device because the dongle is in cdc_ether (network) mode. Check ip link show for a usb0-style interface. If you actually want a serial modem for AT commands, you have to unbind cdc_ether and bind the option driver so /dev/ttyUSB* appears. If your connection is already “100% working” in network mode, you can just treat it like an Ethernet device—no TTY required. root@rpi4:/dev# usb-devices | awk '/12d1/' RS= T: Bus=01 Lev=02 Prnt=03 Port=02 Cnt=01 Dev#= 5 Spd=480 MxCh= 0 D: Ver= 2.00 Cls=02(commc) Sub=00 Prot=00 MxPS=64 #Cfgs= 1 P: Vendor=12d1 ProdID=1591 Rev=01.02 S: Manufacturer=HUAWEI_MOBILE S: Product=HUAWEI_MOBILE C: #Ifs= 2 Cfg#= 1 Atr=80 MxPwr=2mA I: If#= 0 Alt= 0 #EPs= 1 Cls=02(commc) Sub=06 Prot=00 Driver=cdc_ether <<< E: Ad=83(I) Atr=03(Int.) MxPS= 16 Ivl=2ms I: If#= 1 Alt= 1 #EPs= 2 Cls=0a(data ) Sub=00 Prot=00 Driver=cdc_ether <<< E: Ad=02(O) Atr=02(Bulk) MxPS= 512 Ivl=0ms E: Ad=82(I) Atr=02(Bulk) MxPS= 512 Ivl=0ms
Sometimes they don't need to be QMI or MBIMQMI or MBIM devices typically create a /dev/cdc-wdm# interface and use specialized drivers (qmi_wwan or cdc_mbim), along with user-space tools like qmicli or mbimcli. CDC-Ether (NCM) devices show up simply as a USB-Ethernet interface
STACKWISE CABLES
- STACK-T1 can be used for C3850 and in C9300 (BUT NOT IN THE LM SERIES(*)
- FOR THE C9300LM WE NEED StackWise-320 Kit: The C9300L-STAK-KIT2-M ; The 9300LM don't run stackpower but need these more complicated cables
