dokucama

This is an old revision of the document!

PYTHON

Python virtual environment: Isolated “environments” where specific versions of Python can be installed along with independent sets of libraries and dependencies.
This is to create a venv, activate it and manage it with vscode:

python3 -m venv enauto    # To create an empty folder with all python r
cd <folder>
source bin/activate     # be sure the interpreter is now selected 'venv' on the low right bar in vscode
python3 -m pip install requests    # we can install stuff just in this venv and also execute from vscode
source my_virtual_env/bin/activate    # activates
deactivate    # once we are done, deactivate the environment
# In vscode you might require to deselect the venv manually (+"ctrl+shift+P"> python: Clear Workspace Interpreter Settings)

FLAGS AND KNOBS:

raw flag : raw_dummy_str = r“This is a \n raw string” # treat the backlash as a backlash
strip method : string.strip([chars]) # copy of the string with both leading and trailing characters removed

Interactive input data/INPUT FROM USERS

nb = input('Choose a number: ')

PRINT NOTES:
Link
Use , to separate strings and variables while printing by using print f:

print(f'I am {nb}') # mood is a Variable
!
print("{name}: {ip} {netmask}".format(
  name=interface["name"],
  ip=interface["ietf-ip:ipv4"]["address"][0]["ip"],
  netmask=interface["ietf-ip:ipv4"]["address"][0]["netmask"],

f or F ^^ used before the string (quote or double quotes) is 'new' string formatting in python. Much simpler External Link

TABLES
To present data use prettytable

PASS ARGUMENTS
Command line argument (ie whatever is after the script invocation). External Link

for arg in sys.argv[1:]:
    addr = ipaddress.ip_interface(arg)
    print("address =", addr)
    print("network =", addr.network)

The below is intended to be reuse by many other modules!:
http://stackoverflow.com/questions/419163/what-does-if-name-main-do

import sys
  def main(argv1, argv2, argv3):
    print "this is the first variable: ", argv1
    print "this is the second variable: ", argv2
    print "this is the third variable: ", argv3
  if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2], sys.argv[3])

COLLECTIONS

Dictionary methods(more)
- car.update({“color”: “White”}) # adds the element between brackets to the dictionary car
- car.get('color')

CONDITIONALS:

if ..: elif : else:
- we could have if without any of the other two.
- Too many if is a sign of poor coding. We can reduce them with: switch , functions , object-oriented programming
for name in names: <indented-body-of-instructions>
while <condition> : and-then-the-rest-indented

ITERATIONS:
Parallel iteration
Lists (zip lists intertwined like jacket list), then access both elements simultaneously:

sizes = ['small', 'medium', 'large']
animals = ['frog', 'wolf', 'elephant']
for size, animal in zip(sizes, animals): # read this from right to left (first zip lists and the go through it):
    print(f'Isee a {size} {animal}')

TO ITERATE THROUGH A DICTIONARY: - .items

animals = {'small': 'frog','medium': 'wolf','large': 'elephant'}
for size, animal in animals.items():
   print(f'I see a {size} {animal}')

TO GO THROUGH A LIST AND HAVE A COUNTER - enumerate():

Used when we want to go through a list and also have a counter.
The enumerate() function allows iterable objects and returns an enumerate object in the form of a tuple containing the index and the value

for page_num, is_checked in enumerate(pages_checked, 1):    # pages_checked is a list of 'False' and 'True' ; page_num is the position ; is_checked is one of the list elements
  if not is_checked:
    return page_num

LIST COMPREHENSION:
Easiest usage is to replace a for-if section:

sample = [ 'sfsdf', 'ggg' , 'wrwer']
print([ r for r in sample if 's' in r ])

Also for nested for loops (here it flattening lists):

flat_list = [item for sublist in l for item in sublist]

Main usage is for creating a list based on existing list:External Link

[expression for item in list]
h_letters = [ letter for letter in 'human' ]
  print( h_letters)

We can now identify where list comprehensions are used. If you noticed, human is a string, not a list. This is the power of list comprehension. It can identify when it receives a string or a tuple and work on it like a list. \\External Link List comprehension example:

REGEX

https://regex101.com/
TextFSM
ALL INFO HERE] * All regex can be done with or without [[https://www.stat.berkeley.edu/~spector/extension/python/notes/node79.html|compiling but is more efficient.

Resources here1 and here2
https://panda314159.net/doku.php?id=scripting:python_scripts:regex1.py
regex search ways in python: MATCH, SEARCH, FINDALL, and SUB

FINDALL: Finds all instances of the matched string. All the occurrences go inside a LIST. findall() matches all occurrences of a pattern, not just the first one. We can also use compile

re.findall(   'pattern', 'did you find one pattern or many patterns') 
['pattern', 'pattern']

Example: find any serial number 8 characters long after the pattern: “Caller pc :0x”:

file = fileh.read()
serials = re.findall(r'Caller pc    :0x([a-zA-Z0-9]{8}\s' , file)

SUB: Can replace a parameter.
First you need to find the parts of the string that need to be rewritten (you do this with re.sub). Then you rewrite that parts.

re.sub( 'pattern', 'turnip', 'did you find one pattern or many patterns') 
'did you find one turnip or many turnips'

SEARCH:

name_regex = re.compile(r"^vrf\s+(?P<name>\S+)")
name_match = name_regex.search(vrf)
sub_dict = {}
vrf_dict = {name_match.group("name"): sub_dict}

MATCH:

Compile the expressions
Then used the compiled variables as functions to do the matching.
If we see things like name_regex = re.compile(r“^vrf\s+(?P<name>\S+)”) 'name' is a matching group (ie multiple matches go there and can be accessed later as a dict or list.

regex1 = re.compile("^192\.168\.222\.\d{1,3}$")
regex1.match(var_to_match)
<_sre.SRE_Match object; span=(0, 13), match='192.168.222.5'>  # no result would mean no match

searches at the beginning of the string. Dont use match as it just matches at the Beginning of the string.

vrf = re.match( r'hostname (.*)', file)
print "The hostname of the Router is:\t", vrf.group(1)

LISTS -square brackets-
Important because we can 'feed' a for loop with a list. No need of index or counter.

for name in namelist:
  print name

To add elements to a list we can append them “.append”

a += 1 to increment a variable (counter)

OOP:

Put def description code between 
"""
Here
"""

That way it appears in the function help. To find the help section of a function we just do help(functionname.
It could be we just know the method but we don't know the function name. EG: we have a var=“string” and we want to know the var.strip(?) method help. In that case we do help(var). Thet will point to an error as below which in turn will show us the way:

HELP IN PYTHON (INLINE DOCUMENTATION):

help(var)
print(dir(int))    # shows all methods for int, included the dunder ones
help(str.strip)    # this is help for the method .strip (str is the py name of string (built-in type)
useful_function.__annotations__[‘return’]
To see the the methods available for an object: ''dir (obj)''.   # In python we have: objects dot method. To see the online help we do ''help(module)''

Class object polyphormism (so same function can be used for all animals). Abstract classes , not to be instantiated !?

FILE NOTES:

The correct way of opening files in python is by using with. Reason is that files are automatically closed. With the old method, even if we think we close them, sometimes, due to unexpected exceptions, they can end up not closed and corrupted. Link1 , Link2 To create a list from a file:

with open('/var/tmp/list1', 'r') as f:
 myNames = [line.strip() for line in f]

To show the current folder path and open a file in it

import os
with open(os.path.join(sys.path[0], "my_file.txt"), "r") as f:
    print(f.read())

Most recommended and advanced than os for folder for path manipulation is pathlib. See this LINK.
The main difference is the returned type of object (now is PosixPath instead of string, which gives more possibilities).

from pathlib import Path
Path.cwd()

IMPORTANT: In python, we don't need to 'load' the file in a variable to manipulate it. We can just use the file description to iterate on the elements an use different file object methods to extract info on demand.
Useful in file path and name manipulation is the function .strip() without parameters. Like that, IT REMOVES END WHITE SPACES.
See this entry on reading-and-writing-files-in-python

Handler = open(file.txt) 
Handler.read,  write, delete, close..... 
Handler.truncate()   # means Delete the file (for that handler)

# To create a list (transforming value into list)
list('Hello')

# To create a list from a text file:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

# To access to the second element to the end
var_list[-2] 

# to //slice// a part of the list
var_list[1:3]

# beginning implicit
var_list[:3]

# delete element
del var_list[-2]

# Is the element in the list?
'element' in list

With methods

list_var.index('value_to_find')
list_var.insert(pos,'value')
list_var.append('value_to_append')

Configuration generated from list read. Multi-line template. Python triple quote usage example:

with open('C:/Users/jsantos/Desktop/dser', 'r') as f:
 dser = [line.strip() for line in f]

for i in range(len(dmer)):
    interface = i+1
    print("""
interface GigabitEthernet4/""" + str(interface) +"""
 description MER """ + dmer[i] +""" PXXX Desk X (gi4/""" + str(interface) +""")
 switchport
 switchport access vlan 20
 switchport trunk native vlan 998
 switchport mode access
 switchport voice vlan 34
 platform qos vlan-based
 platform qos trust cos
 no snmp trap link-status
 macro description datavoice-port
 no cdp enable
 spanning-tree portfast edge
 spanning-tree bpduguard enable
 !
""")

Slicing lists

print namelist[:2]   # prints up to but not including index 2. So 0 and 1.

Updating list .append
.pop(0) # number is number of elements to be 'popped'

FOR LOOP WITH LISTS

for i in list(range(5)):
 print(i)

This is a very useful way of accessing elements of a list with arbitrary length

for i in range(len(supplies)):
 supplies[i]

DICTIONARY-Curly braces or the set() function-

 stuff = {'name': 'Zed', 'age': 39, 'height': 6 * 12 + 2}

Print entry (with key):

print stuff1["age"]

Add entry (with key):

Dictionary_1["Hank"] = 30

Remove entry (with key):

del Dictionary_1["Hank"]

Iterate through a dictionary External Link

.item : allows to iterate through the key and values simultaneously. When we iterate, by definition, we transform the dict in a list

STRING MANITUPATION FOR LOG PARSING To interleave string and variables, we can use the plus '+' sign or a more variable expression with the '%':

'The cost of %s is %s dollars' %(concept, amount)

Generate dictionary: This is to extract space separated elements and insert them in a dictionary. See that no append command is needed here:

if line.startswith("name"):
  words=line.split()
  existing_names[words[1]]=words[2]

If we want to extract one column in a space separated string, we use this:

sn = sn+text.split(" ")[-1]    # if possite, column counted from the left;  If negative, counted from the right (end)

RUN BASH FROM PYTHON
We use subprocess: http://www.cyberciti.biz/faq/python-execute-unix-linux-command-examples/ If we want to run a python script in a new bash session (cli window, we call it as follow (for linux)):

import subprocess
pid = subprocess.Popen(args=[
"gnome-terminal", "--command=python /home/jotasancent/Documents/PycharmProjects/my_linux_boot1/my_linux_boot1.py"]).pid
print pid

For example with usage of interactive command, file management and bash commands invokation, see: http://ec2-52-28-93-253.eu-central-1.compute.amazonaws.com/doku.php?id=scripting:python:interactive_bash_commands

MODULES
IMPORT MODULES - import

FUNCTIONS
def means declaring functions. Then space and name, brackets with the variable inside plus colon.
Call the function differently from the variables. Function to start with upper case.

def func_name
(optional_args):
  statements
  return vale

TEMPLATE

#!/usr/bin/envpython
# Imports
import random
# Module Constants and Global/Module Variables
FORTUNES= [
    "There is a good chance your code will work, eventually.",
    "I see Network DevOps in your future."
]
# Module Functions and Classes
def generate_fortune() -> str:
    return random.choice(FORTUNES)
    
def main():
    print(generate_fortune())
    
# Check to see if this file is the "__main__" script being executed
if __name__ == '__main__':
    main()

PYTHON DEBUGGER (built in):

next : executes the function completely
step : steps into the functions
skip :

https://docs.python.org/3/library/pdb.html

NETWORKING
Modules

socket — Low-level networking interface - https://docs.python.org/2/library/socket.html

Good example of socket server, client… http://www.binarytides.com/python-socket-server-code-example/

asyncore — Asynchronous socket handler https://docs.python.org/2/library/asyncore.html

IP MANIPULATION (ipaddress module)
ipaddress module (library) with plenty of examples: https://docs.python.org/3/library/ipaddress.html

FACTORY FUNCTIONS:

ipaddress.ip_address(address)
ipaddress.ip_network(address, strict=True)
ipaddress.ip_interface(address). Example:

>>> from ipaddress import ip_network, ip_address, IPv4Address
# Check IP in subnet
>>> ipaddress.ip_address('192.168.0.1') in ipaddress.ip_network('192.168.0.0/24')
True
# Check subnet in supernet
>>> ip_network('192.168.0.0/26').subnet_of(ip_network('192.168.0.0/24'))
True

for arg in sys.argv[1:]:
    addr = ipaddress.ip_interface(arg)
    print("address =", addr)
    print("network =", addr.network)
    if addr.version == 4:
        print("netmask =", addr.netmask)
        print("broadcast =", addr.network.broadcast_address)
    print()

To find whether or not an IP is in a network:

an_address = ipaddress.ip_address('192.168.0.1')
a_network = ipaddress.ip_network('192.168.0.0/24')

__author__ = 'jotasancent'
import ipaddress
print(ipaddress.ip_address('192.168.0.1').version)

__author__ = 'jotasancent'
import ipaddress
# This is to declare the clas
ip1=ipaddress.IPv4Address('110.168.0.1')
ip2=ipaddress.IPv4Address('172.16.0.10')
if ip1 > ip2:
    print("true")
else:
    print("false")

ipaddress.ip_address("127.0.0.1").is_multicast
False
ipaddress.ip_address("127.0.0.1").is_private
True

OBJECT ORIENTED PROG.

We can use oop extensively in SDKs:
Follow the example in ./PLURALSIGHT/automating-cisco-dna-center-operations-using-apis/02/demos/dnac_requester.py

Inside a file( in this case dnac_requester.py)we define a class
First thing, we type the contructor: def (double underscore)init(self, host, username..)
Inside the constructor the variables are referred like this : self.host = host

The Key part comes here:
In a different file from where we defined the class, we import the class like this:
- from classfile import TheClass
now we can instantiate the class we defined above like this:
- dnac = DNACRequester(host=“10.10.20.85”, username=“admin”…)
So we can use it later on like this:
- add_resp = dnac.req(“dna/intent/api/v1/site”, method=“post”, jsonbody=data)
- Important to note that the self vaalues are now defines and are part of the instantiated class!

HOW CLASSES WORK:

Class is a Template of an Instance. Class is made up of functions. OOP:we define attributes for the objects defined in classes

class Animal:
  __name = None

double underscore is for private variables, Here they are ATTRIBUTES OF THE CLASS

Then I can define funcions inside the class, if they need to refer to the class, I use the reserved variable self:. We have object instances of a class, then they are used, for instance in a function. For the function to refer to variables of the Calling OBJECT, we use that Keyword!

def set_name(self, name):
 self.__name = name  # this is called Encapsulation

class constructors (called to initialize an object). Is no more than a function with special variables like this:

  def __init__(self, name, height, weight, sound):
   self.__name = name
   self.__height = height

Then we define setters and getters for all the variables. setters and getters are functions used to transfer values to the objects

  def get_sound(self):
      return self.__sound

Inheritance means whatever coming from another class has their variables and functions (among others, the setters and the getters)

  super (Dog, self).__init__(name, height, weight, sound)

super is from super-class (from which one is inheriting)

Method overloading different task based on the attributes that were sent in. i.e.: if this attribute is sent in, do this, if not differently

Polymorphism refer to objects as their super class and then automatically have the correct functions called

Classes have constructors, defined as “def init(self, radius)
- abstract classes do not have constructors

CLASS DECORATORS @

When we decorate a function with a class, that function becomes an instance of the class.
We can add functionality to the function by defining methods in the decorating class.
This can all be achieved without modifying the original function source code.
To decorate a function with a class, we must use @syntax.

NETMIKO NOTES

Is bulid on top of paramiko (which is more focused to servers)
For a comprenhensive list of netmiko examples see this LINK

From this Link. Note that we need to use pip3 IF WE ARE USING PYTHON3 3

pip3 install netmiko

ConnectHandler requires 4 arguments as follows:

!
device = { 
    'device_type': 'cisco_ios',
    'host': host,
    'username': 'pyclass',
    'password': getpass,
} 
net_connect = ConnectHandler(**device)

Types : https://github.com/ktbyers/netmiko/blob/master/netmiko/ssh_dispatcher.py#L70

Each device in netmiko is a json entry. In it, one fields says what type of device we are connecting to from the list netmiko library is designed to access to.
No worries because we can generate it from a csv. https://www.linuxjournal.com/content/use-case-network-automation
“from a simple text file in the CSV format. I include the csv module, so I can use the csv.Dictreader function, which returns CSV fields as a Python dictionary…”

https://github.com/ktbyers/netmiko/blob/master/netmiko/ssh_dispatcher.py#L70

..
Then to apply this change to a set of devices, we make a list with all those json devices: https://pynet.twb-tech.com/blog/automation/netmiko.html
Next, I need to create a Python list that includes all of these devices: all_devices = [cisco3, cisco_asa, cisco_xrv, arista8, hp_procurve, juniper_srx]

NAPALM NOTES (paramiko notes):
Main page: https://github.com/napalm-automation/napalm Good examples (and nornir) https://cyruslab.net/2020/03/01/pythonfirst-use-of-napalm/

Install:

# these steps: https://www.gns3.com/community/featured/install-netmiko-paramiko-pexpect  (python3.7 > 3.6)
pip3 install --ignore-installed PyYAML
python3.6 -m pip install napalm

IOS support: External Link

PYCHARM

Place the caret on the desired line of the source code (note there should be code on that line). We can set a dummy function like time.sleep(5.5)
Left click or ctrl-shift-alt-f8

To see inline documentation about Any element. Select the word and press:

Ctrl-Q

Change python interpreter (per project basis)

File > Settings

Comment blocks:

Code | Comment with Block Comment.

LOOPS
WHILE LOOP. We can set an upper limit and stop once we get there:

while count !=10:
  ...
count += 1

API

xml: encoding useful to manipulate data objects
json: encoding too. Based on javascript and resembling a lot to python dictionaries.
rest: instead of python, we will used rest to manipulate objects. Can be complemented with python
wsdl and soap: These are schemas and data models. Useful to model devices. soap is the predecessor of rest. Can be used to avoid the overhead of rest.

JSON
For simple and quick queries on the cli, use jq tool:

jq  '.prefixes[] | select(.region=="us-east-1")' < ipranges.json > us-east-1_prefixes.txt
jq  -r '.devices[] | .hostname, .version '  # extracts values for keys 'devices' and 'hostname' under title 'devices'  ( we use it with postman response.json )

MANIPULATE JSON OUTPUT:
Many cloud api responses are just a dictionary with a key named data and then a big element structure which is a list of dictionaries.
To start accessing the data we do resources = data[“resources”] so now resources is a big list and we can use for/if and things like end_result.append(res_elements[“instances”][0][“attributes”][“network_security_group_id”]) to access element in the structure.

json ~= python dictionaries.

BELOW typical work path:

Convert API response to json
Store json data into a file
Retrieve the file into dict format to work on it

# Convert the api response to dictionary and store it
json_data = response.json()
with open("backend_results.json", "w") as fp:
    json.dump(json_data,fp)

POST method. More info here

# data to be sent to api
data = {'api_dev_key':API_KEY,
        'api_option':'paste',
        'api_paste_code':source_code,
        'api_paste_format':'python'}
r = requests.post(url = API_ENDPOINT, data = data)

JSON.DUMPS (dumps is for pretty format, dump is to save json into file):

print (json.dumps(json_data["response"], indent=2))

with open("outputs/result.json", "w") as handle:
  json.dump(self.data, handle, indent=2)

JSON.LOADS https://www.geeksforgeeks.org/json-loads-in-python/
To parse a valid JSON string and convert it into a Python Dictionary. It is mainly used for deserializing native string, byte, or byte array which consists of JSON data into Python Dictionary.

x = """{
	"Name": "Jennifer Smith",
	"Contact Number": 7867567898,
	"Email": "jen123@gmail.com",
	"Hobbies":["Reading", "Sketching", "Horse Riding"]
	}"""
y = json.loads(x)
print(y)

REST
Tree-based. ACI:
Anything can reach anything. What is stopping people with talking each other? Contracts.
No routing protocol in is-is. Except a (hidden is-is).
Apic talks southbound openflow/opflex. Northbound talks rest
In the gui: welcome admin > debug info ←- to know where I'm in the gui at any time!. The DN.
Browse the schema. APIC object browser!
Push via DN. Extract via a class. Use Postman. Create a tenant.
API inspector: as we click on the guy, all the actions get recorded on the inspector. Create a tenant.
ACI/SDN python way.

F5
To see the wsld and manage the f5 via python: https://10.30.253.141/iControl/iControlPortal.cgi
Check out example in : https://82.46.188.134/dokuwiki/www/dokuwiki/doku.php?id=scripting:python_scripts:f5-exaple1.py

ENTER DATA ON THE DB - POSTGRES SQL -
f5 –> Postgres
Also when data gets quite complex to fit in a text file, its time to dump it into a table.
See sql_database_f5_to_a10_1.py

import psycopg2
import openpyxl
...
conn = psycopg2.connect(DSN)

MAIL ALERT SENT
TODO
FIND FILES AND PUT TOGETHER A MAIL
TODO
PROCESS POSTMAN OUTPUT
TODO OAuth usage
CSV PROCESSING
panda vs csv modules
PEXPECT AND NETMIKO/PARAMIKO*
TODO
ACCESS TO PYTHON DOCUMENTATION ONLINE
EXTRACT DATA FROM SSH OUTPUT
ERROR HANDLING/EXCEPTIONS After the except <type> we (indent) the actions to be made. Classically is a print with a human readable error. Useful for debugging. Typical errors to catch:

File management: IOError,

ATOM NOTES
Atom flight manual

column-select : Alt-Shift-Down
sublime-select :
https://atom.io/packages/remote-sync

Install new packager: Control + comma > Install
anything with apm <whatever> is executed in the linux cli
script is the atom package to execute python. Open an integrated terminal, use this External Link to run python scripts and pass arguments.
- Execute : ctrl-shift p # enters atom commands
- Execute : ctrl-shift b # executes the code
- ctrl+` opens linux cli
- Comment out blocks: ctrl + /
- Split : right-click and choose “Split left” or “Split right”.
- Highlight https://atom.io/packages/quick-highlight
  - ctrl-p
    - quick-highlight:toggle toggle highlight for selected or cursor word.
    - quick-highlight:clear clear all highlight.
- Open terminal ( needs : platformio-ide-terminal )
  - alt-shift-t (to see all options: ctrl-p and type platformio )

Split screen (panes): Ctrl+K Up/Down/Left/Right
atom debugger. Install python-debugger AND autocomplete-python

alt-r: hide/show the debugger view
alt-shift-r: toggle breakpoint at the current line

Atom for ansible:

NGROK:

Demoing web sites without deploying
Building webhook consumers on your dev machine
Testing mobile apps connected to your locally running backend
Stable addresses for your connected devices that are deployed in the field
Running personal cloud services from your home

DUNDER AND SPECIAL METHODS

print(dir(int))    # shows all methods for int, included the dunder ones

CODING NOTES

https://app.coderpad.io/dashboard/questions/examples/
Use a 'list of unchecked cells''… At the end, the index of the only unchecked cell will be the missing page number.

  pages_qty = len(page_numbers) + 1
  pages_checked = [False] * pages_qty.   # This create a list with all elements 'False'

WELL KNOWN ALGORITMS:

Breadth-First Search algorithms
Hash table implementation
Palindrome checker
Moving-window average algorithm
Sorted lists of numbers

dokucama

User Tools

Site Tools

Page Tools