This is an old revision of the document!
PYTHON
Python virtual environment: Isolated “environments” where specific versions of Python can be installed along with independent sets of libraries and dependencies.
This is to create a venv, activate it and manage it with vscode:
python3 -m venv enauto # To create an empty folder with all python r cd <folder> source bin/activate # be sure the interpreter is now selected 'venv' on the low right bar in vscode python3 -m pip install requests # we can install stuff just in this venv and also execute from vscode source my_virtual_env/bin/activate # activates deactivate # once we are done, deactivate the environment # In vscode you might require to deselect the venv manually (+"ctrl+shift+P"> python: Clear Workspace Interpreter Settings)
FLAGS AND KNOBS:
- raw flag : raw_dummy_str = r“This is a \n raw string” # treat the backlash as a backlash
- strip method : string.strip([chars]) # copy of the string with both leading and trailing characters removed
Interactive input data/INPUT FROM USERS
nb = input('Choose a number: ')
PRINT NOTES:
Link
Use , to separate strings and variables while printing by using print f:
print(f'I am {nb}') # mood is a Variable
!
print("{name}: {ip} {netmask}".format(
name=interface["name"],
ip=interface["ietf-ip:ipv4"]["address"][0]["ip"],
netmask=interface["ietf-ip:ipv4"]["address"][0]["netmask"],
f or F ^^ used before the string (quote or double quotes) is 'new' string formatting in python. Much simpler External Link
TABLES
To present data use prettytable
PASS ARGUMENTS
Command line argument (ie whatever is after the script invocation). External Link
for arg in sys.argv[1:]:
addr = ipaddress.ip_interface(arg)
print("address =", addr)
print("network =", addr.network)
The below is intended to be reuse by many other modules!:
http://stackoverflow.com/questions/419163/what-does-if-name-main-do
import sys
def main(argv1, argv2, argv3):
print "this is the first variable: ", argv1
print "this is the second variable: ", argv2
print "this is the third variable: ", argv3
if __name__ == "__main__":
main(sys.argv[1], sys.argv[2], sys.argv[3])
COLLECTIONS
- Dictionary methods(more)
- car.update({“color”: “White”}) # adds the element between brackets to the dictionary
car - car.get('color')
CONDITIONALS:
- if ..: elif : else:
- we could have if without any of the other two.
- Too many if is a sign of poor coding. We can reduce them with: switch , functions , object-oriented programming
- for name in names: <indented-body-of-instructions>
- while <condition> : and-then-the-rest-indented
ITERATIONS:
Parallel iteration
Lists (zip lists intertwined like jacket list), then access both elements simultaneously:
sizes = ['small', 'medium', 'large']
animals = ['frog', 'wolf', 'elephant']
for size, animal in zip(sizes, animals): # read this from right to left (first zip lists and the go through it):
print(f'Isee a {size} {animal}')
TO ITERATE THROUGH A DICTIONARY: - .items
animals = {'small': 'frog','medium': 'wolf','large': 'elephant'}
for size, animal in animals.items():
print(f'I see a {size} {animal}')
TO GO THROUGH A LIST AND HAVE A COUNTER - enumerate():
- Used when we want to go through a list and also have a counter.
- The
enumerate()function allows iterable objects and returns an enumerate object in the form of a tuple containing the index and the value
for page_num, is_checked in enumerate(pages_checked, 1): # pages_checked is a list of 'False' and 'True' ; page_num is the position ; is_checked is one of the list elements
if not is_checked:
return page_num
LIST COMPREHENSION:
Easiest usage is to replace a for-if section:
sample = [ 'sfsdf', 'ggg' , 'wrwer'] print([ r for r in sample if 's' in r ])
Also for nested for loops (here it flattening lists):
flat_list = [item for sublist in l for item in sublist]
Main usage is for creating a list based on existing list:External Link
[expression for item in list] h_letters = [ letter for letter in 'human' ] print( h_letters)
We can now identify where list comprehensions are used. If you noticed, human is a string, not a list. This is the power of list comprehension. It can identify when it receives a string or a tuple and work on it like a list.
\\External Link
List comprehension example:
REGEX
Resources here1 and here2
https://panda314159.net/doku.php?id=scripting:python_scripts:regex1.py
regex search ways in python: MATCH, SEARCH, FINDALL, and SUB
FINDALL: Finds all instances of the matched string. All the occurrences go inside a LIST. findall() matches all occurrences of a pattern, not just the first one. We can also use compile
re.findall( 'pattern', 'did you find one pattern or many patterns') ['pattern', 'pattern']
Example: find any serial number 8 characters long after the pattern: “Caller pc :0x”:
file = fileh.read()
serials = re.findall(r'Caller pc :0x([a-zA-Z0-9]{8}\s' , file)
SUB: Can replace a parameter.
First you need to find the parts of the string that need to be rewritten (you do this with re.sub). Then you rewrite that parts.
re.sub( 'pattern', 'turnip', 'did you find one pattern or many patterns') 'did you find one turnip or many turnips'
SEARCH:
name_regex = re.compile(r"^vrf\s+(?P<name>\S+)")
name_match = name_regex.search(vrf)
sub_dict = {}
vrf_dict = {name_match.group("name"): sub_dict}
MATCH:
- Compile the expressions
- Then used the compiled variables as functions to do the matching.
- If we see things like
name_regex = re.compile(r“^vrf\s+(?P<name>\S+)”)'name' is a matching group (ie multiple matches go there and can be accessed later as a dict or list.
regex1 = re.compile("^192\.168\.222\.\d{1,3}$")
regex1.match(var_to_match)
<_sre.SRE_Match object; span=(0, 13), match='192.168.222.5'> # no result would mean no match
searches at the beginning of the string. Dont use match as it just matches at the Beginning of the string.
vrf = re.match( r'hostname (.*)', file) print "The hostname of the Router is:\t", vrf.group(1)
LISTS -square brackets-
Important because we can 'feed' a for loop with a list. No need of index or counter.
for name in namelist: print name
To add elements to a list we can append them “.append”
a += 1 to increment a variable (counter)
OOP:
Put def description code between """ Here """
That way it appears in the function help. To find the help section of a function we just do help(functionname.
It could be we just know the method but we don't know the function name. EG: we have a var=“string” and we want to know the var.strip(?) method help. In that case we do help(var). Thet will point to an error as below which in turn will show us the way:
HELP IN PYTHON (INLINE DOCUMENTATION):
help(var) print(dir(int)) # shows all methods for int, included the dunder ones help(str.strip) # this is help for the method .strip (str is the py name of string (built-in type) useful_function.__annotations__[‘return’] To see the the methods available for an object: ''dir (obj)''. # In python we have: objects dot method. To see the online help we do ''help(module)''
Class object polyphormism (so same function can be used for all animals). Abstract classes , not to be instantiated !?
FILE NOTES:
The correct way of opening files in python is by using with. Reason is that files are automatically closed. With the old method, even if we think we close them, sometimes, due to unexpected exceptions, they can end up not closed and corrupted.
Link1 , Link2
To create a list from a file:
with open('/var/tmp/list1', 'r') as f:
myNames = [line.strip() for line in f]
To show the current folder path and open a file in it. own directory / own folder.
import os
with open(os.path.join(sys.path[0], "my_file.txt"), "r") as f:
print(f.read())
Most recommended and advanced than os for folder for path manipulation is pathlib. See this LINK.
The main difference is the returned type of object (now is PosixPath instead of string, which gives more possibilities).
from pathlib import Path Path.cwd()
IMPORTANT: In python, we don't need to 'load' the file in a variable to manipulate it. We can just use the file description to iterate on the elements an use different file object methods to extract info on demand.
Useful in file path and name manipulation is the function .strip() without parameters. Like that, IT REMOVES END WHITE SPACES.
See this entry on reading-and-writing-files-in-python
Handler = open(file.txt)
Handler.read, write, delete, close.....
Handler.truncate() # means Delete the file (for that handler)
# To create a list (transforming value into list)
list('Hello')
# To create a list from a text file:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
# To access to the second element to the end var_list[-2] # to //slice// a part of the list var_list[1:3] # beginning implicit var_list[:3] # delete element del var_list[-2]
# Is the element in the list? 'element' in list
With methods
list_var.index('value_to_find')
list_var.insert(pos,'value')
list_var.append('value_to_append')
Configuration generated from list read. Multi-line template. Python triple quote usage example:
with open('C:/Users/jsantos/Desktop/dser', 'r') as f:
dser = [line.strip() for line in f]
for i in range(len(dmer)):
interface = i+1
print("""
interface GigabitEthernet4/""" + str(interface) +"""
description MER """ + dmer[i] +""" PXXX Desk X (gi4/""" + str(interface) +""")
switchport
switchport access vlan 20
switchport trunk native vlan 998
switchport mode access
switchport voice vlan 34
platform qos vlan-based
platform qos trust cos
no snmp trap link-status
macro description datavoice-port
no cdp enable
spanning-tree portfast edge
spanning-tree bpduguard enable
!
""")
Slicing lists
print namelist[:2] # prints up to but not including index 2. So 0 and 1.
Updating list
.append
.pop(0) # number is number of elements to be 'popped'
FOR LOOP WITH LISTS
for i in list(range(5)): print(i)
This is a very useful way of accessing elements of a list with arbitrary length
for i in range(len(supplies)): supplies[i]
DICTIONARY-Curly braces or the set() function-
stuff = {'name': 'Zed', 'age': 39, 'height': 6 * 12 + 2}
Print entry (with key):
print stuff1["age"]
Add entry (with key):
Dictionary_1["Hank"] = 30
Remove entry (with key):
del Dictionary_1["Hank"]
Iterate through a dictionary External Link
.item: allows to iterate through the key and values simultaneously. When we iterate, by definition, we transform the dict in a list
STRING MANITUPATION FOR LOG PARSING
To interleave string and variables, we can use the plus '+' sign or a more variable expression with the '%':
'The cost of %s is %s dollars' %(concept, amount)
Generate dictionary: This is to extract space separated elements and insert them in a dictionary. See that no append command is needed here:
if line.startswith("name"):
words=line.split()
existing_names[words[1]]=words[2]
If we want to extract one column in a space separated string, we use this:
sn = sn+text.split(" ")[-1] # if possite, column counted from the left; If negative, counted from the right (end)
RUN BASH FROM PYTHON
We use subprocess: http://www.cyberciti.biz/faq/python-execute-unix-linux-command-examples/
If we want to run a python script in a new bash session (cli window, we call it as follow (for linux)):
import subprocess pid = subprocess.Popen(args=[ "gnome-terminal", "--command=python /home/jotasancent/Documents/PycharmProjects/my_linux_boot1/my_linux_boot1.py"]).pid print pid
For example with usage of interactive command, file management and bash commands invokation, see:
http://ec2-52-28-93-253.eu-central-1.compute.amazonaws.com/doku.php?id=scripting:python:interactive_bash_commands
MODULES
IMPORT MODULES - import
FUNCTIONS
def means declaring functions. Then space and name, brackets with the variable inside plus colon.
Call the function differently from the variables. Function to start with upper case.
def func_name (optional_args): statements return vale
TEMPLATE
#!/usr/bin/envpython
# Imports
import random
# Module Constants and Global/Module Variables
FORTUNES= [
"There is a good chance your code will work, eventually.",
"I see Network DevOps in your future."
]
# Module Functions and Classes
def generate_fortune() -> str:
return random.choice(FORTUNES)
def main():
print(generate_fortune())
# Check to see if this file is the "__main__" script being executed
if __name__ == '__main__':
main()
PYTHON DEBUGGER (built in):
- next : executes the function completely
- step : steps into the functions
- skip :
https://docs.python.org/3/library/pdb.html
NETWORKING
Modules
socket — Low-level networking interface - https://docs.python.org/2/library/socket.html
Good example of socket server, client… http://www.binarytides.com/python-socket-server-code-example/
asyncore — Asynchronous socket handler https://docs.python.org/2/library/asyncore.html
IP MANIPULATION (ipaddress module)
ipaddress module (library) with plenty of examples: https://docs.python.org/3/library/ipaddress.html
FACTORY FUNCTIONS:
- ipaddress.ip_address(address)
- ipaddress.ip_network(address, strict=True)
- ipaddress.ip_interface(address). Example:
>>> from ipaddress import ip_network, ip_address, IPv4Address
# Check IP in subnet
>>> ipaddress.ip_address('192.168.0.1') in ipaddress.ip_network('192.168.0.0/24')
True
# Check subnet in supernet
>>> ip_network('192.168.0.0/26').subnet_of(ip_network('192.168.0.0/24'))
True
for arg in sys.argv[1:]:
addr = ipaddress.ip_interface(arg)
print("address =", addr)
print("network =", addr.network)
if addr.version == 4:
print("netmask =", addr.netmask)
print("broadcast =", addr.network.broadcast_address)
print()
To find whether or not an IP is in a network:
an_address = ipaddress.ip_address('192.168.0.1')
a_network = ipaddress.ip_network('192.168.0.0/24')
__author__ = 'jotasancent'
import ipaddress
print(ipaddress.ip_address('192.168.0.1').version)
__author__ = 'jotasancent'
import ipaddress
# This is to declare the clas
ip1=ipaddress.IPv4Address('110.168.0.1')
ip2=ipaddress.IPv4Address('172.16.0.10')
if ip1 > ip2:
print("true")
else:
print("false")
ipaddress.ip_address("127.0.0.1").is_multicast
False
ipaddress.ip_address("127.0.0.1").is_private
True
OBJECT ORIENTED PROG.
- We can use oop extensively in SDKs:
- Follow the example in
./PLURALSIGHT/automating-cisco-dna-center-operations-using-apis/02/demos/dnac_requester.py
- Inside a file( in this case dnac_requester.py)we define a
class - First thing, we type the contructor: def (double underscore)init(self, host, username..)
- Inside the constructor the variables are referred like this :
self.host = host
- The Key part comes here:
- In a different file from where we defined the class, we import the class like this:
from classfile import TheClass
- now we can instantiate the class we defined above like this:
dnac = DNACRequester(host=“10.10.20.85”, username=“admin”…)
- So we can use it later on like this:
- add_resp = dnac.req(“dna/intent/api/v1/site”, method=“post”, jsonbody=data)
- Important to note that the self vaalues are now defines and are part of the instantiated class!
HOW CLASSES WORK:
Class is a Template of an Instance. Class is made up of functions.
OOP:we define attributes for the objects defined in classes
class Animal: __name = None
double underscore is for private variables, Here they are ATTRIBUTES OF THE CLASS
Then I can define funcions inside the class, if they need to refer to the class, I use the reserved variable self:. We have object instances of a class, then they are used, for instance in a function. For the function to refer to variables of the Calling OBJECT, we use that Keyword!
def set_name(self, name): self.__name = name # this is called Encapsulation
class constructors (called to initialize an object). Is no more than a function with special variables like this:
def __init__(self, name, height, weight, sound): self.__name = name self.__height = height
Then we define setters and getters for all the variables. setters and getters are functions used to transfer values to the objects
def get_sound(self):
return self.__sound
Inheritance means whatever coming from another class has their variables and functions (among others, the setters and the getters)
super (Dog, self).__init__(name, height, weight, sound)
super is from super-class (from which one is inheriting)
Method overloading different task based on the attributes that were sent in. i.e.: if this attribute is sent in, do this, if not differently
Polymorphism refer to objects as their super class and then automatically have the correct functions called
- Classes have constructors, defined as “def init(self, radius)
- abstract classes do not have constructors
CLASS DECORATORS @
- When we decorate a function with a class, that function becomes an instance of the class.
- We can add functionality to the function by defining methods in the decorating class.
- This can all be achieved without modifying the original function source code.
- To decorate a function with a class, we must use @syntax.
NETMIKO NOTES
Is bulid on top of paramiko (which is more focused to servers)
For a comprenhensive list of netmiko examples see this LINK
From this Link. Note that we need to use pip3 IF WE ARE USING PYTHON3 3
pip3 install netmiko
ConnectHandler requires 4 arguments as follows:
!
device = {
'device_type': 'cisco_ios',
'host': host,
'username': 'pyclass',
'password': getpass,
}
net_connect = ConnectHandler(**device)
Types : https://github.com/ktbyers/netmiko/blob/master/netmiko/ssh_dispatcher.py#L70
Each device in netmiko is a json entry. In it, one fields says what type of device we are connecting to from the list netmiko library is designed to access to.
No worries because we can generate it from a csv. https://www.linuxjournal.com/content/use-case-network-automation
“from a simple text file in the CSV format. I include the csv module, so I can use the csv.Dictreader function, which returns CSV fields as a Python dictionary…”
https://github.com/ktbyers/netmiko/blob/master/netmiko/ssh_dispatcher.py#L70
..
Then to apply this change to a set of devices, we make a list with all those json devices: https://pynet.twb-tech.com/blog/automation/netmiko.html
Next, I need to create a Python list that includes all of these devices: all_devices = [cisco3, cisco_asa, cisco_xrv, arista8, hp_procurve, juniper_srx]
NAPALM NOTES (paramiko notes):
Main page: https://github.com/napalm-automation/napalm
Good examples (and nornir) https://cyruslab.net/2020/03/01/pythonfirst-use-of-napalm/
Install:
# these steps: https://www.gns3.com/community/featured/install-netmiko-paramiko-pexpect (python3.7 > 3.6) pip3 install --ignore-installed PyYAML python3.6 -m pip install napalm
IOS support: External Link
PYCHARM
- Place the caret on the desired line of the source code (note there should be code on that line). We can set a dummy function like time.sleep(5.5)
- Left click or ctrl-shift-alt-f8
To see inline documentation about Any element. Select the word and press:
Ctrl-Q
Change python interpreter (per project basis)
File > Settings
Comment blocks:
Code | Comment with Block Comment.
LOOPS
WHILE LOOP. We can set an upper limit and stop once we get there:
while count !=10: ... count += 1
API
- xml: encoding useful to manipulate data objects
- json: encoding too. Based on javascript and resembling a lot to python dictionaries.
- rest: instead of python, we will used rest to manipulate objects. Can be complemented with python
- wsdl and soap: These are schemas and data models. Useful to model devices. soap is the predecessor of rest. Can be used to avoid the overhead of rest.
JSON
For simple and quick queries on the cli, use jq tool:
jq '.prefixes[] | select(.region=="us-east-1")' < ipranges.json > us-east-1_prefixes.txt jq -r '.devices[] | .hostname, .version ' # extracts values for keys 'devices' and 'hostname' under title 'devices' ( we use it with postman response.json )
MANIPULATE JSON OUTPUT:
Many cloud api responses are just a dictionary with a key named data and then a big element structure which is a list of dictionaries.
To start accessing the data we do resources = data[“resources”] so now resources is a big list and we can use for/if and things like end_result.append(res_elements[“instances”][0][“attributes”][“network_security_group_id”]) to access element in the structure.
json ~= python dictionaries.
BELOW typical work path:
- Convert API response to json
- Store json data into a file (not pretty)
- Retrieve the file into dict format to work on it
# Convert the api response to dictionary and store it
json_data = response.json()
with open("backend_results.json", "w") as fp:
json.dump(json_data,fp)
POST method. More info here
# data to be sent to api
data = {'api_dev_key':API_KEY,
'api_option':'paste',
'api_paste_code':source_code,
'api_paste_format':'python'}
r = requests.post(url = API_ENDPOINT, data = data)
JSON.DUMPS (dumps is for pretty format, dump is to save json into file):
print (json.dumps(json_data["response"], indent=2))
with open("outputs/result.json", "w") as handle:
json.dump(self.data, handle, indent=2)
JSON.LOADS https://www.geeksforgeeks.org/json-loads-in-python/
To parse a valid JSON string and convert it into a Python Dictionary. It is mainly used for deserializing native string, byte, or byte array which consists of JSON data into Python Dictionary.
x = """{
"Name": "Jennifer Smith",
"Contact Number": 7867567898,
"Email": "jen123@gmail.com",
"Hobbies":["Reading", "Sketching", "Horse Riding"]
}"""
y = json.loads(x)
print(y)
REST
Tree-based.
ACI:
Anything can reach anything. What is stopping people with talking each other? Contracts.
No routing protocol in is-is. Except a (hidden is-is).
Apic talks southbound openflow/opflex. Northbound talks rest
In the gui: welcome admin > debug info ←- to know where I'm in the gui at any time!. The DN.
Browse the schema. APIC object browser!
Push via DN. Extract via a class. Use Postman. Create a tenant.
API inspector: as we click on the guy, all the actions get recorded on the inspector. Create a tenant.
ACI/SDN python way.
F5
To see the wsld and manage the f5 via python: https://10.30.253.141/iControl/iControlPortal.cgi
Check out example in : https://82.46.188.134/dokuwiki/www/dokuwiki/doku.php?id=scripting:python_scripts:f5-exaple1.py
ENTER DATA ON THE DB - POSTGRES SQL -
f5 –> Postgres
Also when data gets quite complex to fit in a text file, its time to dump it into a table.
See sql_database_f5_to_a10_1.py
import psycopg2 import openpyxl ... conn = psycopg2.connect(DSN)
MAIL ALERT SENT
TODO
FIND FILES AND PUT TOGETHER A MAIL
TODO
PROCESS POSTMAN OUTPUT
TODO
OAuth usage
CSV PROCESSING
panda vs csv modules
PEXPECT AND NETMIKO/PARAMIKO*
TODO
ACCESS TO PYTHON DOCUMENTATION ONLINE
EXTRACT DATA FROM SSH OUTPUT
ERROR HANDLING/EXCEPTIONS
After the except <type> we (indent) the actions to be made. Classically is a print with a human readable error. Useful for debugging.
Typical errors to catch:
- File management: IOError,
ATOM NOTES
Atom flight manual
column-select: Alt-Shift-Downsublime-select:
- Install new packager: Control + comma > Install
- anything with apm <whatever> is executed in the linux cli
- script is the atom package to execute python. Open an integrated terminal, use this External Link to run python scripts and pass arguments.
- Execute : ctrl-shift p # enters atom commands
- Execute : ctrl-shift b # executes the code
- ctrl+` opens linux cli
- Comment out blocks: ctrl + /
- Split : right-click and choose “Split left” or “Split right”.
- Highlight https://atom.io/packages/quick-highlight
- ctrl-p
- quick-highlight:toggle toggle highlight for selected or cursor word.
- quick-highlight:clear clear all highlight.
- Open terminal ( needs : platformio-ide-terminal )
- alt-shift-t (to see all options: ctrl-p and type platformio )
Split screen (panes): Ctrl+K Up/Down/Left/Right
atom debugger. Install python-debugger AND autocomplete-python
- alt-r: hide/show the debugger view
- alt-shift-r: toggle breakpoint at the current line
Atom for ansible:
NGROK:
- Demoing web sites without deploying
- Building webhook consumers on your dev machine
- Testing mobile apps connected to your locally running backend
- Stable addresses for your connected devices that are deployed in the field
- Running personal cloud services from your home
DUNDER AND SPECIAL METHODS
print(dir(int)) # shows all methods for int, included the dunder ones
CODING NOTES
- Use a 'list of unchecked cells''… At the end, the index of the only unchecked cell will be the missing page number.
pages_qty = len(page_numbers) + 1 pages_checked = [False] * pages_qty. # This create a list with all elements 'False'
WELL KNOWN ALGORITMS:
- Breadth-First Search algorithms
- Hash table implementation
- Palindrome checker
- Moving-window average algorithm
- Sorted lists of numbers
