Intermediate Python

Content
Lists - Advanced Python 01 2
Tuples - Advanced Python 02 8
Dictionary - Advanced Python 03 14
Sets - Advanced Python 04 19
Strings - Advanced Python 05 26
Collections - Advanced Python 06 32
Itertools - Advanced Python 07 37
Lambda Functions - Advanced Python 08 41
Exceptions And Errors - Advanced Python 09 44
Logging - Advanced Python 10 49
JSON - Advanced Python 11 57
Random Numbers - Advanced Python 12 64
Decorators - Advanced Python 13 69
Generators - Advanced Python 14 76
Threading vs Multiprocessing - Advanced Python 15 81
Multithreading - Advanced Python 16 85
Multiprocessing - Advanced Python 17 93
Function arguments - Advanced Python 18 103
The Asterisk (*) operator - Advanced Python 19 110
Shallow vs Deep Copying - Advanced Python 20 114
Context Managers - Advanced Python 21 118
1
Lists - Advanced Python 01
List is a collection data type which is ordered and mutable.
Unlike Sets, Lists allow duplicate elements.

List is a collection data type which is ordered and mutable.

Unlike Sets, Lists allow duplicate elements. They are useful for
preserving a sequence of data and further iterating over it.
Lists are created with square brackets.

my_list = ["banana", "cherry", "apple"]

Comparison of basic built-in collection data types

in Python:¶
• List is a collection which is ordered and mutable. Allows
duplicate members.
• Tuple is a collection which is ordered and immutable.
Allows duplicate members.
• Set is a collection which is unordered and unindexed. No
duplicate members.
• Dictionary is a collection which is unordered, mutable and
indexed. No duplicate members.
• Strings are immutable sequences of Unicode code points.
Creating a list¶
Lists are created with square brackets or the built-in list
function.

list_1 = ["banana", "cherry", "apple"]

print(list_1)

# Or create an empty list with the list function

list_2 = list()
print(list_2)
2
# Lists allow different data types
list_3 = [5, True, "apple"]
print(list_3)

# Lists allow duplicates

list_4 = [0, 0, 1, 1]
print(list_4)
['banana', 'cherry', 'apple']
[]
[5, True, 'apple']
[0, 0, 1, 1]

Access elements¶
You access the list items by referring to the index number.
Note that the indices start at 0.

item = list_1[0]
print(item)

# You can also use negative indexing, e.g -1 refers to the

last item,
# -2 to the second last item, and so on
item = list_1[-1]
print(item)
banana
apple

Change items¶
Just refer to the index number and assign a new value.

# Lists can be altered after their creation

list_1[2] = "lemon"
print(list_1)
['banana', 'cherry', 'lemon']

Useful methods¶
3
Have a look at the Python Documentation to see all list
methods: https://docs.python.org/3/tutorial/
datastructures.html

my_list = ["banana", "cherry", "apple"]

# len() : get the number of elements in a list

print("Length:", len(my_list))

# append() : adds an element to the end of the list

my_list.append("orange")

# insert() : adds an element at the specified position

my_list.insert(1, "blueberry")
print(my_list)

# pop() : removes and returns the item at the given

position, default is the last item
item = my_list.pop()
print("Popped item: ", item)

# remove() : removes an item from the list

my_list.remove("cherry") # Value error if not in the list
print(my_list)

# clear() : removes all items from the list

my_list.clear()
print(my_list)

# reverse() : reverse the items

my_list = ["banana", "cherry", "apple"]
my_list.reverse()
print('Reversed: ', my_list)

# sort() : sort items in ascending order

my_list.sort()
print('Sorted: ', my_list)
4
# use sorted() to get a new list, and leave the original
unaffected.
# sorted() works on any iterable type, not just lists
my_list = ["banana", "cherry", "apple"]
new_list = sorted(my_list)

# create list with repeated elements

list_with_zeros = [0] * 5
print(list_with_zeros)

# concatenation
list_concat = list_with_zeros + my_list
print(list_concat)

# convert string to list

string_to_list = list('Hello')
print(string_to_list)
Length: 3
['banana', 'blueberry', 'cherry', 'apple', 'orange']
Popped item: orange
['banana', 'blueberry', 'apple']
[]
Reversed: ['apple', 'cherry', 'banana']
Sorted: ['apple', 'banana', 'cherry']
[0, 0, 0, 0, 0]
[0, 0, 0, 0, 0, 'banana', 'cherry', 'apple']
['H', 'e', 'l', 'l', 'o']

Copy a list¶
Be careful when copying references.

list_org = ["banana", "cherry", "apple"]

# this just copies the reference to the list, so be

careful
list_copy = list_org
5
# now modifying the copy also affects the original
list_copy.append(True)
print(list_copy)
print(list_org)

# use copy(), or list(x) to actually copy the list

# slicing also works: list_copy = list_org[:]
list_org = ["banana", "cherry", "apple"]

list_copy = list_org.copy()
# list_copy = list(list_org)
# list_copy = list_org[:]

# now modifying the copy does not affect the original

list_copy.append(True)
print(list_copy)
print(list_org)
['banana', 'cherry', 'apple', True]
['banana', 'cherry', 'apple', True]
['banana', 'cherry', 'apple', True]
['banana', 'cherry', 'apple']

Iterating¶
# Iterating over a list by using a for in loop
for i in list_1:
print(i)
banana
cherry
lemon

Check if an item exists¶

if "banana" in list_1:
print("yes")
else:
print("no")
6
yes

Slicing¶
Access sub parts of the list wih the use of colon (:), just as with
strings.

# a[start:stop:step], default step is 1

a = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
b = a[1:3] # Note that the last index is not included
print(b)
b = a[2:] # until the end
print(b)
b = a[:3] # from beginning
print(b)
a[0:3] = [0] # replace sub-parts, you need an iterable
here
print(a)
b = a[::2] # start to end with every second item
print(b)
a = a[::-1] # reverse the list with a negative step:
print(a)
b = a[:] # copy a list with slicing
print(b)
[2, 3]
[3, 4, 5, 6, 7, 8, 9, 10]
[1, 2, 3]
[0, 4, 5, 6, 7, 8, 9, 10]
[0, 5, 7, 9]
[10, 9, 8, 7, 6, 5, 4, 0]
[10, 9, 8, 7, 6, 5, 4, 0]

List comprehension¶
A elegant and fast way to create a new list from an existing list.

List comprehension consists of an expression followed by a for

statement inside square brackets.

7
a = [1, 2, 3, 4, 5, 6, 7, 8]
b = [i * i for i in a] # squares each element
print(b)
[1, 4, 9, 16, 25, 36, 49, 64]

Nested lists¶
Lists can contain other lists (or other container types).

a = [[1, 2], [3, 4]]

print(a)
print(a[0])
[[1, 2], [3, 4]]
[1, 2]

Tuples - Advanced Python 02

A tuple is a collection of objects which is ordered and
immutable. Tuples are similar to lists, the main di erence ist
the immutability.

A tuple is a collection of objects which is ordered and

immutable. Tuples are similar to lists, the main di erence ist
the immutability. In Python tuples are written with round
brackets and comma separated values.

my_tuple = ("Max", 28, "New York")

Reasons to use a tuple over a list¶

• Generally used for objects that belong together.
• Use tuple for heterogeneous (di erent) datatypes and list
for homogeneous (similar) datatypes.
• Since tuple are immutable, iterating through tuple is
slightly faster than with list.

8
ff
ff
ff
 • Tuples with their immutable elements can be used as key
for a dictionary. This is not possible with lists.
• If you have data that doesn't change, implementing it as
tuple will guarantee that it remains write-protected.
Create a tuple¶
Tuples are created with round brackets and comma separated
values. Or use the built-in tuple function.

tuple_1 = ("Max", 28, "New York")

tuple_2 = "Linda", 25, "Miami" # Parentheses are optional

# Special case: a tuple with only one element needs to

have a comma at the end,
# otherwise it is not recognized as tuple
tuple_3 = (25,)
print(tuple_1)
print(tuple_2)
print(tuple_3)

# Or convert an iterable (list, dict, string) with the

built-in tuple function
tuple_4 = tuple([1,2,3])
print(tuple_4)
('Max', 28, 'New York')
('Linda', 25, 'Miami')
(25,)
(1, 2, 3)

Access elements¶
You access the tuple items by referring to the index number.
Note that the indices start at 0.

item = tuple_1[0]
print(item)
# You can also use negative indexing, e.g -1 refers to the
last item,
9
# -2 to the second last item, and so on
item = tuple_1[-1]
print(item)
Max
New York

Add or change items¶

Not possible and will raise a TypeError.

tuple_1[2] = "Boston"
----------------------------------------------------------
-----------------
TypeError Traceback (most
recent call last)
<ipython-input-5-c391d8981369> in <module>
----> 1 tuple_1[2] = "Boston"

TypeError: 'tuple' object does not support item assignment

Delete a tuple¶
del tuple_2

Iterating¶
# Iterating over a tuple by using a for in loop
for i in tuple_1:
print(i)
Max
28
New York

Check if an item exists¶

if "New York" in tuple_1:
print("yes")
else:
print("no")
yes

10
Useful methods¶
my_tuple = ('a','p','p','l','e',)

# len() : get the number of elements in a tuple

print(len(my_tuple))

# count(x) : Return the number of items that is equal to x

print(my_tuple.count('p'))

# index(x) : Return index of first item that is equal to x

print(my_tuple.index('l'))

# repetition
my_tuple = ('a', 'b') * 5
print(my_tuple)

# concatenation
my_tuple = (1,2,3) + (4,5,6)
print(my_tuple)

# convert list to a tuple and vice versa

my_list = ['a', 'b', 'c', 'd']
list_to_tuple = tuple(my_list)
print(list_to_tuple)

tuple_to_list = list(list_to_tuple)
print(tuple_to_list)

# convert string to tuple

string_to_tuple = tuple('Hello')
print(string_to_tuple)
5
2
3
('a', 'b', 'a', 'b', 'a', 'b', 'a', 'b', 'a', 'b')
11
(1, 2, 3, 4, 5, 6)
('a', 'b', 'c', 'd')
['a', 'b', 'c', 'd']
('H', 'e', 'l', 'l', 'o')

Slicing¶
Access sub parts of the tuple with the use of colon (:), just as
with strings.

# a[start:stop:step], default step is 1

a = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
b = a[1:3] # Note that the last index is not included
print(b)
b = a[2:] # until the end
print(b)
b = a[:3] # from beginning
print(b)
b = a[::2] # start to end with every second item
print(b)
b = a[::-1] # reverse tuple
print(b)
(2, 3)
(3, 4, 5, 6, 7, 8, 9, 10)
(1, 2, 3)
(1, 3, 5, 7, 9)
(10, 9, 8, 7, 6, 5, 4, 3, 2, 1)

Unpack tuple¶
# number of variables have to match number of tuple
elements
tuple_1 = ("Max", 28, "New York")
name, age, city = tuple_1
print(name)
print(age)
print(city)

12
# tip: unpack multiple elements to a list with *
my_tuple = (0, 1, 2, 3, 4, 5)
item_first, *items_between, item_last = my_tuple
print(item_first)
print(items_between)
print(item_last)
Max
28
New York
0
[1, 2, 3, 4]
5

Nested tuples¶
Tuples can contain other tuples (or other container types).

a = ((0, 1), ('age', 'height'))

print(a)
print(a[0])
((0, 1), ('age', 'height'))
(0, 1)

Compare tuple and list¶

The immutability of tuples enables Python to make internal
optimizations. Thus, tuples can be more e cient when working
with large data.

# compare the size

import sys
my_list = [0, 1, 2, "hello", True]
my_tuple = (0, 1, 2, "hello", True)
print(sys.getsizeof(my_list), "bytes")
print(sys.getsizeof(my_tuple), "bytes")

# compare the execution time of a list vs. tuple creation

statement
import timeit
13
ffi
print(timeit.timeit(stmt="[0, 1, 2, 3, 4, 5]",
number=1000000))
print(timeit.timeit(stmt="(0, 1, 2, 3, 4, 5)",
number=1000000))
104 bytes
88 bytes
0.12474981700000853
0.014836141000017733

Dictionary - Advanced
Python 03
A dictionary is a collection which is unordered, changeable and
indexed. A dictionary consists of a collection of key-value
pairs.

A dictionary is a collection which is unordered, changeable and

indexed. A dictionary consists of a collection of key-value
pairs. Each key-value pair maps the key to its associated
value. A dictionary is written in braces. Each key is separated
from its value by a colon (:), and the items are separated by
commas.

my_dict = {"name":"Max", "age":28, "city":"New York"}

Create a dictionary¶
Create a dictionary with braces, or with the built-in dict
function.

my_dict = {"name":"Max", "age":28, "city":"New York"}

print(my_dict)

# or use the dict constructor, note: no quotes necessary

for keys
my_dict_2 = dict(name="Lisa", age=27, city="Boston")

14
print(my_dict_2)
{'name': 'Max', 'age': 28, 'city': 'New York'}
{'name': 'Lisa', 'age': 27, 'city': 'Boston'}

Access items¶
name_in_dict = my_dict["name"]
print(name_in_dict)

# KeyError if no key is found

# print(my_dict["lastname"])
Max

Add and change items¶

Simply add or access a key and asign the value.

# add a new key

my_dict["email"] = "[email protected]"
print(my_dict)

# or overwrite the now existing key

my_dict["email"] = "[email protected]"
print(my_dict)
{'name': 'Max', 'age': 28, 'city': 'New York', 'email':
'[email protected]'}
{'name': 'Max', 'age': 28, 'city': 'New York', 'email':
'[email protected]'}

Delete items¶
# delete a key-value pair
del my_dict["email"]

# this returns the value and removes the key-value pair

print("popped value:", my_dict.pop("age"))

# return and removes the last inserted key-value pair

# (in versions before Python 3.7 it removes an arbitrary
pair)
15
print("popped item:", my_dict.popitem())

print(my_dict)

# clear() : remove all pairs

# my_dict.clear()
popped value: 28
popped item: ('city', 'New York')
{'name': 'Max'}

Check for keys¶

my_dict = {"name":"Max", "age":28, "city":"New York"}
# use if .. in ..
if "name" in my_dict:
print(my_dict["name"])

# use try except

try:
print(my_dict["firstname"])
except KeyError:
print("No key found")
Max
No key found

Looping through dictionary¶

# loop over keys
for key in my_dict:
print(key, my_dict[key])

# loop over keys

for key in my_dict.keys():
print(key)

# loop over values

for value in my_dict.values():
print(value)
16
# loop over keys and values
for key, value in my_dict.items():
print(key, value)
name Max
age 28
city New York
name
age
city
Max
28
New York
name Max
age 28
city New York

Copy a dictionary¶
Be careful when copying references.

dict_org = {"name":"Max", "age":28, "city":"New York"}

# this just copies the reference to the dict, so be

careful
dict_copy = dict_org

# now modifying the copy also affects the original

dict_copy["name"] = "Lisa"
print(dict_copy)
print(dict_org)

# use copy(), or dict(x) to actually copy the dict

dict_org = {"name":"Max", "age":28, "city":"New York"}

dict_copy = dict_org.copy()
# dict_copy = dict(dict_org)

17
# now modifying the copy does not affect the original
dict_copy["name"] = "Lisa"
print(dict_copy)
print(dict_org)
{'name': 'Lisa', 'age': 28, 'city': 'New York'}
{'name': 'Lisa', 'age': 28, 'city': 'New York'}
{'name': 'Lisa', 'age': 28, 'city': 'New York'}
{'name': 'Max', 'age': 28, 'city': 'New York'}

Merge two dictionaries¶

# Use the update() method to merge 2 dicts
# existing keys are overwritten, new keys are added
my_dict = {"name":"Max", "age":28, "email":"[email protected]"}
my_dict_2 = dict(name="Lisa", age=27, city="Boston")

my_dict.update(my_dict_2)
print(my_dict)
{'name': 'Lisa', 'age': 27, 'email': '[email protected]',
'city': 'Boston'}

Possible key types¶

Any immutable type, like strings or numbers can be used as a
key. Also, a tuple can be used if it contains only immutable
elements.

# use numbers as key, but be careful

my_dict = {3: 9, 6: 36, 9:81}
# do not mistake the keys as indices of a list, e.g
my_dict[0] is not possible here
print(my_dict[3], my_dict[6], my_dict[9])

# use a tuple with immutable elements (e.g. number,

string)
my_tuple = (8, 7)
my_dict = {my_tuple: 15}

18
print(my_dict[my_tuple])
# print(my_dict[8, 7])

# a list is not possible because it is not immutable

# this will raise an Error:
# my_list = [8, 7]
# my_dict = {my_list: 15}
9 36 81
15

Nested dictionaries¶
The values can also be container types (e.g. lists, tuples,
dictionaries).

my_dict_1 = {"name": "Max", "age": 28}

my_dict_2 = {"name": "Alex", "age": 25}
nested_dict = {"dictA": my_dict_1,
"dictB": my_dict_2}
print(nested_dict)
{'dictA': {'name': 'Max', 'age': 28}, 'dictB': {'name':
'Alex', 'age': 25}}

Sets - Advanced Python 04

A Set is an unordered collection data type that is unindexed,
mutable, and has no duplicate elements. Sets are created with
braces.

my_set = {"apple", "banana", "cherry"}

Create a set¶
Use curly braces or the built-in set function.

my_set = {"apple", "banana", "cherry"}

print(my_set)

19
# or use the set function and create from an iterable,
e.g. list, tuple, string
my_set_2 = set(["one", "two", "three"])
my_set_2 = set(("one", "two", "three"))
print(my_set_2)

my_set_3 = set("aaabbbcccdddeeeeeffff")
print(my_set_3)

# careful: an empty set cannot be created with {}, as this

is interpreted as dict
# use set() instead
a = {}
print(type(a))
a = set()
print(type(a))
{'banana', 'apple', 'cherry'}
{'three', 'one', 'two'}
{'b', 'c', 'd', 'e', 'f', 'a'}
<class 'dict'>
<class 'set'>

Add elements¶
my_set = set()

# use the add() method to add elements

my_set.add(42)
my_set.add(True)
my_set.add("Hello")

# note: the order does not matter, and might differ when
printed
print(my_set)

# nothing happens when the element is already present:

my_set.add(42)
20
print(my_set)
{True, 42, 'Hello'}
{True, 42, 'Hello'}

Remove elements¶
# remove(x): removes x, raises a KeyError if element is
not present
my_set = {"apple", "banana", "cherry"}
my_set.remove("apple")
print(my_set)

# KeyError:
# my_set.remove("orange")

# discard(x): removes x, does nothing if element is not

present
my_set.discard("cherry")
my_set.discard("blueberry")
print(my_set)

# clear() : remove all elements

my_set.clear()
print(my_set)

# pop() : return and remove a random element

a = {True, 2, False, "hi", "hello"}
print(a.pop())
print(a)
{'banana', 'cherry'}
{'banana'}
set()
False
{True, 2, 'hi', 'hello'}

Check if element is in Set¶

my_set = {"apple", "banana", "cherry"}

21
if "apple" in my_set:
print("yes")
yes

Iterating¶
# Iterating over a set by using a for in loop
# Note: order is not important
my_set = {"apple", "banana", "cherry"}
for i in my_set:
print(i)
banana
apple
cherry

Union and Intersection¶

odds = {1, 3, 5, 7, 9}
evens = {0, 2, 4, 6, 8}
primes = {2, 3, 5, 7}

# union() : combine elements from both sets, no

duplication
# note that this does not change the two sets
u = odds.union(evens)
print(u)

# intersection(): take elements that are in both sets

i = odds.intersection(evens)
print(i)

i = odds.intersection(primes)
print(i)

i = evens.intersection(primes)
print(i)
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
set()

22
 {3, 5, 7}
{2}

Di erence of sets¶
setA = {1, 2, 3, 4, 5, 6, 7, 8, 9}
setB = {1, 2, 3, 10, 11, 12}

# difference() : returns a set with all the elements from

the setA that are not in setB.
diff_set = setA.difference(setB)
print(diff_set)

# A.difference(B) is not the same as B.difference(A)

diff_set = setB.difference(setA)
print(diff_set)

# symmetric_difference() : returns a set with all the

elements that are in setA and setB but not in both
diff_set = setA.symmetric_difference(setB)
print(diff_set)

# A.symmetric_difference(B) = B.symmetric_difference(A)
diff_set = setB.symmetric_difference(setA)
print(diff_set)
{4, 5, 6, 7, 8, 9}
{10, 11, 12}
{4, 5, 6, 7, 8, 9, 10, 11, 12}
{4, 5, 6, 7, 8, 9, 10, 11, 12}

Updating sets¶
setA = {1, 2, 3, 4, 5, 6, 7, 8, 9}
setB = {1, 2, 3, 10, 11, 12}

# update() : Update the set by adding elements from

another set.
setA.update(setB)
print(setA)
23
ff
# intersection_update() : Update the set by keeping only
the elements found in both
setA = {1, 2, 3, 4, 5, 6, 7, 8, 9}
setA.intersection_update(setB)
print(setA)

# difference_update() : Update the set by removing

elements found in another set.
setA = {1, 2, 3, 4, 5, 6, 7, 8, 9}
setA.difference_update(setB)
print(setA)

# symmetric_difference_update() : Update the set by only

keeping the elements found in either set, but not in both
setA = {1, 2, 3, 4, 5, 6, 7, 8, 9}
setA.symmetric_difference_update(setB)
print(setA)

# Note: all update methods also work with other iterables

as argument, e.g lists, tuples
# setA.update([1, 2, 3, 4, 5, 6])
{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
{1, 2, 3}
{4, 5, 6, 7, 8, 9}
{4, 5, 6, 7, 8, 9, 10, 11, 12}

Copying¶
set_org = {1, 2, 3, 4, 5}

# this just copies the reference to the set, so be careful

set_copy = set_org

# now modifying the copy also affects the original

set_copy.update([3, 4, 5, 6, 7])
print(set_copy)
print(set_org)
24
# use copy() to actually copy the set
set_org = {1, 2, 3, 4, 5}
set_copy = set_org.copy()

# now modifying the copy does not affect the original

set_copy.update([3, 4, 5, 6, 7])
print(set_copy)
print(set_org)
{1, 2, 3, 4, 5, 6, 7}
{1, 2, 3, 4, 5, 6, 7}
{1, 2, 3, 4, 5, 6, 7}
{1, 2, 3, 4, 5}

Subset, Superset, and Disjoint¶

setA = {1, 2, 3, 4, 5, 6}
setB = {1, 2, 3}
# issubset(setX): Returns True if setX contains the set
print(setA.issubset(setB))
print(setB.issubset(setA)) # True

# issuperset(setX): Returns True if the set contains setX

print(setA.issuperset(setB)) # True
print(setB.issuperset(setA))

# isdisjoint(setX) : Return True if both sets have a null

intersection, i.e. no same elements
setC = {7, 8, 9}
print(setA.isdisjoint(setB))
print(setA.isdisjoint(setC))
False
True
True
False
False
True

25
Frozenset¶
Frozen set is just an immutable version of normal set. While
elements of a set can be modi ed at any time, elements of
frozen set remains the same after creation. Creation with:
my_frozenset = frozenset(iterable)

a = frozenset([0, 1, 2, 3, 4])

# The following is not allowed:

# a.add(5)
# a.remove(1)
# a.discard(1)
# a.clear()

# Also no update methods are allowed:

# a.update([1,2,3])

# Other set operations work

odds = frozenset({1, 3, 5, 7, 9})
evens = frozenset({0, 2, 4, 6, 8})
print(odds.union(evens))
print(odds.intersection(evens))
print(odds.difference(evens))
frozenset({0, 1, 2, 3, 4, 5, 6, 7, 8, 9})
frozenset()
frozenset({1, 3, 5, 7, 9})

Strings - Advanced Python 05

A string is a sequence of characters. String literals in Python
are enclosed by either double or single quotes.

my_string = 'Hello'

26
fi
Python strings are immutable which means they cannot be
changed after they are created.

Creation¶
# use singe or double quotes
my_string = 'Hello'
my_string = "Hello"
my_string = "I' m a 'Geek'"

# escaping backslash
my_string = 'I\' m a "Geek"'
my_string = 'I\' m a \'Geek\''
print(my_string)

# triple quotes for multiline strings

my_string = """Hello
World"""
print(my_string)

# backslash if you want to continue in the next line

my_string = "Hello \
World"
print(my_string)
I' m a 'Geek'
Hello
World
Hello World
Access characters and substrings¶
my_string = "Hello World"

# get character by referring to index

b = my_string[0]
print(b)

# Substrings with slicing

b = my_string[1:3] # Note that the last index is not
included
print(b)
b = my_string[:5] # from beginning
print(b)
b = my_string[6:] # until the end
print(b)
27
b = my_string[::2] # start to end with every second item
print(b)
b = my_string[::-1] # reverse the string with a negative
step:
print(b)
H
el
Hello
World
HloWrd
dlroW olleH

Concatenate two or more strings¶

# concat strings with +
greeting = "Hello"
name = "Tom"
sentence = greeting + ' ' + name
print(sentence)
Hello Tom
Iterating¶
# Iterating over a string by using a for in loop
my_string = 'Hello'
for i in my_string:
print(i)
H
e
l
l
o
Check if a character or substring exists¶
if "e" in "Hello":
print("yes")
if "llo" in "Hello":
print("yes")
yes
yes
Useful methods¶
my_string = " Hello World "
28
# remove white space
my_string = my_string.strip()
print(my_string)

# number of characters
print(len(my_string))

# Upper and lower cases

print(my_string.upper())
print(my_string.lower())

# startswith and endswith

print("hello".startswith("he"))
print("hello".endswith("llo"))

# find first index of a given substring, -1 otherwise

print("Hello".find("o"))

# count number of characters/substrings

print("Hello".count("e"))

# replace a substring with another string (only if the

substring is found)
# Note: The original string stays the same
message = "Hello World"
new_message = message.replace("World", "Universe")
print(new_message)

# split the string into a list

my_string = "how are you doing"
a = my_string.split() # default argument is " "
print(a)
my_string = "one,two,three"
a = my_string.split(",")
print(a)

# join elements of a list into a string

my_list = ['How', 'are', 'you', 'doing']
a = ' '.join(my_list) # the given string is the separator,
e.g. ' ' between each argument
print(a)
Hello World
11
HELLO WORLD
29
hello world
['how', 'are', 'you', 'doing']
['one', 'two', 'three']
True
True
4
1
Hello Universe
How are you doing
Format¶
New style is with .format() and old style is with % operator.

# use braces as placeholders

a = "Hello {0} and {1}".format("Bob", "Tom")
print(a)

# the positions are optional for the default order

a = "Hello {} and {}".format("Bob", "Tom")
print(a)

a = "The integer value is {}".format(2)

print(a)

# some special format rules for numbers

a = "The float value is {0:.3f}".format(2.1234)
print(a)
a = "The float value is {0:e}".format(2.1234)
print(a)
a = "The binary value is {0:b}".format(2)
print(a)

# old style formatting by using % operator

print("Hello %s and %s" % ("Bob", "Tom")) # must be a
tuple for multiple arguments
val = 3.14159265359
print("The decimal value is %d" % val)
print("The float value is %f" % val)
print("The float value is %.2f" % val)
Hello Bob and Tom
Hello Bob and Tom
The integer value is 2
The float value is 2.123
The float value is 2.123400e+00

30
The binary value is 10
Hello Bob and Tom
The decimal value is 10
The float value is 10.123450
The float value is 10.12
f-Strings¶
New since Python 3.6. Use the variables directly inside the braces.

name = "Eric"
age = 25
a = f"Hello, {name}. You are {age}."
print(a)
pi = 3.14159
a = f"Pi is {pi:.3f}"
print(a)
# f-Strings are evaluated at runtime, which allows
expressions
a = f"The value is {2*60}"
print(a)
Hello, Eric. You are 25.
Pi is 3.142
The value is 120
More on immutability and concatenation¶
# since a string is immutable, adding strings with +, or
+= always
# creates a new string, and therefore is expensive for
multiple operations
# --> join method is much faster
from timeit import default_timer as timer
my_list = ["a"] * 1000000

# bad
start = timer()
a = ""
for i in my_list:
a += i
end = timer()
print("concatenate string with + : %.5f" % (end - start))

# good
start = timer()
a = "".join(my_list)
31
 end = timer()
print("concatenate string with join(): %.5f" % (end -
start))
concat string with + : 0.34527
concat string with join(): 0.01191

Collections - Advanced
Python 06
The collections module in Python implements specialized
container datatypes providing alternatives to Python’s general
purpose built-in containers, dict, list, set, and tuple.

The following tools exist:

- namedtuple : factory function for creating tuple subclasses with
named elds

- OrderedDict : dict subclass that remembers the order entries

were added

- - Counter : dict subclass for counting hashable objects

- - defaultdict : dict subclass that calls a factory function to supply
missing values

- - deque : list-like container with fast appends and pops on either

end

In Python 3 some more modules exist (ChainMap, UserDict,

UserList, UserString). See https://docs.python.org/3/library/
collections.html for further references.

Counter¶
A counter is a container that stores elements as dictionary keys,
and their counts are stored as dictionary values.

from collections import Counter

32
fi
a = "aaaaabbbbcccdde"
my_counter = Counter(a)
print(my_counter)

print(my_counter.items())
print(my_counter.keys())
print(my_counter.values())

my_list = [0, 1, 0, 1, 2, 1, 1, 3, 2, 3, 2, 4]
my_counter = Counter(my_list)
print(my_counter)

# most common items

print(my_counter.most_common(1))

# Return an iterator over elements repeating each as many

times as its count.
# Elements are returned in arbitrary order.
print(list(my_counter.elements()))
Counter({'a': 5, 'b': 4, 'c': 3, 'd': 2, 'e': 1})
dict_items([('a', 5), ('b', 4), ('c', 3), ('d', 2), ('e',
1)])
dict_keys(['a', 'b', 'c', 'd', 'e'])
dict_values([5, 4, 3, 2, 1])
Counter({1: 4, 2: 3, 0: 2, 3: 2, 4: 1})
[(1, 4)]
[0, 0, 1, 1, 1, 1, 2, 2, 2, 3, 3, 4]
namedtuple¶
namedtuples are easy to create, lightweight object types. They
assign meaning to each position in a tuple and allow for more
readable, self-documenting code. They can be used wherever
regular tuples are used, and they add the ability to access elds by
name instead of position index.

from collections import namedtuple

# create a namedtuple with its class name as string and
its fields as string
# fields have to be separated by comma or space in the
given string
Point = namedtuple('Point','x, y')
pt = Point(1, -4)
print(pt)

33
fi
 print(pt._fields)
print(type(pt))
print(pt.x, pt.y)

Person = namedtuple('Person','name, age')

friend = Person(name='Tom', age=25)
print(friend.name, friend.age)
Point(x=1, y=-4)
('x', 'y')
<class '__main__.Point'>
1 -4
Tom 25
OrderedDict¶
OrderedDicts are just like regular dictionaries but they remember
the order that items were inserted. When iterating over an ordered
dictionary, the items are returned in the order their keys were rst
added. If a new entry overwrites an existing entry, the original
insertion position is left unchanged. They have become less
important now that the built-in dict class gained the ability to
remember insertion order (guaranteed since Python 3.7). But some
di erences still remain, e.g. the OrderedDict is designed to be
good at reordering operations.

from collections import OrderedDict

ordinary_dict = {}
ordinary_dict['a'] = 1
ordinary_dict['b'] = 2
ordinary_dict['c'] = 3
ordinary_dict['d'] = 4
ordinary_dict['e'] = 5
# this may be in orbitrary order prior to Python 3.7
print(ordinary_dict)

ordered_dict = OrderedDict()
ordered_dict['a'] = 1
ordered_dict['b'] = 2
ordered_dict['c'] = 3
ordered_dict['d'] = 4
ordered_dict['e'] = 5
print(ordered_dict)
# same functionality as with ordinary dict, but always
ordered
34
ff
fi
for k, v in ordinary_dict.items():
print(k, v)
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}
OrderedDict([('a', 1), ('b', 2), ('c', 3), ('d', 4), ('e',
5)])
a 1
b 2
c 3
d 4
e 5
defaultdict¶
The defaultdict is a container that's similar to the usual dict
container, but the only di erence is that a defaultdict will have a
default value if that key has not been set yet. If you didn't use a
defaultdict you'd have to check to see if that key exists, and if it
doesn't, set it to what you want.

from collections import defaultdict

# initialize with a default integer value, i.e 0

d = defaultdict(int)
d['yellow'] = 1
d['blue'] = 2
print(d.items())
print(d['green'])

# initialize with a default list value, i.e an empty list

d = defaultdict(list)
s = [('yellow', 1), ('blue', 2), ('yellow', 3), ('blue',
4), ('red', 5)]
for k, v in s:
d[k].append(v)

print(d.items())
print(d['green'])
dict_items([('yellow', 1), ('blue', 2)])
0
dict_items([('yellow', [1, 3]), ('blue', [2, 4]), ('red',
[5])])
[]
deque¶
35
ff
 A deque is a double-ended queue. It can be used to add or remove
elements from both ends. Deques support thread safe, memory
e cient appends and pops from either side of the deque with
approximately the same O(1) performance in either direction. The
more commonly used stacks and queues are degenerate forms of
deques, where the inputs and outputs are restricted to a single
end.

from collections import deque

d = deque()

# append() : add elements to the right end

d.append('a')
d.append('b')
print(d)

# appendleft() : add elements to the left end

d.appendleft('c')
print(d)

# pop() : return and remove elements from the right

print(d.pop())
print(d)

# popleft() : return and remove elements from the left

print(d.popleft())
print(d)

# clear() : remove all elements

d.clear()
print(d)

d = deque(['a', 'b', 'c', 'd'])

# extend at right or left side

d.extend(['e', 'f', 'g'])
d.extendleft(['h', 'i', 'j']) # note that 'j' is now at
the left most position
print(d)

# count(x) : returns the number of found elements

print(d.count('h'))

# rotate 1 positions to the right

36
ffi
d.rotate(1)
print(d)

# rotate 2 positions to the left

d.rotate(-2)
print(d)
deque(['a', 'b'])
deque(['c', 'a', 'b'])
b
deque(['c', 'a'])
c
deque(['a'])
deque([])
deque(['j', 'i', 'h', 'a', 'b', 'c', 'd', 'e', 'f', 'g'])
1
deque(['g', 'j', 'i', 'h', 'a', 'b', 'c', 'd', 'e', 'f'])
deque(['i', 'h', 'a', 'b', 'c', 'd', 'e', 'f', 'g', ‚j'])

Itertools - Advanced Python

07
The Python itertools module is a collection of tools for
handling iterators. Simply put, iterators are data types that can
be used in a for loop.

product()¶
This tool computes the cartesian product of input iterables.
It is equivalent to nested for-loops. For example, product(A, B)
returns the same as ((x,y) for x in A for y in B).

from itertools import product

prod = product([1, 2], [3, 4])

print(list(prod)) # note that we convert the iterator to a
list for printing

# to allow the product of an iterable with itself, specify

the number of repetitions
prod = product([1, 2], [3], repeat=2)
37
print(list(prod)) # note that we convert the iterator to a
list for printing
[(1, 3), (1, 4), (2, 3), (2, 4)]
[(1, 3, 1, 3), (1, 3, 2, 3), (2, 3, 1, 3), (2, 3, 2, 3)]
permutations()¶
This tool returns successive length permutations of elements in an
iterable, with all possible orderings, and no repeated elements.

from itertools import permutations

perm = permutations([1, 2, 3])

print(list(perm))

# optional: the length of the permutation tuples

perm = permutations([1, 2, 3], 2)
print(list(perm))
[(1, 2, 3), (1, 3, 2), (2, 1, 3), (2, 3, 1), (3, 1, 2),
(3, 2, 1)]
[(1, 2), (1, 3), (2, 1), (2, 3), (3, 1), (3, 2)]
combinations() and
combinations_with_replacement()¶
r-length tuples, in sorted order. So, if the input iterable is sorted,
the combination tuples will be produced in sorted order.
combinations() does not allow repeated elements, but
combinations_with_replacement() does.

from itertools import combinations,

combinations_with_replacement

# the second argument is mandatory and specifies the

length of the output tuples.
comb = combinations([1, 2, 3, 4], 2)
print(list(comb))

comb = combinations_with_replacement([1, 2, 3, 4], 2)

print(list(comb))
[(1, 2), (1, 3), (1, 4), (2, 3), (2, 4), (3, 4)]
[(1, 1), (1, 2), (1, 3), (1, 4), (2, 2), (2, 3), (2, 4),
(3, 3), (3, 4), (4, 4)]

38
accumulate()¶
Make an iterator that returns accumulated sums, or accumulated
results of other binary functions.

from itertools import accumulate

# return accumulated sums

acc = accumulate([1,2,3,4])
print(list(acc))

# other possible functions are possible

import operator
acc = accumulate([1,2,3,4], func=operator.mul)
print(list(acc))

acc = accumulate([1,5,2,6,3,4], func=max)

print(list(acc))
[1, 3, 6, 10]
[1, 2, 6, 24]
[1, 5, 5, 6, 6, 6]
groupby()¶
Make an iterator that returns consecutive keys and groups from the
iterable. The key is a function computing a key value for each
element. If not speci ed or is None, key defaults to an identity
function and returns the element unchanged. Generally, the iterable
needs to already be sorted on the same key function.

from itertools import groupby

# use a function as key

def smaller_than_3(x):
return x < 3

group_obj = groupby([1, 2, 3, 4], key=smaller_than_3)

for key, group in group_obj:
print(key, list(group))

# or use a lamda expression, e.g. words with an 'i':

group_obj = groupby(["hi", "nice", "hello", "cool"],
key=lambda x: "i" in x)

39
fi
 for key, group in group_obj:
print(key, list(group))

persons = [{'name': 'Tim', 'age': 25}, {'name': 'Dan',

'age': 25},
{'name': 'Lisa', 'age': 27}, {'name': 'Claire',
'age': 28}]

for key, group in groupby(persons, key=lambda x:

x['age']):
print(key, list(group))
True [1, 2]
False [3, 4]
True ['hi', 'nice']
False ['hello', 'cool']
25 [{'name': 'Tim', 'age': 25}, {'name': 'Dan', 'age':
25}]
27 [{'name': 'Lisa', 'age': 27}]
28 [{'name': 'Claire', 'age': 28}]
In nite iterators: count(), cycle(),
repeat()¶
from itertools import count, cycle, repeat
# count(x): count from x: x, x+1, x+2, x+3...
for i in count(10):
print(i)
if i >= 13:
break

# cycle(iterable) : cycle infinitely through an iterable

print("")
sum = 0
for i in cycle([1, 2, 3]):
print(i)
sum += i
if sum >= 12:
break

# repeat(x): repeat x infinitely or n times

print("")
for i in repeat("A", 3):
print(i)
10
40
fi
 11
12
13

1
2
3
1
2
3

A
A
A

Lambda Functions -
Advanced Python 08
A lambda function is a small (one line) anonymous function
that is de ned without a name.

A lambda function is a small (one line) anonymous function that is

de ned without a name. A lambda function can take any number of
arguments, but can only have one expression. While normal
functions are de ned using the def keyword, in Python anonymous
functions are de ned using the lambda keyword.

lambda arguments: expression

Lambda functions are used when a simple function is used only

once or for a short period in your code. It's most common use is as
an argument to higher-order functions (functions that takes in other
functions as arguments). They are also used along with built-in
functions like map(), filter(), reduce().

# a lambda function that adds 10 to the input argument

f = lambda x: x+10
val1 = f(5)
val2 = f(100)
print(val1, val2)
41
fi
fi
fi
fi
# a lambda function that multiplies two input arguments
and returns the result
f = lambda x,y: x*y
val3 = f(2,10)
val4 = f(7,5)
print(val3, val4)
15 110
20 35
Usage example: Lamdba inside another
function¶
Return a customized lambda function from another function and
create di erent function variations depending on your needs.

def myfunc(n):
return lambda x: x * n

doubler = myfunc(2)
print(doubler(6))

tripler = myfunc(3)
print(tripler(6))
12
18
Custom sorting using a lambda function
as key parameter¶
The key function transforms each element before sorting.

points2D = [(1, 9), (4, 1), (5, -3), (10, 2)]

sorted_by_y = sorted(points2D, key= lambda x: x[1])
print(sorted_by_y)

mylist = [- 1, -4, -2, -3, 1, 2, 3, 4]

sorted_by_abs = sorted(mylist, key= lambda x: abs(x))
print(sorted_by_abs)
[(5, -3), (4, 1), (10, 2), (1, 9)]
[-1, 1, -2, 2, -3, 3, -4, 4]
Use lambda for map function¶
42
ff
map(func, seq), transforms each element with the function.

a = [1, 2, 3, 4, 5, 6]
b = list(map(lambda x: x * 2 , a))

# However, try to prefer list comprehension

# Use map if you have an already defined function
c = [x*2 for x in a]
print(b)
print(c)
[2, 4, 6, 8, 10, 12]
[2, 4, 6, 8, 10, 12]
Use lambda for lter function¶
filter(func, seq), returns all elements for which func evaluates
to True.

a = [1, 2, 3, 4, 5, 6, 7, 8]
b = list(filter(lambda x: (x%2 == 0) , a))

# However, the same can be achieved with list

comprehension
c = [x for x in a if x%2 == 0]
print(b)
print(c)
[2, 4, 6, 8]
[2, 4, 6, 8]
reduce¶
reduce(func, seq), repeatedly applies the func to the elements
and returns a single value.
func takes 2 arguments.

from functools import reduce

a = [1, 2, 3, 4]
product_a = reduce(lambda x, y: x*y, a)
print(product_a)
sum_a = reduce(lambda x, y: x+y, a)
print(sum_a)
24
10

43
fi
Exceptions And Errors -
Advanced Python 09
A Python program terminates as soon as it encounters an
error. In Python, an error can be a syntax error or an exception.

A Python program terminates as soon as it encounters an error. In

Python, an error can be a syntax error or an exception. In this
article we will have a look at: - Syntax Error vs. Exception - How to
raise Exceptions - How to handle Exceptions - Most common built-
in Exceptions - How to de ne your own Exception

Syntax Errors¶
A Syntax Error occurs when the parser detects a syntactically
incorrect statement. A syntax error can be for example a typo,
missing brackets, no new line (see code below), or wrong
identation (this will actually raise its own IndentationError, but its
subclassed from a SyntaxError).

a = 5 print(a)
File "<ipython-input-5-fed4b61d14cd>", line 1
a = 5 print(a)
^
SyntaxError: invalid syntax
Exceptions¶
Even if a statement is syntactically correct, it may cause an error
when it is executed. This is called an Exception Error. There are
several di erent error classes, for example trying to add a number
and a string will raise a TypeError.

a = 5 + '10'
----------------------------------------------------------
-----------------
TypeError Traceback (most
recent call last)

44
ff
fi
<ipython-input-6-893398416ed7> in <module>
----> 1 a = 5 + '10'

TypeError: unsupported operand type(s) for +: 'int' and

'str'
Raising an Exception¶
If you want to force an exception to occur when a certain condition
is met, you can use the raise keyword.

x = -5
if x < 0:
raise Exception('x should not be negative.')
----------------------------------------------------------
-----------------
Exception Traceback (most
recent call last)
<ipython-input-4-2a9e7e673803> in <module>
1 x = -5
2 if x < 0:
----> 3 raise Exception('x should not be negative.')

Exception: x should not be negative.

You can also use the assert statement, which will throw an
AssertionError if your assertion is not True. This way, you can
actively test some conditions that have to be ful lled instead of
waiting for your program to unexpectedly crash midway. Assertion
is also used in unit testing.

x = -5
assert (x >= 0), 'x is not positive.'
# --> Your code will be fine if x >= 0
----------------------------------------------------------
-----------------
AssertionError Traceback (most
recent call last)
<ipython-input-7-f9b059c51e45> in <module>
1 x = -5
----> 2 assert (x >= 0), 'x is not positive.'
3 # --> Your code will be fine if x >= 0
AssertionError: x is not positive.
Handling Exceptions¶
45
fi
You can use a try and except block to catch and handle
exceptions. If you can catch an exceptions your program won't
terminate, and can continue.

# This will catch all possible exceptions

try:
a = 5 / 0
except:
print('some error occured.')

# You can also catch the type of exception

try:
a = 5 / 0
except Exception as e:
print(e)

# It is good practice to specify the type of Exception you

want to catch.
# Therefore, you have to know the possible errors
try:
a = 5 / 0
except ZeroDivisionError:
print('Only a ZeroDivisionError is handled here')

# You can run multiple statements in a try block, and

catch different possible exceptions
try:
a = 5 / 1 # Note: No ZeroDivisionError here
b = a + '10'
except ZeroDivisionError as e:
print('A ZeroDivisionError occured:', e)
except TypeError as e:
print('A TypeError occured:', e)
some error occured.
division by zero
Only a ZeroDivisionError is handled here
TypeError occured: unsupported operand type(s) for +:
'float' and 'str'
else clause¶

You can use an else statement that is run if no exception occured.

try:
a = 5 / 1
46
 except ZeroDivisionError as e:
print('A ZeroDivisionError occured:', e)
else:
print('Everything is ok')
Everything is ok
finally clause¶

You can use a nally statement that always runs, no matter if there
was an exception or not. This is for example used to make some
cleanup operations.

try:
a = 5 / 1 # Note: No ZeroDivisionError here
b = a + '10'
except ZeroDivisionError as e:
print('A ZeroDivisionError occured:', e)
except TypeError as e:
print('A TypeError occured:', e)
else:
print('Everything is ok')
finally:
print('Cleaning up some stuff...')
A TypeError occured: unsupported operand type(s) for +:
'float' and 'str'
Cleaning up some stuff...
Common built-in Exceptions¶
You can nd all built-in Exceptions here: https://docs.python.org/3/
library/exceptions.html - ImportError: If a module cannot be
imported - NameError: If you try to use a variable that was not
de ned - FileNotFoundError: If you try to open a le that does not
exist or you specify the wrong path - ValueError: When an
operation or function receives an argument that has the right type
but an inappropriate value, e.g. try to remove a value from a list
that does not exist - TypeError: Raised when an operation or
function is applied to an object of inappropriate type. - IndexError:
If you try to access an invalid index of a sequence, e.g a list or a
tuple. - KeyError: If you try to access a non existing key of a
dictionary.

# ImportError
import nonexistingmodule
47
fi
fi
fi
fi
 # NameError
a = someundefinedvariable

# FileNotFoundError
with open('nonexistingfile.txt') as f:
read_data = f.read()

# ValueError
a = [0, 1, 2]
a.remove(3)

# TypeError
a = 5 + "10"

# IndexError
a = [0, 1, 2]
value = a[5]

# KeyError
my_dict = {"name": "Max", "city": "Boston"}
age = my_dict["age"]
De ne your own Exceptions¶
You can de ne your own exception class that should be derived
from the built-in Exception class. Most exceptions are de ned with
names that end in 'Error', similar to the naming of the standard
exceptions. Exception classes can be de ned like any other class,
but are usually kept simple, often only o ering a number of
attributes that allow information about the error to be extracted by
handlers.

# minimal example for own exception class

class ValueTooHighError(Exception):
pass

# or add some more information for handlers

class ValueTooLowError(Exception):
def __init__(self, message, value):
self.message = message
self.value = value

def test_value(a):
48
fi
fi
ff
fi
fi
 if a > 1000:
raise ValueTooHighError('Value is too high.')
if a < 5:
raise ValueTooLowError('Value is too low.', a) #
Note that the constructor takes 2 arguments here
return a

try:
test_value(1)
except ValueTooHighError as e:
print(e)
except ValueTooLowError as e:
print(e.message, 'The value is:', e.value)
Value is too low. The value is: 1

Logging - Advanced Python

10
The logging module in Python is a powerful built-in module so
you can quickly add logging to your application.

import logging

Log Level¶
There are 5 di erent log levels indicating the serverity of events. By
default, the system logs only events with level WARNING and
above.

import logging
logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')
WARNING:root:This is a warning message
ERROR:root:This is an error message
CRITICAL:root:This is a critical message
Con guration¶
49
fi
ff
 With basicConfig(**kwargs) you can customize the root logger.
The most common parameters are the level, the format, and the
lename. See https://docs.python.org/3/library/
logging.html#logging.basicCon g for all possible arguments. See
also https://docs.python.org/3/library/logging.html#logrecord-
attributes for possible formats and https://docs.python.org/3/
library/time.html#time.strftime how to set the time string. Note that
this function should only be called once, and typically rst thing
after importing the module. It has no e ect if the root logger
already has handlers con gured. For example calling
logging.info(...) before the basicCon g will already set a
handler.

import logging
logging.basicConfig(level=logging.DEBUG, format='%
(asctime)s - %(name)s - %(levelname)s - %(message)s',
datefmt='%m/%d/%Y %H:%M:%S')
# Now also debug messages will get logged with a different
format.
logging.debug('Debug message')

# This would log to a file instead of the console.

# logging.basicConfig(level=logging.DEBUG,
filename='app.log')
Logging in modules and logger
hierarchy¶
Best practice in your application with multiple modules is to create
an internal logger using the __name__ global variable. This will
create a logger with the name of your module and ensures no
name collisions. The logging module creates a hierarchy of loggers,
starting with the root logger, and adding the new logger to this
hierarchy. If you then import your module in another module, log
messages can be associated with the correct module through the
logger name. Note that changing the basicCon g of the root logger
will also a ect the log events of the other (lower) loggers in the
hierarchy.

# helper.py
# -------------------------------------

50
fi
ff
fi
fi
ff
fi
fi
fi
import logging
logger = logging.getLogger(__name__)
logger.info('HELLO')

# main.py
# -------------------------------------
import logging
logging.basicConfig(level=logging.INFO, format='%(name)s -
%(levelname)s - %(message)s')
import helper

# --> Output when running main.py

# helper - INFO - HELLO
Propagation¶
By default, all created loggers will pass the log events to the
handlers of higher loggers, in addition to any handlers attached to
the created logger. You can deactivate this by setting propagate =
False. Sometimes when you wonder why you don't see log
messages from another module, then this property may be the
reason.

# helper.py
# -------------------------------------
import logging
logger = logging.getLogger(__name__)
logger.propagate = False
logger.info('HELLO')

# main.py
# -------------------------------------
import logging
logging.basicConfig(level=logging.INFO, format='%(name)s -
%(levelname)s - %(message)s')
import helper

# --> No output when running main.py since the helper

module logger does not propagate its messages to the root
logger
LogHandlers¶

51
 Handler objects are responsible for dispatching the appropriate log
messages to the handler's speci c destination. For example you
can use di erent handlers to send log messaged to the standard
output stream, to les, via HTTP, or via Email. Typically you
con gure each handler with a level (setLevel()), a formatter
(setFormatter()), and optionally a lter (addFilter()). See https://
docs.python.org/3/howto/logging.html#useful-handlers for possible
built-in handlers. Of course you can also implement your own
handlers by deriving from these classes.

import logging

logger = logging.getLogger(__name__)

# Create handlers
stream_handler = logging.StreamHandler()
file_handler = logging.FileHandler('file.log')

# Configure level and formatter and add it to handlers

stream_handler.setLevel(logging.WARNING) # warning and
above is logged to the stream
file_handler.setLevel(logging.ERROR) # error and above is
logged to a file

stream_format = logging.Formatter('%(name)s - %
(levelname)s - %(message)s')
file_format = logging.Formatter('%(asctime)s - %(name)s -
%(levelname)s - %(message)s')
stream_handler.setFormatter(stream_format)
file_handler.setFormatter(file_format)

# Add handlers to the logger

logger.addHandler(stream_handler)
logger.addHandler(file_handler)

logger.warning('This is a warning') # logged to the stream

logger.error('This is an error') # logged to the stream
AND the file!
Example of a lter¶
class InfoFilter(logging.Filter):

52
fi
ff
fi
fi
fi
fi
 # overwrite this method. Only log records for which
this
# function evaluates to True will pass the filter.
def filter(self, record):
return record.levelno == logging.INFO

# Now only INFO level messages will be logged

stream_handler.addFilter(InfoFilter())
logger.addHandler(stream_handler)
Other con guration methods¶
We have seen how to con gure logging creating loggers, handlers,
and formatters explicitely in code. There are two other con gration
methods: - Creating a logging con g le and reading it using the
fileConfig() function. See example below. - Creating a dictionary
of con guration information and passing it to the dictConfig()
function. See https://docs.python.org/3/library/
logging.con g.html#logging.con g.dictCon g for more information.

.conf le¶
Create a .conf (or sometimes stored as .ini) le, de ne the loggers,
handlers, and formatters and provide the names as keys. After their
names are de ned, they are con gured by adding the words
logger, handler, and formatter before their names separated by an
underscore. Then you can set the properties for each logger,
handler, and formatter. In the example below, the root logger and a
logger named simpleExample will be con gured with a
StreamHandler.

logging.conf
[loggers]
keys=root,simpleExample

[handlers]
keys=consoleHandler

[formatters]
keys=simpleFormatter

[logger_root]
level=DEBUG
53
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
fi
handlers=consoleHandler

[logger_simpleExample]
level=DEBUG
handlers=consoleHandler
qualname=simpleExample
propagate=0

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=simpleFormatter
args=(sys.stdout,)

[formatter_simpleFormatter]
format=%(asctime)s - %(name)s - %(levelname)s - %
(message)s
# Then use the config file in the code
import logging
import logging.config

logging.config.fileConfig('logging.conf')

# create logger with the name from the config file.

# This logger now has StreamHandler with DEBUG Level and
the specified format
logger = logging.getLogger('simpleExample')

logger.debug('debug message')
logger.info('info message')
Capture Stack traces¶
Logging the traceback in your exception logs can be very helpful
for troubleshooting issues. You can capture the traceback in
logging.error() by setting the exc_info parameter to True.

import logging

try:
a = [1, 2, 3]
value = a[3]
except IndexError as e:
logging.error(e)
logging.error(e, exc_info=True)
54
ERROR:root:list index out of range
ERROR:root:list index out of range
Traceback (most recent call last):
File "<ipython-input-6-df97a133cbe6>", line 5, in
<module>
value = a[3]
IndexError: list index out of range
If you don't capture the correct Exception, you can also use the
traceback.format_exc() method to log the exception.

import logging
import traceback

try:
a = [1, 2, 3]
value = a[3]
except:
logging.error("uncaught exception: %s",
traceback.format_exc())
Rotating FileHandler¶
When you have a large application that logs many events to a le,
and you only need to keep track of the most recent events, then
use a RotatingFileHandler that keeps the les small. When the log
reaches a certain number of bytes, it gets "rolled over". You can
also keep multiple backup log les before overwriting them.

import logging
from logging.handlers import RotatingFileHandler

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# roll over after 2KB, and keep backup logs app.log.1,

app.log.2 , etc.
handler = RotatingFileHandler('app.log', maxBytes=2000,
backupCount=5)
logger.addHandler(handler)

for _ in range(10000):
logger.info('Hello, world!')
TimedRotatingFileHandler¶
55
fi
fi
fi
If your application will be running for a long time, you can use a
TimedRotatingFileHandler. This will create a rotating log based on
how much time has passed. Possible time conditions for the when
parameter are: - second (s) - minute (m) - hour (h) - day (d) - w0-w6
(weekday, 0=Monday) - midnight

import logging
import time
from logging.handlers import TimedRotatingFileHandler

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)

# This will create a new log file every minute, and 5

backup files with a timestamp before overwriting old logs.
handler = TimedRotatingFileHandler('timed_test.log',
when='m', interval=1, backupCount=5)
logger.addHandler(handler)

for i in range(6):
logger.info('Hello, world!')
time.sleep(50)
Logging in JSON Format¶
If your application generates many logs from di erent modules,
and especially in a microservice architecture, it can be challenging
to locate the important logs for your analysis. Therefore, it is best
practice to log your messages in JSON format, and send them to a
centralized log management system. Then you can easily search,
visualize, and analyze your log records.
I would recommend using this Open Source JSON logger: https://
github.com/madzak/python-json-logger
pip install python-json-logger

import logging
from pythonjsonlogger import jsonlogger

logger = logging.getLogger()

logHandler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
logHandler.setFormatter(formatter)

56
ff
logger.addHandler(logHandler)

JSON - Advanced Python 11

JSON (JavaScript Object Notation) is a leightweight data
format for data exchange.

JSON (JavaScript Object Notation) is a leightweight data format for

data exchange. In Python you have the built-in json module for
encoding and decoding JSON data. Simply import it and you are
ready to work with JSON data:

import json
Some advantages of JSON:

- JSON exists as a "sequence of bytes" which is very useful in the

case we need to transmit (stream) data over a network.

- Compared to XML, JSON is much smaller, translating into faster

data transfers, and better experiences.

- JSON is extremely human-friendly since it is textual, and

simultaneously machine-friendly.

JSON format¶
{
"firstName": "Jane",
"lastName": "Doe",
"hobbies": ["running", "swimming", "singing"],
"age": 28,
"children": [
{
"firstName": "Alex",
"age": 5
},
{
"firstName": "Bob",
"age": 7
}
]
57
 }
JSON supports primitive types (strings, numbers, boolean), as well
as nested arrays and objects. Simple Python objects are translated
to JSON according to the following conversion:

JSON
Python

dict object
list, tuple array
str string
int, long, oat number
TRUE TRUE
FALSE FALSE
None null

From Python to JSON (Serialization,

Encode)¶
Convert Python objects into a JSON string with the json.dumps()
method.

import json

person = {"name": "John", "age": 30, "city": "New York",

"hasChildren": False, "titles": ["engineer",
"programmer"]}

# convert into JSON:

person_json = json.dumps(person)
# use different formatting style
person_json2 = json.dumps(person, indent=4, separators=(";
", "= "), sort_keys=True)

# the result is a JSON string:

print(person_json)
print(person_json2)
{"name": "John", "age": 30, "city": "New York",
"hasChildren": false, "titles":["engineer", "programmer"]}
{
58
fl
 "age"= 30;
"city"= "New York";
"hasChildren"= false;
"name"= "John";
"titles"= [
"engineer";
"programmer"
]
}
Or convert Python objects into JSON objects and save them into a
le with the json.dump() method.

import json

person = {"name": "John", "age": 30, "city": "New York",

"hasChildren": False, "titles": ["engineer",
"programmer"]}

with open('person.json', 'w') as f:

json.dump(person, f) # you can also specify indent
etc...
FROM JSON to Python (Deserialization,
Decode)¶
Convert a JSON string into a Python object with the json.loads()
method. The result will be a Python dictionary.

import json
person_json = """
{
"age": 30,
"city": "New York",
"hasChildren": false,
"name": "John",
"titles": [
"engineer",
"programmer"
]
}
"""
person = json.loads(person_json)
print(person)

59
fi
{'age': 30, 'city': 'New York', 'hasChildren': False,
'name': 'John', 'titles': ['engineer', 'programmer']}
Or load data from a le and convert it to a Python object with the
json.load() method.

import json

with open('person.json', 'r') as f:

person = json.load(f)
print(person)
{'name': 'John', 'age': 30, 'city': 'New York',
'hasChildren': False, 'titles': ['engineer',
'programmer']}
Working with Custom Objects¶
Encoding¶
Encoding a custom object with the default JSONEncoder will raise a
TypeError. We can specify a custom encoding function that will
store the class name and all object variables in a dictionary. Use
this function for the default argument in the json.dump() method.

import json
def encode_complex(z):
if isinstance(z, complex):
# just the key of the class name is important, the
value can be arbitrary.
return {z.__class__.__name__: True, "real":z.real,
"imag":z.imag}
else:
raise TypeError(f"Object of type
'{z.__class__.__name__}' is not JSON serializable")

z = 5 + 9j
zJSON = json.dumps(z, default=encode_complex)
print(zJSON)
{"complex": true, "real": 5.0, "imag": 9.0}
You can also create a custom Encoder class, and overwrite the
default() method. Use this for the cls argument in the
json.dump() method, or use the encoder directly.

from json import JSONEncoder

60
fi
class ComplexEncoder(JSONEncoder):

def default(self, o):

if isinstance(z, complex):
return {z.__class__.__name__: True,
"real":z.real, "imag":z.imag}
# Let the base class default method handle other
objects or raise a TypeError
return JSONEncoder.default(self, o)

z = 5 + 9j
zJSON = json.dumps(z, cls=ComplexEncoder)
print(zJSON)
# or use encoder directly:
zJson = ComplexEncoder().encode(z)
print(zJSON)
{"complex": true, "real": 5.0, "imag": 9.0}
{"complex": true, "real": 5.0, "imag": 9.0}
Decoding¶
Decoding a custom object with the defaut JSONDecoder is possible,
but it will be decoded into a dictionary. Write a custom decode
function that will take a dictionary as input, and creates your
custom object if it can nd the object class name in the dictionary.
Use this function for the object_hook argument in the json.load()
method.

# Possible but decoded as a dictionary

z = json.loads(zJSON)
print(type(z))
print(z)

def decode_complex(dct):
if complex.__name__ in dct:
return complex(dct["real"], dct["imag"])
return dct

# Now the object is of type complex after decoding

z = json.loads(zJSON, object_hook=decode_complex)
print(type(z))
print(z)
<class 'dict'>
{'complex': True, 'real': 5.0, 'imag': 9.0}
<class 'complex'>
61
fi
(5+9j)
Template encode and decode
functions¶
This works for all custom classes if all class variables are given in
the __init__ method.

class User:
# Custom class with all class variables given in the
__init__()
def __init__(self, name, age, active, balance,
friends):
self.name = name
self.age = age
self.active = active
self.balance = balance
self.friends = friends

class Player:
# Other custom class
def __init__(self, name, nickname, level):
self.name = name
self.nickname = nickname
self.level = level

def encode_obj(obj):
"""
Takes in a custom object and returns a dictionary
representation of the object.
This dict representation also includes the object's
module and class names.
"""

# Populate the dictionary with object meta data

obj_dict = {
"__class__": obj.__class__.__name__,
"__module__": obj.__module__
}

# Populate the dictionary with object properties

obj_dict.update(obj.__dict__)

62
 return obj_dict

def decode_dct(dct):
"""
Takes in a dict and returns a custom object associated
with the dict.
It makes use of the "__module__" and "__class__"
metadata in the dictionary
to know which object type to create.
"""
if "__class__" in dct:
# Pop ensures we remove metadata from the dict to
leave only the instance arguments
class_name = dct.pop("__class__")

# Get the module name from the dict and import it

module_name = dct.pop("__module__")

# We use the built in __import__ function since

the module name is not yet known at runtime
module = __import__(module_name)

# Get the class from the module

class_ = getattr(module,class_name)

# Use dictionary unpacking to initialize the

object
# Note: This only works if all __init__()
arguments of the class are exactly the dict keys
obj = class_(**dct)
else:
obj = dct
return obj

# User class works with our encoding and decoding methods

user = User(name = "John",age = 28, friends = ["Jane",
"Tom"], balance = 20.70, active = True)

userJSON = json.dumps(user,default=encode_obj,indent=4,
sort_keys=True)
print(userJSON)

user_decoded = json.loads(userJSON,
object_hook=decode_dct)
63
print(type(user_decoded))

# Player class also works with our custom encoding and

decoding
player = Player('Max', 'max1234', 5)
playerJSON =
json.dumps(player,default=encode_obj,indent=4,
sort_keys=True)
print(playerJSON)

player_decoded = json.loads(playerJSON,
object_hook=decode_dct)
print(type(player_decoded))
{
"__class__": "User",
"__module__": "__main__",
"active": true,
"age": 28,
"balance": 20.7,
"friends": [
"Jane",
"Tom"
],
"name": "John"
}
<class '__main__.User'>
{
"__class__": "Player",
"__module__": "__main__",
"level": 5,
"name": "Max",
"nickname": "max1234"
}
<class ‚__main__.Player'>

Random Numbers -
Advanced Python 12
Python de nes a set of functions that are used to generate or
manipulate random numbers. This tutorial covers the random
module.
64
fi
Python de nes a set of functions that are used to generate or
manipulate random numbers. This article covers:

• the random module

• reproduce numbers with random.seed()
• create cryptographically strong random numbers with the
secrets module
• create random nd arrays with numpy.random
The random module¶
This module implements pseudo-random number generators for
various distributions. It uses the Mersenne Twister algorithm
(https://en.wikipedia.org/wiki/Mersenne_Twister) as its core
generator. It is called pseudo-random, because the numbers seem
random, but are reproducable.

import random

# random float in [0,1)

a = random.random()
print(a)

# random float in range [a,b]

a = random.uniform(1,10)
print(a)

# random integer in range [a,b]. b is included

a = random.randint(1,10)
print(a)

# random integer in range [a,b). b is excluded

a = random.randrange(1,10)
print(a)

# random float from a normal distribution with mu and

sigma
a = random.normalvariate(0, 1)
print(a)

# choose a random element from a sequence

a = random.choice(list("ABCDEFGHI"))
65
fi
print(a)

# choose k unique random elements from a sequence

a = random.sample(list("ABCDEFGHI"), 3)
print(a)

# choose k elements with replacement, and return k sized

list
a = random.choices(list("ABCDEFGHI"),k=3)
print(a)

# shuffle list in place

a = list("ABCDEFGHI")
random.shuffle(a)
print(a)
0.10426373452067317
3.34983979352444
3
4
-1.004568769635799
E
['G', 'C', 'B']
['E', 'D', 'E']
['D', 'I', 'G', 'H', 'E', 'B', 'C', 'F', 'A']
The seed generator¶
With random.seed(), you can make results reproducible, and the
chain of calls after random.seed() will produce the same trail of
data. The sequence of random numbers becomes deterministic, or
completely determined by the seed value.

print('Seeding with 1...\n')

random.seed(1)
print(random.random())
print(random.uniform(1,10))
print(random.choice(list("ABCDEFGHI")))

print('\nRe-seeding with 42...\n')

random.seed(42) # Re-seed

print(random.random())
print(random.uniform(1,10))
print(random.choice(list("ABCDEFGHI")))
66
print('\nRe-seeding with 1...\n')
random.seed(1) # Re-seed

print(random.random())
print(random.uniform(1,10))
print(random.choice(list("ABCDEFGHI")))

print('\nRe-seeding with 42...\n')

random.seed(42) # Re-seed

print(random.random())
print(random.uniform(1,10))
print(random.choice(list("ABCDEFGHI")))
Seeding with 1...

0.13436424411240122
8.626903632435095
B

Re-seeding with 42...

0.6394267984578837
1.2250967970040025
E

Re-seeding with 1...

0.13436424411240122
8.626903632435095
B

Re-seeding with 42...

0.6394267984578837
1.2250967970040025
E
The secrets module¶
The secrets module is used for generating cryptographically strong
random numbers suitable for managing data such as passwords,
account authentication, security tokens, and related secrets.
In particularly, secrets should be used in preference to the default

67
pseudo-random number generator in the random module, which is
designed for modelling and simulation, not security or
cryptography.

import secrets

# random integer in range [0, n).

a = secrets.randbelow(10)
print(a)

# return an integer with k random bits.

a = secrets.randbits(5)
print(a)

# choose a random element from a sequence

a = secrets.choice(list("ABCDEFGHI"))
print(a)
6
6
E
Random numbers with NumPy¶
Create random numbers for nd arrays. The NumPy pseudorandom
number generator is di erent from the Python standard library
pseudorandom number generator.
Importantly, seeding the Python pseudorandom number generator
does not impact the NumPy pseudorandom number generator. It
must be seeded and used separately.

import numpy as np

np.random.seed(1)
# rand(d0,d1,…,dn)
# generate nd array with random floats, arrays has size
(d0,d1,…,dn)
print(np.random.rand(3))
# reset the seed
np.random.seed(1)
print(np.random.rand(3))

# generate nd array with random integers in range [a,b)

with size n
values = np.random.randint(0, 10, (5,3))
68
ff
print(values)

# generate nd array with Gaussian values, array has size

(d0,d1,…,dn)
# values from standard normal distribution with mean 0.0
and standard deviation 1.0
values = np.random.randn(5)
print(values)

# randomly shuffle a nd array.

# only shuffles the array along the first axis of a multi-
dimensional array
arr = np.array([[1,2,3], [4,5,6], [7,8,9]])
np.random.shuffle(arr)
print(arr)
[4.17022005e-01 7.20324493e-01 1.14374817e-04]
[4.17022005e-01 7.20324493e-01 1.14374817e-04]
[[5 0 0]
[1 7 6]
[9 2 4]
[5 2 4]
[2 4 7]]
[-2.29230928 -1.41555249 0.8858294 0.63190187
0.04026035]
[[4 5 6]
[7 8 9]
[1 2 3]]

Decorators - Advanced
Python 13
A decorator is a function that takes another function and
extends the behavior of this function without explicitly
modifying it.

There are 2 kinds of decorators:

• Function decoratos
• Class decorators
A function is decorated with the @ symbol:
69
@my_decorator
def my_function():
pass
Function decorators¶
In order to understand the decorator pattern, we have to
understand that functions in Python are rst class objects, which
means that – like any other object – they can be de ned inside
another function, passed as argument to another function, or
returned from other functions.

A decorator is a function that takes another function as argument,

wraps its behaviour inside an inner function, and returns the
wrapped function. As a consequence, the decorated function now
has extended functionality!

# A decorator function takes another function as argument,

wraps its behaviour inside
# an inner function, and returns the wrapped function.
def start_end_decorator(func):

def wrapper():
print('Start')
func()
print('End')
return wrapper

def print_name():
print('Alex')

print_name()

print()

# Now wrap the function by passing it as argument to the

decorator function
# and asign it to itself -> Our function has extended
behaviour!
print_name = start_end_decorator(print_name)
print_name()
Alex

Start
70
fi
fi
 Alex
End
The decorator syntax¶
Instead of wrapping our function and asigning it to itself, we can
achieve the same thing simply by decorating our function with an @.

@start_end_decorator
def print_name():
print('Alex')

print_name()
Start
Alex
End
What about function arguments¶
If our function has input arguments and we try to wrap it with our
decorator above, it will raise a TypeError since we have to call our
function inside the wrapper with this arguments, too. However, we
can x this by using *args and **kwargs in the inner function:

def start_end_decorator_2(func):

def wrapper(*args, **kwargs):

print('Start')
func(*args, **kwargs)
print('End')
return wrapper

@start_end_decorator_2
def add_5(x):
return x + 5

result = add_5(10)
print(result)
Start
End
None
Return values¶

71
fi
 Note that in the example above, we do not get the result back, so
as next step we also have to return the value from our inner
function:

def start_end_decorator_3(func):

def wrapper(*args, **kwargs):

print('Start')
result = func(*args, **kwargs)
print('End')
return result
return wrapper

@start_end_decorator_3
def add_5(x):
return x + 5

result = add_5(10)
print(result)
Start
End
15
What about the function identity?¶
If we have a look at the name of our decorated function, and
inspect it with the built-in help function, we notice that Python
thinks our function is now the wrapped inner function of the
decorator function.

print(add_5.__name__)
help(add_5)
wrapper
Help on function wrapper in module __main__:

wrapper(*args, **kwargs)
To x this, use the functools.wraps decorator, which will preserve
the information about the original function. This is helpful for
introspection purposes, i.e. the ability of an object to know about
its own attributes at runtime:

import functools
def start_end_decorator_4(func):

72
fi
 @functools.wraps(func)
def wrapper(*args, **kwargs):
print('Start')
result = func(*args, **kwargs)
print('End')
return result
return wrapper

@start_end_decorator_4
def add_5(x):
return x + 5
result = add_5(10)
print(result)
print(add_5.__name__)
help(add_5)
Start
End
15
add_5
Help on function add_5 in module __main__:

add_5(x)
The nal template for own decorators¶
Now that we have all parts, our template for any decorator looks
like this:

import functools

def my_decorator(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
# Do something before
result = func(*args, **kwargs)
# Do something after
return result
return wrapper
Decorator function arguments¶
Note that functools.wraps is a decorator that takes an argument
for itself. We can think of this as 2 inner functions, so an inner
function within an inner function. To make this clearer, we look at

73
fi
another example: A repeat decorator that takes a number as input.
Within this function, we have the actual decorator function that
wraps our function and extends its behaviour within another inner
function. In this case, it repeats the input function the given number
of times.

def repeat(num_times):
def decorator_repeat(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
for _ in range(num_times):
result = func(*args, **kwargs)
return result
return wrapper
return decorator_repeat

@repeat(num_times=3)
def greet(name):
print(f"Hello {name}")

greet('Alex')
Hello Alex
Hello Alex
Hello Alex
Nested Decorators¶
We can apply several decorators to a function by stacking them on
top of each other. The decorators are being executed in the order
they are listed.

# a decorator function that prints debug information about

the wrapped function
def debug(func):
@functools.wraps(func)
def wrapper(*args, **kwargs):
args_repr = [repr(a) for a in args]
kwargs_repr = [f"{k}={v!r}" for k, v in
kwargs.items()]
signature = ", ".join(args_repr + kwargs_repr)
print(f"Calling {func.__name__}({signature})")
result = func(*args, **kwargs)
print(f"{func.__name__!r} returned {result!r}")
return result
74
 return wrapper

@debug
@start_end_decorator_4
def say_hello(name):
greeting = f'Hello {name}'
print(greeting)
return greeting

# now `debug` is executed first and calls

`@start_end_decorator_4`, which then calls `say_hello`
say_hello(name='Alex')
Calling say_hello(name='Alex')
Start
Hello Alex
End
'say_hello' returned 'Hello Alex'
Class decorators¶
We can also use a class as a decorator. Therefore, we have to
implement the __call__() method to make our object callable.
Class decorators are typically used to maintain a state, e.g. here
we keep track of the number of times our function is executed. The
__call__ method does essentially the same thing as the wrapper()
method we have seen earlier. It adds some functionality, executes
the function, and returns its result. Note that here we use
functools.update_wrapper() instead of functools.wraps to
preserve the information about our function.

import functools

class CountCalls:
# the init needs to have the func as argument and
stores it
def __init__(self, func):
functools.update_wrapper(self, func)
self.func = func
self.num_calls = 0

# extend functionality, execute function, and return

the result
def __call__(self, *args, **kwargs):
self.num_calls += 1
75
 print(f"Call {self.num_calls} of
{self.func.__name__!r}")
return self.func(*args, **kwargs)

@CountCalls
def say_hello(num):
print("Hello!")

say_hello(5)
say_hello(5)
Call 1 of 'say_hello'
Hello!
Call 2 of 'say_hello'
Hello!
Some typical use cases¶
• Use a timer decorator to calculate the execution time of a
function
• Use a debug decorator to print out some more information
about the called function and its arguments
• Use a check decorator to check if the arguments ful ll some
requirements and adapt the bevaviour accordingly
• Register functions (plugins)
• Slow down code with time.sleep() to check network
behaviour
• Cache the return values for memoization (https://
en.wikipedia.org/wiki/Memoization)
• Add information or update a state

Generators - Advanced
Python 14
Generators are functions that can be paused and resumed on
the y, returning an object that can be iterated over.

76
fl
fi
Unlike lists, they are lazy and thus produce items one at a time and
only when asked. So they are much more memory e cient when
dealing with large datasets.
A generator is de ned like a normal function but with the yield
statement instead of return.

def my_generator():
yield 1
yield 2
yield 3
Execution of a generator function¶
Calling the function does not execute it. Instead, the function
returns a generator object which is used to control execution.
Generator objects execute when next() is called. When calling
next() the rst time, execution begins at the start of the function
and continues until the rst yield statement where the value to the
right of the statement is returned. Subsequent calls to next()
continue from the yield statement (and loop around) until another
yield is reached. If yield is not called because of a condition or
the end is reached, a StopIteration exception is raised:

def countdown(num):
print('Starting')
while num > 0:
yield num
num -= 1

# this will not print 'Starting'

cd = countdown(3)

# this will print 'Starting' and the first value

print(next(cd))

# will print the next values

print(next(cd))
print(next(cd))

# this will raise a StopIteration

print(next(cd))
Starting
3

77
fi
fi
fi
ffi
2
1
----------------------------------------------------------
-----------------
StopIteration Traceback (most
recent call last)
<ipython-input-1-3941498e0bf0> in <module>
16
17 # this will raise a StopIteration
---> 18 print(next(cd))

StopIteration:
# you can iterate over a generator object with a for in
loop
cd = countdown(3)
for x in cd:
print(x)
Starting
3
2
1
# you can use it for functions that take iterables as
input
cd = countdown(3)
sum_cd = sum(cd)
print(sum_cd)

cd = countdown(3)
sorted_cd = sorted(cd)
print(sorted_cd)
Starting
6
Starting
[1, 2, 3]
Big advantage: Generators save
memory!¶
Since the values are generated lazily, i.e. only when needed, it
saves a lot of memory, especially when working with large data.
Furthermore, we do not need to wait until all the elements have
been generated before we start to use them.

78
# without a generator, the complete sequence has to be
stored here in a list
def firstn(n):
num, nums = 0, []
while num < n:
nums.append(num)
num += 1
return nums

sum_of_first_n = sum(firstn(1000000))
print(sum_of_first_n)
import sys
print(sys.getsizeof(firstn(1000000)), "bytes")
499999500000
8697464 bytes
# with a generator, no additional sequence is needed to
store the numbers
def firstn(n):
num = 0
while num < n:
yield num
num += 1

sum_of_first_n = sum(firstn(1000000))
print(sum_of_first_n)
import sys
print(sys.getsizeof(firstn(1000000)), "bytes")
499999500000
120 bytes
Another example: Fibonacci numbers¶
def fibonacci(limit):
a, b = 0, 1 # first two fibonacci numbers
while a < limit:
yield a
a, b = b, a + b

fib = fibonacci(30)
# generator objects can be converted to a list (only used
for printing here)
print(list(fib))
[0, 1, 1, 2, 3, 5, 8, 13, 21]
Generator expressions¶
79
Just like list comprehensions, generators can be written in the
same syntax except with parenthesis instead of square brackets.
Be careful not to mix them up, since generator expressions are
often slower than list comprehensions because of the overhead of
function calls (https://stackover ow.com/questions/11964130/list-
comprehension-vs-generator-expressions-weird-timeit-results/
11964478#11964478)

# generator expression
mygenerator = (i for i in range(1000) if i % 2 == 0)
print(sys.getsizeof(mygenerator), "bytes")

# list comprehension
mylist = [i for i in range(1000) if i % 2 == 0]
print(sys.getsizeof(mylist), "bytes")
120 bytes
4272 bytes
Concept behind a generator¶
This class implements our generator as an iterable object. It has to
implement __iter__ and __next__ to make it iterable, keep track of
the current state (the current number in this case), and take care of
a StopIteration. It can be used to understand the concept behind
generators. However, there is a lot of boilerplate code, and the
logic is not as clear as with a simple function using the yield
keyword.

class firstn:
def __init__(self, n):
self.n = n
self.num = 0

def __iter__(self):
return self

def __next__(self):
if self.num < self.n:
cur = self.num
self.num += 1
return cur
else:
raise StopIteration()

80
fl
firstn_object = firstn(1000000)
print(sum(firstn_object))
499999500000

Threading vs
Multiprocessing - Advanced
Python 15
Overview and comparison of threads and processes, and how
to use it in Python.

We have two common approaches to run code in parallel (achieve

multitasking and speed up your program) : via threads or via
multiple processes.

Process¶
A Process is an instance of a program, e.g. a Python interpreter.
They are independent from each other and do not share the same
memory.

Key facts: - A new process is started independently from the rst

process - Takes advantage of multiple CPUs and cores - Separate
memory space - Memory is not shared between processes - One
GIL (Global interpreter lock) for each process, i.e. avoids GIL
limitation - Great for CPU-bound processing - Child processes are
interruptable/killable

• Starting a process is slower that starting a thread

• Larger memory footprint
• IPC (inter-process communication) is more complicated
Threads¶

81
fi
A thread is an entity within a process that can be scheduled for
execution (Also known as "leightweight process"). A Process can
spawn multiple threads. The main di erence is that all threads
within a process share the same memory.

Key facts: - Multiple threads can be spawned within one process -

Memory is shared between all threads - Starting a thread is faster
than starting a process - Great for I/O-bound tasks - Leightweight -
low memory footprint

• One GIL for all threads, i.e. threads are limited by GIL
• Multithreading has no e ect for CPU-bound tasks due to the
GIL
• Not interruptible/killable -> be careful with memory leaks
• increased potential for race conditions
Threading in Python¶
Use the threading module.

Note: The following example usually won't bene t from multiple

threads since it is CPU-bound. It should just show the example of
how to use threads.

from threading import Thread

def square_numbers():
for i in range(1000):
result = i * i

if __name__ == "__main__":
threads = []
num_threads = 10

# create threads and asign a function for each thread

for i in range(num_threads):
thread = Thread(target=square_numbers)
threads.append(thread)

# start all threads

82
ff
ff
fi
 for thread in threads:
thread.start()

# wait for all threads to finish

# block the main thread until these threads are
finished
for thread in threads:
thread.join()
When is Threading useful¶
Despite the GIL it is useful for I/O-bound tasks when your program
has to talk to slow devices, like a hard drive or a network
connection. With threading the program can use the time waiting
for these devices and intelligently do other tasks in the meantime.

Example: Download website information from multiple sites. Use a

thread for each site.

Multiprocessing¶
Use the multiprocessing module. The syntax is very similar to
above.

from multiprocessing import Process

import os

def square_numbers():
for i in range(1000):
result = i * i

if __name__ == "__main__":
processes = []
num_processes = os.cpu_count()

# create processes and asign a function for each

process
for i in range(num_processes):
process = Process(target=square_numbers)
processes.append(process)

83
 # start all processes
for process in processes:
process.start()

# wait for all processes to finish

# block the main thread until these processes are
finished
for process in processes:
process.join()
When is Multiprocessing useful¶
It is useful for CPU-bound tasks that have to do a lot of CPU
operations for a large amount of data and require a lot of
computation time. With multiprocessing you can split the data into
equal parts an do parallel computing on di erent CPUs.

Example: Calculate the square numbers for all numbers from 1 to

1000000. Divide the numbers into equal sized parts and use a
process for each subset.

GIL - Global interpreter lock¶

This is a mutex (or a lock) that allows only one thread to hold
control of the Python interpreter. This means that the GIL allows
only one thread to execute at a time even in a multi-threaded
architecture.

Why is it needed?¶
It is needed because CPython's (reference implementation of
Python) memory management is not thread-safe. Python uses
reference counting for memory management. It means that objects
created in Python have a reference count variable that keeps track
of the number of references that point to the object. When this
count reaches zero, the memory occupied by the object is
released. The problem was that this reference count variable
needed protection from race conditions where two threads
increase or decrease its value simultaneously. If this happens, it
can cause either leaked memory that is never released or

84
ff
incorrectly release the memory while a reference to that object still
exists.

How to avoid the GIL¶

The GIL is very controversial in the Python community. The main
way to avoid the GIL is by using multiprocessing instead of
threading. Another (however uncomfortable) solution would be to
avoid the CPython implementation and use a free-threaded Python
implementation like Jython or IronPython. A third option is to move
parts of the application out into binary extensions modules, i.e. use
Python as a wrapper for third party libraries (e.g. in C/C++). This is
the path taken by numpy and scipy.

Multithreading - Advanced
Python 16
In this tutorial we talk about how to use the `threading` module
in Python.

In this article we talk about how to use the threading module in

Python.

• How to create and start multiple threads

• How to wait for threads to complete
• How to share data between threads
• How to use Locks to prevent race conditions
• What is a daemon thread
• How to use a Queue for thread-safe data/task processing.
Create and run threads¶
You create a thread with threading.Thread(). It takes two
important arguments:
85
 • target: a callable object (function) for this thread to be
invoked when the thread starts
• args: the (function) arguments for the target function. This
must be a tuple
Start a thread with thread.start()

Call thread.join() to tell the program that it should wait for this
thread to complete before it continues with the rest of the code.

from threading import Thread

def square_numbers():
for i in range(1000):
result = i * i

if __name__ == "__main__":
threads = []
num_threads = 10

# create threads and asign a function for each thread

for i in range(num_threads):
thread = Thread(target=square_numbers)
threads.append(thread)

# start all threads

for thread in threads:
thread.start()

# wait for all threads to finish

# block the main thread until these threads are
finished
for thread in threads:
thread.join()
Share data between threads¶
Since threads live in the same memory space, they have access to
the same (public) data. Thus, you can for example simply use a
global variable to which all threads have read and write access.

Task: Create two threads, each thread should access the current
database value, modify it (in this case only increase it by 1), and
86
write the new value back into the database value. Each thread
should do this operation 10 times.

from threading import Thread

import time

# all threads can access this global variable

database_value = 0

def increase():
global database_value # needed to modify the global
value

# get a local copy (simulate data retrieving)

local_copy = database_value

# simulate some modifying operation

local_copy += 1
time.sleep(0.1)

# write the calculated new value into the global

variable
database_value = local_copy

if __name__ == "__main__":

print('Start value: ', database_value)

t1 = Thread(target=increase)
t2 = Thread(target=increase)

t1.start()
t2.start()

t1.join()
t2.join()

print('End value:', database_value)

print('end main')
Start value: 0
End value: 1
end main

87
How to use Locks¶
Notice that in the above example, the 2 threads should increment
the value by 1, so 2 increment operations are performed. But why
is the end value 1 and not 2?

Race condition¶
A race condition happened here. A race condition occurs when two
or more threads can access shared data and they try to change it
at the same time. Because the thread scheduling algorithm can
swap between threads at any time, you don't know the order in
which the threads will attempt to access the shared data. In our
case, the rst thread accesses the database_value (0) and stores it
in a local copy. It then increments it (local_copy is now 1). With our
time.sleep() function that just simulates some time consuming
operations, the programm will swap to the second thread in the
meantime. This will also retrieve the current database_value (still 0)
and increment the local_copy to 1. Now both threads have a local
copy with value 1, so both will write the 1 into the global
database_value. This is why the end value is 1 and not 2.

Avoid race conditions with Locks¶

A lock (also known as mutex) is a synchronization mechanism for
enforcing limits on access to a resource in an environment where
there are many threads of execution. A Lock has two states:
locked and unlocked. If the state is locked, it does not allow other
concurrent threads to enter this code section until the state is
unlocked again.

Two functions are important: - lock.acquire() : This will lock the

state and block - lock.release() : This will unlock the state again.

Important: You should always release the block again after it was
acquired!

In our example the critical code section where database values are
retrieved and modi ed is now locked. This prevents the second
88
fi
fi
thread from mody ng the global data at the same time. Not much
has changed in our code. All new changes are commented in the
code below.

# import Lock
from threading import Thread, Lock
import time

database_value = 0

def increase(lock):
global database_value

# lock the state

lock.acquire()

local_copy = database_value
local_copy += 1
time.sleep(0.1)
database_value = local_copy

# unlock the state

lock.release()

if __name__ == "__main__":

# create a lock
lock = Lock()

print('Start value: ', database_value)

# pass the lock to the target function

t1 = Thread(target=increase, args=(lock,)) # notice
the comma after lock since args must be a tuple
t2 = Thread(target=increase, args=(lock,))

t1.start()
t2.start()

t1.join()
t2.join()

print('End value:', database_value)

89
fi
 print('end main')
Start value: 0
End value: 2
end main
Use the lock as a context manager¶
After lock.acquire() you should never forget to call
lock.release() to unblock the code. You can also use a lock as a
context manager, wich will safely lock and unlock your code. It is
recommended to use a lock this way:

def increase(lock):
global database_value

with lock:
local_copy = database_value
local_copy += 1
time.sleep(0.1)
database_value = local_copy
Using Queues in Python¶
Queues can be used for thread-safe/process-safe data exchanges
and data processing both in a multithreaded and a multiprocessing
environment.

The queue¶
A queue is a linear data structure that follows the First In First Out
(FIFO) principle. A good example is a queue of customers that are
waiting in line, where the customer that came rst is served rst.

from queue import Queue

# create queue
q = Queue()

# add elements
q.put(1) # 1
q.put(2) # 2 1
q.put(3) # 3 2 1

90
fi
fi
# now q looks like this:
# back --> 3 2 1 --> front

# get and remove first element

first = q.get() # --> 1
print(first)

# q looks like this:

# back --> 3 2 --> front
1
Using a queue in multithreading¶
Operations with a queue are thread-safe. Important methods are:

• q.get() : Remove and return the rst item. By default, it

blocks until the item is available.
• q.put(item) : Puts element at the end of the queue. By
default, it blocks until a free slot is available.
• q.task_done() : Indicate that a formerly enqueued task is
complete. For each get() you should call this after you are
done with your task for this item.
• q.join() : Blocks until all items in the queue have been
gotten and proccessed (task_done() has been called for
each item).
• q.empty() : Return True if the queue is empty.
The following example uses a queue to exchange numbers from
0...19. Each thread invokes the worker method. Inside the in nite
loop the thread is waiting until items are available due to the
blocking q.get() call. When items are available, they are
processed (i.e. just printed here), and then q.task_done() tells the
queue that processing is complete. In the main thread, 10 daemon
threads are created. This means that they automatically die when
the main thread dies, and thus the worker method and in nite loop
is no longer invoked. Then the queue is lled with items and the
worker method can continue with available items. At the end
q.join() is necessary to block the main thread until all items have
been gotten and proccessed.

from threading import Thread, Lock, current_thread

91
fi
fi
fi
fi
from queue import Queue

def worker(q, lock):

while True:
value = q.get() # blocks until the item is
available

# do stuff...
with lock:
# prevent printing at the same time with this
lock
print(f"in {current_thread().name} got
{value}")
# ...

# For each get(), a subsequent call to task_done()

tells the queue
# that the processing on this item is complete.
# If all tasks are done, q.join() can unblock
q.task_done()

if __name__ == '__main__':
q = Queue()
num_threads = 10
lock = Lock()

for i in range(num_threads):
t = Thread(name=f"Thread{i+1}", target=worker,
args=(q, lock))
t.daemon = True # dies when the main thread dies
t.start()

# fill the queue with items

for x in range(20):
q.put(x)

q.join() # Blocks until all items in the queue have

been gotten and processed.

print('main done')
in Thread1 got 0
in Thread2 got 1
in Thread2 got 11
in Thread2 got 12
92
in Thread2 got 13
in Thread2 got 14
in Thread2 got 15
in Thread2 got 16
in Thread2 got 17
in Thread2 got 18
in Thread2 got 19
in Thread8 got 5
in Thread4 got 9
in Thread1 got 10
in Thread5 got 2
in Thread6 got 3
in Thread9 got 6
in Thread7 got 4
in Thread10 got 7
in Thread3 got 8
main done
Daemon threads¶
In the above example, daemon threads are used. Daemon threads
are background threads that automatically die when the main
program ends. This is why the in nite loops inside the worker
methods can be exited. Without a daemon process we would have
to use a signalling mechanism such as a threading.Event to stop
the worker. But be careful with daemon processes: They are
abruptly stopped and their resources (e.g. open les or database
transactions) may not be released/completed properly.

Multiprocessing - Advanced
Python 17
In this tutorial we talk about how to use the `multiprocessing`
module in Python.

In this article we talk about how to use the multiprocessing

module in Python.

• How to create and start multiple processes

• How to wait for processes to complete
93
fi
fi
 • How to share data between processes
• How to use Locks to prevent race conditions
• How to use a Queue for process-safe data/task processing.
• How to use a Pool to manage multiple worker processes
Create and run processes¶
You create a process with multiprocessing.Process(). It takes
two important arguments:

• target: a callable object (function) for this process to be

invoked when the process starts
• args: the (function) arguments for the target function. This
must be a tuple
Start a process with process.start()

Call process.join() to tell the program that it should wait for this
process to complete before it continues with the rest of the code.

from multiprocessing import Process

import os

def square_numbers():
for i in range(1000):
result = i * i

if __name__ == "__main__":
processes = []
num_processes = os.cpu_count()
# number of CPUs on the machine. Usually a good choise
for the number of processes

# create processes and asign a function for each

process
for i in range(num_processes):
process = Process(target=square_numbers)
processes.append(process)

# start all processes

for process in processes:

94
 process.start()

# wait for all processes to finish

# block the main programm until these processes are
finished
for process in processes:
process.join()
Share data between processes¶
Since processes don't live in the same memory space, they do not
have access to the same (public) data. Thus, they need special
shared memory objects to share data.

Data can be stored in a shared memory variable using Value or

Array.

• Value(type, value): Create a ctypes object of type type.

Access the value with .target.
• Array(type, value): Create a ctypes array with elements of
type type. Access the values with [].
Task: Create two processes, each process should have access to a
shared variable and modify it (in this case only increase it
repeatedly by 1 for 100 times). Create another two processes that
share an array and modify (increase) all the elements in the array.

from multiprocessing import Process, Value, Array

import time

def add_100(number):
for _ in range(100):
time.sleep(0.01)
number.value += 1

def add_100_array(numbers):
for _ in range(100):
time.sleep(0.01)
for i in range(len(numbers)):
numbers[i] += 1

if __name__ == "__main__":

95
 shared_number = Value('i', 0)
print('Value at beginning:', shared_number.value)

shared_array = Array('d', [0.0, 100.0, 200.0])

print('Array at beginning:', shared_array[:])

process1 = Process(target=add_100,
args=(shared_number,))
process2 = Process(target=add_100,
args=(shared_number,))

process3 = Process(target=add_100_array,
args=(shared_array,))
process4 = Process(target=add_100_array,
args=(shared_array,))

process1.start()
process2.start()
process3.start()
process4.start()

process1.join()
process2.join()
process3.join()
process4.join()

print('Value at end:', shared_number.value)

print('Array at end:', shared_array[:])

print('end main')
Value at beginning: 0
Array at beginning: [0.0, 100.0, 200.0]
Value at end: 144
Array at end: [134.0, 237.0, 339.0]
end main
How to use Locks¶
Notice that in the above example, the 2 processes should
increment the shared value by 1 for 100 times. This results in 200
total operations. But why is the end value not 200?

Race condition¶

96
A race condition happened here. A race condition occurs when two
or more processes or threads can access shared data and they try
to change it at the same time. In our example the two processes
have to read the shared value, increase it by 1, and write it back
into the shared variable. If this happens at the same time, the two
processes read the same value, increase it and write it back. Thus,
both processes write the same increased value back into the
shared object, and the value was not increased by 2. See https://
www.python-engineer.com/learn/advancedpython16_threading/ for
a detailed explanation of race conditions.

Avoid race conditions with Locks¶

A lock (also known as mutex) is a synchronization mechanism for
enforcing limits on access to a resource in an environment where
there are many processes/threads of execution. A Lock has two
states: locked and unlocked. If the state is locked, it does not
allow other concurrent processes/threads to enter this code
section until the state is unlocked again.

Two functions are important: - lock.acquire() : This will lock the

state and block - lock.release() : This will unlock the state again.

Important: You should always release the block again after it was
acquired!

In our example the critical code section where the shared variable
is read and increased is now locked. This prevents the second
process from mody ng the shared object at the same time. Not
much has changed in our code. All new changes are commented in
the code below.

# import Lock
from multiprocessing import Lock
from multiprocessing import Process, Value, Array
import time

def add_100(number, lock):

for _ in range(100):
time.sleep(0.01)
# lock the state
97
fi
 lock.acquire()

number.value += 1

# unlock the state

lock.release()

def add_100_array(numbers, lock):

for _ in range(100):
time.sleep(0.01)
for i in range(len(numbers)):
lock.acquire()
numbers[i] += 1
lock.release()

if __name__ == "__main__":

# create a lock
lock = Lock()

shared_number = Value('i', 0)
print('Value at beginning:', shared_number.value)

shared_array = Array('d', [0.0, 100.0, 200.0])

print('Array at beginning:', shared_array[:])

# pass the lock to the target function

process1 = Process(target=add_100,
args=(shared_number, lock))
process2 = Process(target=add_100,
args=(shared_number, lock))

process3 = Process(target=add_100_array,
args=(shared_array, lock))
process4 = Process(target=add_100_array,
args=(shared_array, lock))

process1.start()
process2.start()
process3.start()
process4.start()

process1.join()
process2.join()
98
 process3.join()
process4.join()

print('Value at end:', shared_number.value)

print('Array at end:', shared_array[:])

print('end main')
Value at beginning: 0
Array at beginning: [0.0, 100.0, 200.0]
Value at end: 200
Array at end: [200.0, 300.0, 400.0]
end main
Use the lock as a context manager¶
After lock.acquire() you should never forget to call
lock.release() to unblock the code. You can also use a lock as a
context manager, wich will safely lock and unlock your code. It is
recommended to use a lock this way:

def add_100(number, lock):

for _ in range(100):
time.sleep(0.01)
with lock:
number.value += 1
Using Queues in Python¶
Data can also be shared between processes with a Queue. Queues
can be used for thread-safe/process-safe data exchanges and
data processing both in a multithreaded and a multiprocessing
environment, which means you can avoid having to use any
synchronization primitives like locks.

The queue¶
A queue is a linear data structure that follows the First In First Out
(FIFO) principle. A good example is a queue of customers that are
waiting in line, where the customer that came rst is served rst.

from multiprocessing import Queue

# create queue

99
fi
fi
q = Queue()

# add elements
q.put(1) # 1
q.put(2) # 2 1
q.put(3) # 3 2 1

# now q looks like this:

# back --> 3 2 1 --> front

# get and remove first element

first = q.get() # --> 1
print(first)

# q looks like this:

# back --> 3 2 --> front
1
Using a queue in multiprocessing¶
Operations with a queue are process-safe. The multiprocessing
Queue implements all the methods of queue.Queue except for
task_done() and join(). Important methods are:

• q.get() : Remove and return the rst item. By default, it

blocks until the item is available.
• q.put(item) : Puts element at the end of the queue. By
default, it blocks until a free slot is available.
• q.empty() : Return True if the queue is empty.
• q.close() : Indicate that no more data will be put on this
queue by the current process.
# communicate between processes with the multiprocessing
Queue
# Queues are thread and process safe
from multiprocessing import Process, Queue

def square(numbers, queue):

for i in numbers:
queue.put(i*i)

def make_negative(numbers, queue):

for i in numbers:
queue.put(i*-1)
100
fi
 if __name__ == "__main__":

numbers = range(1, 6)
q = Queue()

p1 = Process(target=square, args=(numbers,q))
p2 = Process(target=make_negative, args=(numbers,q))

p1.start()
p2.start()

p1.join()
p2.join()

# order might not be sequential

while not q.empty():
print(q.get())

print('end main')
1
4
9
16
25
-1
-2
-3
-4
-5
end main
Process Pools¶
A process pool object controls a pool of worker processes to
which jobs can be submitted It supports asynchronous results with
timeouts and callbacks and has a parallel map implementation. It
can automatically manage the available processors and split data
into smaller chunks which can then be processed in parallel by
di erent processes. See https://docs.python.org/3.7/library/
multiprocessing.html#multiprocessing.pool for all possible
methods. Important methods are:

101
ff
 • map(func, iterable[, chunksize]) : This method chops the
iterable into a number of chunks which it submits to the
process pool as separate tasks. The (approximate) size of
these chunks can be speci ed by setting chunksize to a
positive integer. It blocks until the result is ready.
• close() : Prevents any more tasks from being submitted to
the pool. Once all the tasks have been completed the worker
processes will exit.
• join(): Wait for the worker processes to exit. One must call
close() or terminate() before using join().
• apply(func, args): Call func with arguments args. It blocks
until the result is ready. func is only executed in ONE of the
workers of the pool.
Note: There are also asynchronous variants map_async() and
apply_async() that will not block. They can execute callbacks
when the results are ready.

from multiprocessing import Pool

def cube(number):
return number * number * number

if __name__ == "__main__":
numbers = range(10)

p = Pool()

# by default this allocates the maximum number of

available
# processors for this task --> os.cpu_count()
result = p.map(cube, numbers)

# or
# result = [p.apply(cube, args=(i,)) for i in numbers]

p.close()
p.join()

print(result)
[0, 1, 8, 27, 64, 125, 216, 343, 512, 729]

102
fi
 Function arguments -
Advanced Python 18
In this article we will talk about function parameters and
function arguments in detail.

We will learn:

• The di erence between arguments and parameters

• Positional and keyword arguments
• Default arguments
• Variable-length arguments (*args and **kwargs)
• Container unpacking into function arguments
• Local vs. global arguments
• Parameter passing (by value or by reference?)
Arguments and parameters¶
• Parameters are the variables that are de ned or used inside
parentheses while de ning a function
• Arguments are the value passed for these parameters while
calling a function
def print_name(name): # name is the parameter
print(name)

print_name('Alex') # 'Alex' is the argument

Alex
Positional and keyword arguments¶
We can pass arguments as positional or keyword arguments. Some
bene ts of keyword arguments can be: - We can call arguments by
their names to make it more clear what they represent - We can
rearrange arguments in a way that makes them most readable

def foo(a, b, c):

103
fi
ff
fi
fi
 print(a, b, c)

# positional arguments
foo(1, 2, 3)

# keyword arguments
foo(a=1, b=2, c=3)
foo(c=3, b=2, a=1) # Note that the order is not important
here

# mix of both
foo(1, b=2, c=3)

# This is not allowed:

# foo(1, b=2, 3) # positional argument after keyword
argument
# foo(1, b=2, a=3) # multiple values for argument 'a'
1 2 3
1 2 3
1 2 3
1 2 3
Default arguments¶
Functions can have default arguments with a prede ned value. This
argument can be left out and the default value is then passed to
the function, or the argument can be used with a di erent value.
Note that default arguments must be de ned as the last
parameters in a function.

# default arguments
def foo(a, b, c, d=4):
print(a, b, c, d)

foo(1, 2, 3, 4)
foo(1, b=2, c=3, d=100)

# not allowed: default arguments must be at the end

# def foo(a, b=2, c, d=4):
# print(a, b, c, d)
1 2 3 4
1 2 3 100

104
fi
ff
fi
Variable-length arguments (*args and
**kwargs)¶
• If you mark a parameter with one asterisk (*), you can pass
any number of positional arguments to your function (Typically
called *args)
• If you mark a parameter with two asterisks (**), you can pass
any number of keyword arguments to this function (Typically
called **kwargs).
def foo(a, b, *args, **kwargs):
print(a, b)
for arg in args:
print(arg)
for kwarg in kwargs:
print(kwarg, kwargs[kwarg])

# 3, 4, 5 are combined into args

# six and seven are combined into kwargs
foo(1, 2, 3, 4, 5, six=6, seven=7)
print()

# omitting of args or kwargs is also possible

foo(1, 2, three=3)
1 2
3
4
5
six 6
seven 7

1 2
three 3
Forced keyword arguments¶
Sometimes you want to have keyword-only arguments. You can
enforce that with: - If you write '*,' in your function parameter list,
all parameters after that must be passed as keyword arguments. -
Arguments after variable-length arguments must be keyword
arguments.

105
def foo(a, b, *, c, d):
print(a, b, c, d)

foo(1, 2, c=3, d=4)

# not allowed:
# foo(1, 2, 3, 4)

# arguments after variable-length arguments must be

keyword arguments
def foo(*args, last):
for arg in args:
print(arg)
print(last)

foo(8, 9, 10, last=50)

1 2 3 4
8
9
10
50
Unpacking into agruments¶
• Lists or tuples can be unpacked into arguments with one
asterisk (*) if the length of the container matches the number
of function parameters.
• Dictionaries can be unpacked into arguments with two
asterisks (**) the the length and the keys match the function
parameters.
def foo(a, b, c):
print(a, b, c)

# list/tuple unpacking, length must match

my_list = [4, 5, 6] # or tuple
foo(*my_list)

# dict unpacking, keys and length must match

my_dict = {'a': 1, 'b': 2, 'c': 3}
foo(**my_dict)

# my_dict = {'a': 1, 'b': 2, 'd': 3} # not possible since

wrong keyword
4 5 6
106
1 2 3
Local vs global variables¶
Global variables can be accessed within a function body, but to
modify them, we rst must state global var_name in order to
change the global variable.

def foo1():
x = number # global variable can only be accessed here
print('number in function:', x)

number = 0
foo1()

# modifying the global variable

def foo2():
global number # global variable can now be accessed
and modified
number = 3

print('number before foo2(): ', number)

foo2() # modifies the global variable
print('number after foo2(): ', number)
number in function: 0
number before foo2(): 0
number after foo2(): 3
If we do not write global var_name and assign a new value to a
variable with the same name as the global variable, this will create
a local variable within the function. The global variable remains
unchanged.

number = 0

def foo3():
number = 3 # this is a local variable

print('number before foo3(): ', number)

foo3() # does not modify the global variable
print('number after foo3(): ', number)
number before foo3(): 0
number after foo3(): 0
Parameter passing¶
107
fi
Python uses a mechanism, which is known as "Call-by-Object" or
"Call-by-Object-Reference. The following rules must be
considered: - The parameter passed in is actually a reference to an
object (but the reference is passed by value) - Di erence between
mutable and immutable data types

This means that:

1. Mutable objects (e.g. lists,dict) can be changed within a

method.
2. But if you rebind the reference in the method, the outer
reference will still point at the original object.
3. Immutable objects (e.g. int, string) cannot be changed within a
method.
4. But immutable object CONTAINED WITHIN a mutable object
can be re-assigned within a method.
# immutable objects -> no change
def foo(x):
x = 5 # x += 5 also no effect since x is immutable and
a new variable must be created

var = 10
print('var before foo():', var)
foo(var)
print('var after foo():', var)
var before foo(): 10
var after foo(): 10
# mutable objects -> change
def foo(a_list):
a_list.append(4)

my_list = [1, 2, 3]
print('my_list before foo():', my_list)
foo(my_list)
print('my_list after foo():', my_list)
my_list before foo(): [1, 2, 3]
my_list after foo(): [1, 2, 3, 4]
# immutable objects within a mutable object -> change
def foo(a_list):
a_list[0] = -100
a_list[2] = "Paul"

108
ff
my_list = [1, 2, "Max"]
print('my_list before foo():', my_list)
foo(my_list)
print('my_list after foo():', my_list)
my_list before foo(): [1, 2, 'Max']
my_list after foo(): [-100, 2, 'Paul']
# Rebind a mutable reference -> no change
def foo(a_list):
a_list = [50, 60, 70] # a_list is now a new local
variable within the function
a_list.append(50)

my_list = [1, 2, 3]
print('my_list before foo():', my_list)
foo(my_list)
print('my_list after foo():', my_list)
my_list before foo(): [1, 2, 3]
my_list after foo(): [1, 2, 3]
Be careful with += and = operations for mutable types. The rst
operation has an e ect on the passed argument while the latter has
not:

# another example with rebinding references:

def foo(a_list):
a_list += [4, 5] # this chanches the outer variable

def bar(a_list):
a_list = a_list + [4, 5] # this rebinds the reference
to a new local variable

my_list = [1, 2, 3]
print('my_list before foo():', my_list)
foo(my_list)
print('my_list after foo():', my_list)

my_list = [1, 2, 3]
print('my_list before bar():', my_list)
bar(my_list)
print('my_list after bar():', my_list)
my_list before foo(): [1, 2, 3]
my_list after foo(): [1, 2, 3, 4, 5]
my_list before bar(): [1, 2, 3]
my_list after bar(): [1, 2, 3]

109
ff
fi
The Asterisk (*) operator -
Advanced Python 19
This tutorial covers the asterisk sign (`*`) and its di erent use
cases in Python.

The asterisk sign (*) can be used for di erent cases in Python: -
Multiplication and power operations - Creation of list, tuple, or
string with repeated elements - *args , **kwargs , and keyword-
only parameters - Unpacking lists/tuples/dictionaries for function
arguments - Unpacking containers - Merging containers into list /
Merge dictionaries

Multiplication and power operations¶

# multiplication
result = 7 * 5
print(result)

# power operation
result = 2 ** 4
print(result)
35
16
Creation of list, tuple, or string with
repeated elements¶
# list
zeros = [0] * 10
onetwos = [1, 2] * 5
print(zeros)
print(onetwos)

# tuple
zeros = (0,) * 10
onetwos = (1, 2) * 5
print(zeros)
print(onetwos)
110
ff
ff
# string
A_string = "A" * 10
AB_string = "AB" * 5
print(A_string)
print(AB_string)
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
[1, 2, 1, 2, 1, 2, 1, 2, 1, 2]
(0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
(1, 2, 1, 2, 1, 2, 1, 2, 1, 2)
AAAAAAAAAA
ABABABABAB
*args , **kwargs , and keyword-only
arguments¶
• Use *args for variable-length arguments
• Use **kwargs for variable-length keyword arguments
• Use *, followed by more function parameters to enforce
keyword-only arguments
def my_function(*args, **kwargs):
for arg in args:
print(arg)
for key in kwargs:
print(key, kwargs[key])

my_function("Hey", 3, [0, 1, 2], name="Alex", age=8)

# Parameters after '*' or '*identifier' are keyword-only

parameters and may only be passed using keyword arguments.
def my_function2(name, *, age):
print(name)
print(age)

# my_function2("Michael", 5) --> this would raise a

TypeError
my_function2("Michael", age=5)
Hey
3
[0, 1, 2]
name Alex
age 8
Michael
111
5
Unpacking for function arguments¶
• Lists/tuples/sets/strings can be unpacked into function
arguments with one * if the length matches the parameters.
• Dictionaries can be unpacked with two ** if the length and the
keys match the parameters.
def foo(a, b, c):
print(a, b, c)

# length must match

my_list = [1, 2, 3]
foo(*my_list)

my_string = "ABC"
foo(*my_string)

# length and keys must match

my_dict = {'a': 4, 'b': 5, 'c': 6}
foo(**my_dict)
1 2 3
A B C
4 5 6
Unpacking containers¶
Unpack the elements of a list, tuple, or set into single and multiple
remaining elements. Note that multiple elements are combined in a
list, even if the unpacked container is a tuple or a set.

numbers = (1, 2, 3, 4, 5, 6, 7, 8)

*beginning, last = numbers

print(beginning)
print(last)

print()

first, *end = numbers

print(first)
print(end)

print()
112
first, *middle, last = numbers
print(first)
print(middle)
print(last)
[1, 2, 3, 4, 5, 6, 7]
8

1
[2, 3, 4, 5, 6, 7, 8]

1
[2, 3, 4, 5, 6, 7]
8
Merge iterables into a list / Merge
dictionaries¶
This is possible since Python 3.5 thanks to PEP 448 (https://
www.python.org/dev/peps/pep-0448/).

# dump iterables into a list and merge them

my_tuple = (1, 2, 3)
my_set = {4, 5, 6}
my_list = [*my_tuple, *my_set]
print(my_list)

# merge two dictionaries with dict unpacking

dict_a = {'one': 1, 'two': 2}
dict_b = {'three': 3, 'four': 4}
dict_c = {**dict_a, **dict_b}
print(dict_c)
[1, 2, 3, 4, 5, 6]
{'one': 1, 'two': 2, 'three': 3, 'four': 4}
But be careful with the following merging solution. It does not work
if the dictionary has any non-string keys:
https://stackover ow.com/questions/38987/how-to-merge-two-
dictionaries-in-a-single-expression/39858#39858

dict_a = {'one': 1, 'two': 2}

dict_b = {3: 3, 'four': 4}
dict_c = dict(dict_a, **dict_b)
print(dict_c)

113
fl
# this works:
# dict_c = {**dict_a, **dict_b}
----------------------------------------------------------
-----------------
TypeError Traceback (most
recent call last)
<ipython-input-52-2660fb90a60f> in <module>
1 dict_a = {'one': 1, 'two': 2}
2 dict_b = {3: 3, 'four': 4}
----> 3 dict_c = dict(dict_a, **dict_b)
4 print(dict_c)
5
TypeError: keywords must be strings
Recommended further readings: - https://treyhunner.com/2018/10/
asterisks-in-python-what-they-are-and-how-to-use-them/ - https://
treyhunner.com/2016/02/how-to-merge-dictionaries-in-python/

Shallow vs Deep Copying -

Advanced Python 20
Tutorial about shallow vs deep copying in Python and the
`copy` module.

In Python, assignment statements (obj_b = obj_a) do not create

real copies. It only creates a new variable with the same reference.
So when you want to make actual copies of mutable objects (lists,
dicts) and want to modify the copy without a ecting the original,
you have to be careful.
For 'real' copies we can use the copy module. However, for
compound/nested objects (e.g. nested lists or dicts) and custom
objects there is an important di erence between shallow and deep
copying: - shallow copies: Only one level deep. It creates a new
collection object and populates it with references to the nested
objects. This means mody ng a nested object in the copy deeper
than one level a ects the original. - deep copies: A full independent
clone. It creates a new collection object and then recursively
populates it with copies of the nested objects found in the original.

114
ff
fi
ff
ff
Assignment operation¶
This will only create a new variable with the same reference.
Modifying one will a ect the other.

list_a = [1, 2, 3, 4, 5]
list_b = list_a

list_a[0] = -10
print(list_a)
print(list_b)
[-10, 2, 3, 4, 5]
[-10, 2, 3, 4, 5]
Shallow copy¶
One level deep. Modifying on level 1 does not a ect the other list.
Use copy.copy(), or object-speci c copy functions/copy
constructors.

import copy
list_a = [1, 2, 3, 4, 5]
list_b = copy.copy(list_a)

# not affects the other list

list_b[0] = -10
print(list_a)
print(list_b)
[1, 2, 3, 4, 5]
[-10, 2, 3, 4, 5]
But with nested objects, modifying on level 2 or deeper does a ect
the other!

import copy
list_a = [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
list_b = copy.copy(list_a)

# affects the other!

list_a[0][0]= -10
print(list_a)
print(list_b)
[[-10, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
[[-10, 2, 3, 4, 5], [6, 7, 8, 9, 10]]

115
ff
fi
ff
ff
Note: You can also use the following to create shallow copies:

# shallow copies
list_b = list(list_a)
list_b = list_a[:]
list_b = list_a.copy()
Deep copies¶
Full independent clones. Use copy.deepcopy().

import copy
list_a = [[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
list_b = copy.deepcopy(list_a)

# not affects the other

list_a[0][0]= -10
print(list_a)
print(list_b)
[[-10, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
[[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]
Custom objects¶
You can use the copy module to get shallow or deep copies of
custom objects.

class Person:
def __init__(self, name, age):
self.name = name
self.age = age

# Only copies the reference

p1 = Person('Alex', 27)
p2 = p1
p2.age = 28
print(p1.age)
print(p2.age)
28
28
# shallow copy
import copy
p1 = Person('Alex', 27)
p2 = copy.copy(p1)
p2.age = 28
116
print(p1.age)
print(p2.age)
27
28
Now let's make a nested object:

class Company:
def __init__(self, boss, employee):
self. boss = boss
self.employee = employee

# shallow copy will affect nested objects

boss = Person('Jane', 55)
employee = Person('Joe', 28)
company = Company(boss, employee)

company_clone = copy.copy(company)
company_clone.boss.age = 56
print(company.boss.age)
print(company_clone.boss.age)

print()
# deep copy will not affect nested objects
boss = Person('Jane', 55)
employee = Person('Joe', 28)
company = Company(boss, employee)
company_clone = copy.deepcopy(company)
company_clone.boss.age = 56
print(company.boss.age)
print(company_clone.boss.age)
56
56

55
56

117
Context Managers -
Advanced Python 21
Context managers are a great tool for resource management.
They allow you to allocate and release resources precisely
when you want to.

A well-known example is the with open() statemtent:

with open('notes.txt', 'w') as f:

f.write('some todo...')
This will open a le and makes sure to automatically close it after
program execution leaves the context of the with statement. It also
handles exceptions and makes sure to properly close the le even
in case of an exception. Internally, the above code translates to
something like this:

f = open('notes.txt', 'w')
try:
f.write('some todo...')
finally:
f.close()
We can see that using a context manager and the with statement
is much shorter and more concise.

Examples of context managers¶

• Open and close les
• open and close database connections
• Acquire and release locks:
from threading import Lock
lock = Lock()

# error-prone:
lock.acquire()
# do stuff
# lock should always be released!
lock.release()
118
fi
fi
fi
# Better:
with lock:
# do stuff
Implementing a context manager as a
class¶
To support the with statement for our own classes, we have to
implement the __enter__ and __exit__ methods. Python calls
__enter__ when execution enters the context of the with
statement. In here the resource should be acquired and returned.
When execution leaves the context again, __exit__ is called and
the resource is freed up.

class ManagedFile:
def __init__(self, filename):
print('init', filename)
self.filename = filename

def __enter__(self):
print('enter')
self.file = open(self.filename, 'w')
return self.file

def __exit__(self, exc_type, exc_value,

exc_traceback):
if self.file:
self.file.close()
print('exit')

with ManagedFile('notes.txt') as f:
print('doing stuff...')
f.write('some todo...')
init notes.txt
enter
doing stuff...
exit
Handling exceptions¶
If an exception occurs, Python passes the type, value, and
traceback to the __exit__ method. It can handle the exception

119
here. If anything other than True is returned by the __exit__
method, then the exception is raised by the with statement.

class ManagedFile:
def __init__(self, filename):
print('init', filename)
self.filename = filename

def __enter__(self):
print('enter')
self.file = open(self.filename, 'w')
return self.file

def __exit__(self, exc_type, exc_value,

exc_traceback):
if self.file:
self.file.close()
print('exc:', exc_type, exc_value)
print('exit')

# No exception
with ManagedFile('notes.txt') as f:
print('doing stuff...')
f.write('some todo...')
print('continuing...')

print()

# Exception is raised, but the file can still be closed

with ManagedFile('notes2.txt') as f:
print('doing stuff...')
f.write('some todo...')
f.do_something()
print('continuing...')
init notes.txt
enter
doing stuff...
exc: None None
exit
continuing...

init notes2.txt
enter
doing stuff...

120
exc: <class 'AttributeError'> '_io.TextIOWrapper' object
has no attribute 'do_something'
exit

----------------------------------------------------------
-----------------
AttributeError Traceback (most
recent call last)
<ipython-input-24-ed1604efb530> in <module>
27 print('doing stuff...')
28 f.write('some todo...')
---> 29 f.do_something()
30 print('continuing...')
AttributeError: '_io.TextIOWrapper' object has no
attribute 'do_something'
We can handle the exception in the __exit__ method and return
True.

class ManagedFile:
def __init__(self, filename):
print('init', filename)
self.filename = filename

def __enter__(self):
print('enter')
self.file = open(self.filename, 'w')
return self.file

def __exit__(self, exc_type, exc_value,

exc_traceback):
if self.file:
self.file.close()
if exc_type is not None:
print('Exception has been handled')
print('exit')
return True

with ManagedFile('notes2.txt') as f:
print('doing stuff...')
f.write('some todo...')
f.do_something()
print('continuing...')
init notes2.txt
121
enter
doing stuff...
Exception has been handled
exit
continuing...
Implementing a context manager as a
generator¶
Instead of writing a class, we can also write a generator function
and decorate it with the contextlib.contextmanager decorator.
Then we can also call the function using a with statement. For this
approach, the function must yield the resource in a try statement,
and all the content of the __exit__ method to free up the resource
goes now inside the corresponding finally statement.

from contextlib import contextmanager

@contextmanager
def open_managed_file(filename):
f = open(filename, 'w')
try:
yield f
finally:
f.close()

with open_managed_file('notes.txt') as f:
f.write('some todo...')
The generator rst acquires the resource. It then temporarily
suspends its own execution and yields the resource so it can be
used by the caller. When the caller leaves the with context, the
generator continues to execute and frees up the resource in the
finally statement.

122
fi