Practical 8
Aim: Write a program to implement basic algebraic operation on tensors like addition, subtraction and
rank of Tensors.
Theory:
n deep learning it is common to see a lot of discussion around tensors as the cornerstone
data structure.
Tensor even appears in name of Google’s flagship machine learning library: “TensorFlow“.
Tensors are a type of data structure used in linear algebra, and like vectors and matrices,
you can calculate arithmetic operations with tensors.
you will discover what tensors are and how to manipulate them in Python with NumPy
What are Tensors?
A tensor is a generalization of vectors and matrices and is easily understood as a
multidimensional array.
In the general case, an array of numbers arranged on a regular grid with a variable number
of axes is known as a tensor.
A vector is a one-dimensional or first order tensor and a matrix is a two-dimensional or
second order tensor.
Tensor notation is much like matrix notation with a capital letter representing a tensor and
lowercase letters with subscript integers representing scalar values within the tensor.
1 t111, t121, t131 t112, t122, t132 t113, t123, t133
2T = (t211, t221, t231), (t212, t222, t232), (t213, t223, t233)
3 t311, t321, t331 t312, t322, t332 t313, t323, t333
Many of the operations that can be performed with scalars, vectors, and matrices can be
reformulated to be performed with tensors.
As a tool, tensors and tensor algebra is widely used in the fields of physics and engineering.
It is a term and set of techniques known in machine learning in the training and operation of
deep learning models can be described in terms of tensors.
Tensors in Python
Like vectors and matrices, tensors can be represented in Python using the N-dimensional
array (ndarray).
A tensor can be defined in-line to the constructor of array() as a list of lists.
The example below defines a 3x3x3 tensor as a NumPy ndarray. Three dimensions is
easier to wrap your head around. Here, we first define rows, then a list of rows stacked as
columns, then a list of columns stacked as levels in a cube.
1# create tensor
2from numpy import array
3T = array([
4 [[1,2,3], [4,5,6], [7,8,9]],
5 [[11,12,13], [14,15,16], [17,18,19]],
6 [[21,22,23], [24,25,26], [27,28,29]],
7 ])
8print([Link])
9print(T)
Running the example first prints the shape of the tensor, then the values of the tensor itself.
You can see that, at least in three-dimensions, the tensor is printed as a series of matrices,
one for each layer. For this 3D tensor, axis 0 specifies the level, axis 1 specifies the row,
and axis 2 specifies the column.
1 (3, 3, 3)
2 [[[ 1 2 3]
3 [ 4 5 6]
4 [ 7 8 9]]
6 [[11 12 13]
7 [14 15 16]
8 [17 18 19]]
10 [[21 22 23]
11 [24 25 26]
12 [27 28 29]]]
Element-Wise Tensor Operations
As with matrices, we can perform element-wise arithmetic between tensors.
In this section, we will work through the four main arithmetic operations.
Tensor Addition
The element-wise addition of two tensors with the same dimensions results in a new tensor
with the same dimensions where each scalar value is the element-wise addition of the
scalars in the parent tensors.
1 a111, a121, a131 a112, a122, a132
2 A = (a211, a221, a231), (a112, a122, a132)
4 b111, b121, b131 b112, b122, b132
5 B = (b211, b221, b231), (b112, b122, b132)
8 C=A+B
10 a111 + b111, a121 + b121, a131 + b131 a112 + b112, a122 + b122, a132 + b132
11C = (a211 + b211, a221 + b221, a231 + b231), (a112 + b112, a122 + b122, a132 + b132)
In NumPy, we can add tensors directly by adding arrays.
1 # tensor addition
2 from numpy import array
3 A = array([
4 [[1,2,3], [4,5,6], [7,8,9]],
5 [[11,12,13], [14,15,16], [17,18,19]],
6 [[21,22,23], [24,25,26], [27,28,29]],
7 ])
8 B = array([
9 [[1,2,3], [4,5,6], [7,8,9]],
10 [[11,12,13], [14,15,16], [17,18,19]],
11 [[21,22,23], [24,25,26], [27,28,29]],
12 ])
13C = A + B
14print(C)
Running the example prints the addition of the two parent tensors.
1 [[[ 2 4 6]
2 [ 8 10 12]
3 [14 16 18]]
5 [[22 24 26]
6 [28 30 32]
7 [34 36 38]]
9 [[42 44 46]
10 [48 50 52]
11 [54 56 58]]]
Tensor Subtraction
The element-wise subtraction of one tensor from another tensor with the same dimensions
results in a new tensor with the same dimensions where each scalar value is the element-
wise subtraction of the scalars in the parent tensors.
1 a111, a121, a131 a112, a122, a132
2 A = (a211, a221, a231), (a112, a122, a132)
4 b111, b121, b131 b112, b122, b132
5 B = (b211, b221, b231), (b112, b122, b132)
7 C=A-B
9 a111 - b111, a121 - b121, a131 - b131 a112 - b112, a122 - b122, a132 - b132
10C = (a211 - b211, a221 - b221, a231 - b231), (a112 - b112, a122 - b122, a132 - b132)
In NumPy, we can subtract tensors directly by subtracting arrays.
1 # tensor subtraction
2 from numpy import array
3 A = array([
4 [[1,2,3], [4,5,6], [7,8,9]],
5 [[11,12,13], [14,15,16], [17,18,19]],
6 [[21,22,23], [24,25,26], [27,28,29]],
7 ])
8 B = array([
9 [[1,2,3], [4,5,6], [7,8,9]],
10 [[11,12,13], [14,15,16], [17,18,19]],
11 [[21,22,23], [24,25,26], [27,28,29]],
12 ])
13C = A - B
14print(C)
Running the example prints the result of subtracting the first tensor from the second.
1 [[[0 0 0]
2 [0 0 0]
3 [0 0 0]]
5 [[0 0 0]
6 [0 0 0]
7 [0 0 0]]
9 [[0 0 0]
10 [0 0 0]
11 [0 0 0]]]
Tensor Hadamard Product
The element-wise multiplication of one tensor from another tensor with the same
dimensions results in a new tensor with the same dimensions where each scalar value is
the element-wise multiplication of the scalars in the parent tensors.
As with matrices, the operation is referred to as the Hadamard Product to differentiate it
from tensor multiplication. Here, we will use the “o” operator to indicate the Hadamard
product operation between tensors.
1 a111, a121, a131 a112, a122, a132
2 A = (a211, a221, a231), (a112, a122, a132)
4 b111, b121, b131 b112, b122, b132
5 B = (b211, b221, b231), (b112, b122, b132)
7 C=AoB
9 a111 * b111, a121 * b121, a131 * b131 a112 * b112, a122 * b122, a132 * b132
10C = (a211 * b211, a221 * b221, a231 * b231), (a112 * b112, a122 * b122, a132 * b132)
In NumPy, we can multiply tensors directly by multiplying arrays.
1 # tensor Hadamard product
2 from numpy import array
3 A = array([
4 [[1,2,3], [4,5,6], [7,8,9]],
5 [[11,12,13], [14,15,16], [17,18,19]],
6 [[21,22,23], [24,25,26], [27,28,29]],
7 ])
8 B = array([
9 [[1,2,3], [4,5,6], [7,8,9]],
10 [[11,12,13], [14,15,16], [17,18,19]],
11 [[21,22,23], [24,25,26], [27,28,29]],
12 ])
13C = A * B
14print(C)
Running the example prints the result of multiplying the tensors.
1 [[[ 1 4 9]
2 [ 16 25 36]
3 [ 49 64 81]]
5 [[121 144 169]
6 [196 225 256]
7 [289 324 361]]
9 [[441 484 529]
10 [576 625 676]
11 [729 784 841]]]
Tensor Division
The element-wise division of one tensor from another tensor with the same dimensions
results in a new tensor with the same dimensions where each scalar value is the element-
wise division of the scalars in the parent tensors.
1 a111, a121, a131 a112, a122, a132
2 A = (a211, a221, a231), (a112, a122, a132)
4 b111, b121, b131 b112, b122, b132
5 B = (b211, b221, b231), (b112, b122, b132)
7 C=A/B
9 a111 / b111, a121 / b121, a131 / b131 a112 / b112, a122 / b122, a132 / b132
10C = (a211 / b211, a221 / b221, a231 / b231), (a112 / b112, a122 / b122, a132 / b132)
In NumPy, we can divide tensors directly by dividing arrays.
1 # tensor division
2 from numpy import array
3 A = array([
4 [[1,2,3], [4,5,6], [7,8,9]],
5 [[11,12,13], [14,15,16], [17,18,19]],
6 [[21,22,23], [24,25,26], [27,28,29]],
7 ])
8 B = array([
9 [[1,2,3], [4,5,6], [7,8,9]],
10 [[11,12,13], [14,15,16], [17,18,19]],
11 [[21,22,23], [24,25,26], [27,28,29]],
12 ])
13C = A / B
14print(C)
Running the example prints the result of dividing the tensors.
1 [[[ 1. 1. 1.]
2 [ 1. 1. 1.]
3 [ 1. 1. 1.]]
5 [[ 1. 1. 1.]
6 [ 1. 1. 1.]
7 [ 1. 1. 1.]]
9 [[ 1. 1. 1.]
10 [ 1. 1. 1.]
11 [ 1. 1. 1.]]]
Tensor Product
The tensor product operator is often denoted as a circle with a small x in the middle. We will
denote it here as “(x)”.
Given a tensor A with q dimensions and tensor B with r dimensions, the product of these
tensors will be a new tensor with the order of q + r or, said another way, q + r dimensions.
The tensor product is not limited to tensors, but can also be performed on matrices and
vectors, which can be a good place to practice in order to develop the intuition for higher
dimensions.
Let’s take a look at the tensor product for vectors.
1a = (a1, a2)
3b = (b1, b2)
4
5c = a (x) b
7 a1 * [b1, b2]
8c = (a2 * [b1, b2])
Or, unrolled:
1 a1 * b1, a1 * b2
2c = (a2 * b1, a2 * b2)
Let’s take a look at the tensor product for matrices.
1 a11, a12
2 A = (a21, a22)
4 b11, b12
5 B = (b21, b22)
7 C = A (x) B
9 b11, b12 b11, b12
10 a11 * (b21, b22), a12 * (b21, b22)
11C = [ b11, b12 b11, b12 ]
12 a21 * (b21, b22), a22 * (b21, b22)
Or, unrolled:
1 a11 * b11, a11 * b12, a12 * b11, a12 * b12
2 a11 * b21, a11 * b22, a12 * b21, a12 * b22
3C = (a21 * b11, a21 * b12, a22 * b11, a22 * b12)
4 a21 * b21, a21 * b22, a22 * b21, a22 * b22
The tensor product can be implemented in NumPy using the tensordot() function.
The function takes as arguments the two tensors to be multiplied and the axis on which to
sum the products over, called the sum reduction. To calculate the tensor product, also
called the tensor dot product in NumPy, the axis must be set to 0.
In the example below, we define two order-1 tensors (vectors) with and calculate the tensor
product.
1# tensor product
2from numpy import array
3from numpy import tensordot
4A = array([1,2])
5B = array([3,4])
6C = tensordot(A, B, axes=0)
7print(C)
Running the example prints the result of the tensor product.
The result is an order-2 tensor (matrix) with the lengths 2×2.
1[[3 4]
2 [6 8]]
The tensor product is the most common form of tensor multiplication that you may
encounter, but there are many other types of tensor multiplications that exist, such as the
tensor dot product and the tensor contraction.
Conclusion:
