Introduction to Mathematics in DS, AI, and ML

▶ Nowadays DS, AI, ML find wide range of applications in

science and engineering.
▶ Various tools from DS, AI, ML provide solutions to real-life
problems.
▶ These algorithms require good knowledge of Mathematics and
Statistics.
▶ This course covers basic Mathematics related to DS, AI, and
ML.
▶ Learn how Mathematics is connected to algorithms in DS, AI,
and ML.
▶ Understand algorithms such as PCA, LDA, and weight
approximation.
▶ Study SVD, Least Square Approximation, Minimum Norm
Solution for deep learning in Machine Learning tools.
Additional Topics Covered

▶ Gradient Calculus
▶ Numerical Substitutes
▶ SVM (Support Vector Machines) including soft and hard
margin classifier
▶ Basic knowledge of Probability and Statistics
Vectors in DS, AI, ML

▶ A vector is a mathematical object that encodes a length and

direction.
▶ Vectors are elements of a vector space.
▶ Vector space contains infinite vectors sharing common
properties.
▶ Vector space is closed under addition and scalar multiplication.
▶ Vectors are represented using dimensional arrays.
▶ Vertical arrays are column vectors; horizontal arrays are row
vectors.
▶ Geometrically, vectors represent coordinates in their respective
dimensions.
Vector Representation

▶ In its simplest form, a vector can be represented as ⃗v .

▶ Consider a vector in n-dimensional space:
V⃗ = [V1 , V2 , . . . , Vn ].
▶ If Vi ∈ R (belongs to the set of real numbers), it’s a real
vector space of dimension n.
▶ A vector ⃗v is represented by an arrow in a vector space,
indicating origin, direction, and magnitude.
▶ If Vi ∈ R, then ⃗v belongs to Rn (the vector space of
dimension n).
Details of Vector Space

▶ A vector can be represented as a one-dimensional array.

▶ It can also be written in column form.
▶ For example, if n = 2, we have the vector space R2 , which
consists of ordered pairs of real numbers.
▶ Let V⃗ = (1, 3, 2).
▶ The first component represents the direction along the x-axis.
▶ The second component represents the direction along the
y-axis.
Vector Properties and Algebra

▶ The length of a vector is its magnitude, denoted as |⃗v |.

▶ The angle θ which the vector ⃗v makes with the x-axis
determines its direction.
▶ In 3-dimensional space (R3 ), vectors have components along
⃗ = (1, 2, 3).
the x, y, and z-axes. For example, V
▶ In Rn , a vector has n components in an n-dimensional space.
▶ n-dimensional vectors are challenging to plot directly.
▶ Vector Algebra:
▶ Addition/Subtraction: Two vectors can be added or subtracted
if they are in the same dimension.
Vector Addition and Subtraction

▶ In R2 :
▶ Let ⃗v1 = (1, 2) and ⃗v2 = (1, 1).
▶ ⃗v1 + ⃗v2 = (1 + 1, 2 + 1) = (2, 3).
▶ ⃗v1 − ⃗v2 = (1 − 1, 2 − 1) = (0, 1).
▶ A similar process works in Rn .
▶ In 3-dimensional space (R3 ):
▶ Let ⃗v1 = (x1 , x2 , . . . , xn ) and ⃗v2 = (y1 , y2 , . . . , yn ).
▶ Vector addition: ⃗v1 + ⃗v2 = (x1 + y1 , x2 + y2 , . . . , xn + yn ).
Dot Product and Vector Magnitude

▶ Dot Product:
▶ Let ⃗v1 = (x1 , x2 , . . . , xn ) ∈ Rn and ⃗v2 = (y1 , y2 , . . . , yn ) ∈ Rn .
▶ Dot product: ⃗v1 · ⃗v2 = x1 y1 + x2 y2 + . . . + xn yn .
▶ Example in R3 : ⃗v1 = (1, 2, 1) and ⃗v2 = (2, 3, 1).
▶ ⃗v1 · ⃗v2 = 1 · 2 + 2 · 3 + 1 · 1 = 2 + 6 + 1 = 9.
▶ Vector Magnitude:
▶ Consider ⃗v = (x1 , x2 , . . . , xn ).
▶ The magnitude
√ of
p⃗v is defined as
∥⃗v ∥ = ⃗v · ⃗v = x12 + x22 + . . . + xn2 .
Vector Magnitude and Angle

▶ Vector Magnitude:
▶ Example:p For ⃗v = (1, −1, 2),√the magnitude√is
∥⃗v ∥ = 12 + (−1)2 + 22 = 1 + 1 + 4 = 6.
▶ If the length of a vector is zero, then it is a zero vector.
▶ For a non-zero vector, the magnitude is greater than zero.
▶ Angle Between Two Vectors:
▶ Let ⃗v1 and ⃗v2 be two non-zero vectors.
▶ The angle θ between ⃗v1 and ⃗v2 is given by:

⃗v1 · ⃗v2
cos θ =
∥⃗v1 ∥∥⃗v2 ∥

where the numerator is the dot product of ⃗v1 and ⃗v2 , and the
denominator is the product of their magnitudes.
Linear Combination of Vectors
▶ Consider a set S = {⃗v1 , ⃗v2 , . . . , ⃗vk } of vectors.
▶ A new vector V ⃗ = α1 ⃗v1 + α2 ⃗v2 + . . . + αk ⃗vk is called a linear
combination of vectors ⃗v1 , ⃗v2 , . . . , ⃗vk , where α1 , α2 , . . . , αk are
scalars.
▶ Scalars are values from the field over which the vector space is
defined. For now, assume they are real numbers.
▶ Example with three vectors: V ⃗1 = (1, 3, 2), V ⃗2 = (1, 3, 1), and
⃗
V3 = (0, 1, 2).
▶ Their linear combination:

⃗ = α1 V
V ⃗1 +α2 V
⃗2 +α3 V
⃗3 = α1 (1, 3, 2)+α2 (1, 3, 1)+α3 (0, 1, 2)

▶ In component form: V ⃗ = (α1 + α2 , 2α1 + α2 , −α1 − α3 ).

▶ Different vectors can be formed as linear combinations of
V⃗1 , V
⃗2 , and V
⃗3 by varying α1 , α2 , or α3 .
Linear Independence and Dependence

⃗1 , V
▶ A set of vectors S = {V ⃗2 , . . . , V
⃗n } is linearly independent if
the vector equation:
⃗1 + α2 V
α1 V ⃗ 2 + . . . + αn V
⃗n = ⃗0

holds only when α1 = α2 = . . . = αn = 0.

▶ In other words, we cannot write any vector V ⃗i , i = 1, 2, . . . , n,
in terms of the other vectors in S.
▶ If the vector equation holds and some or all αi ’s are non-zero,
then the set of vectors is linearly dependent.
▶ Otherwise, the set S is linearly independent.
Linear Independence and Dependence

⃗1 , V
▶ A set of vectors S = {V ⃗2 , . . . , V
⃗n } is linearly independent if
the vector equation:
⃗1 + α2 V
α1 V ⃗ 2 + . . . + αn V
⃗n = ⃗0

holds only when α1 = α2 = . . . = αn = 0.

▶ In other words, we cannot write any vector V ⃗i , i = 1, 2, . . . , n,
as a linear combination of the other vectors in S.
▶ If the vector equation holds and some or all αi ’s are non-zero,
then the set of vectors is linearly dependent.
▶ Otherwise, the set S is linearly independent.
Examples of Linear Independence and Dependence
Example 1:
▶ Let S = {(1, 0), (1, 1)}.
▶ Take the linear combination:

α1 · (1, 0) + α2 · (1, 1) = (0, 0)

▶ This expands to the system of equations:

(
α1 + 0 · α2 = 0
0 · α1 + α2 = 0

▶ Solving the system: (

α1 = 0
α2 = 0
▶ Thus, S contains linearly independent vectors.
Example 2:
▶ Consider S = {(1, 1), (3, 3)}.
▶ Take the linear combination:

−3 · (1, 1) + (3, 3) = (0, 0)

▶ Taking α1 = −3 and α2 = 1, the vector equation holds.

▶ Thus, S contains linearly dependent vectors.
Example of Linear Dependence

Example:
▶ Let S = {(1, −1, 0), (1, 0, 1), (0, 1, 1)} ∈ R3 .
▶ Take α1 = 1, α2 = −1, and α3 = 1.
▶ Compute the linear combination:

α1 (1, −1, 0) + α2 (1, 0, 1) + α3 (0, 1, −1)

= 1(1, −1, 0) + (−1)(1, 0, 1) + 1(0, 1, 1)
= (1, −1, 0) + (−1, 0, −1) + (0, 1, 1)
= (0, 0, 0)

▶ Since α1 , α2 , and α3 are non-zero, this means S is linearly

dependent.
Linear Relationships and Orthogonal Vectors
Linear Relationships Between Vectors
▶ Let V2 = V1 + V3 .
▶ Take V2 = (1, 0, 1) = (1, −1, 0) + (0, 1, 1).
▶ An example in R3 .The standard basis in R3 is
(1, 0, 0), (0, 1, 0), (0, 0, 1).
▶ Similarly we can extend the concept in Rn
Orthogonal Vectors
▶ Definition: In Rn , a set of more than n vectors is linearly
dependent (L-D).
▶ For example, if we take any set containing more than 3 vectors
then the set is linearly dependent (L-D).
▶ A set containing a zero vector is linearly dependent by
definition.
▶ Definition: We say that a set of vectors {v1 , v2 , . . . , vn } are
mutually orthogonal if for all vi .vj = 0 i ̸= j.
Example
√ √
▶ In R3 , let S = {(1, 0, −1), (1, 2, 1), (1, − 2, 1)}.
▶ Check:
√
(1, 0, −1) · (1, 2, 1) = 0
√
(1, 0, −1) · (1, − 2, 1) = 0
√ √
(1, 2, 1) · (1, − 2, 1) = 0

▶ So they are mutually orthogonal.

Orthonormal Vectors
▶ Definition: A set of orthogonal vectors is orthonormal if each
vector has length 1.
Example of Orthonormal vectors
▶ Example: Take a set of vectors in R2 .
n   o
▶ Example set: √1 , √1 , √1 , − √1
2 2 2 2
▶ We can verify that each vector has length 1 and they are
orthogonal also
▶ Calculate the length of each vector:
s 
1 2
   2 r r
1 1 1 1 1 1
∥ , ∥= + = + = =1
2 2 2 2 4 4 2
s 
1 2
   2 r r
1 1 1 1 1 1
∥ ,− ∥= + − = + = =1
2 2 2 2 4 4 2
Therefore, each vector has length 1.
▶ Verify orthogonality:
    
1 1 1 1 1 1
· + · − = − =0
2 2 2 2 4 4
Hence, the vectors are orthogonal.
Remarks on Orthogonal Vectors and Feature Vectors
▶ Remark 1: A set of orthogonal vectors is linearly independent.
But civerse is not true.
▶ Remark 2: Later, we can convert a set of linearly independent
vectors into a set of orthogonal vectors.
▶ An example of feature vectors: How do we represent a vector
in data set.
▶ Consider a simple dataset detailing employees’ heights and
weights:
Employee Height (cm), Weight (kg)
e1 170, 70
e2 165, 65
.. ..
. .
▶ Remark 5: Here, each row represents observations or samples.
Height and weight are attributes (features).
▶ SO if we take any vector , i.e, row corresponding to any
sample. Let’s say (α2 , β2 ) that is (165,65). It is a feature
vector of the 2nd employee e1.
Python Implementation in Google Colab:

▶ import numpy - important package from python. numpy is

used for all array types of operations in python.
Content Overview
▶ Dot Product Functionalities:
▶ Linear Algebra: Multidimensional and more.
▶ Example:
▶ V = 1, −1, 2, W = [2, 5, 2].
▶ v + W:
▶ Gives addition of the vectors.
▶ v − W:
▶ Gives difference of the vectors.
▶ 3 ∗ v:
multiplication.
▶ Length of V
▶ Dot Product (V , W ):
▶ Code Example:
▶ V = [Link]([1,-1,2])
▶ W = [Link]([2, 5, 2])
▶ print( v + w)
▶ print ( v - w)
▶ print(3 * V)
▶ print([Link](V))
▶ s = [Link](V, W)