CSCI 551

Numerical and Parallel

Programming
Vector / Matrix Operations Summary

August 31, 2023  Sam Siewert

 Vectors – 1D array for n D space
In general, an n-tuple of one type in 1 dimensional array
Mathematically, a vector can also represent a location in
N dimensions (point in 3D space)
Homogeneous – e.g., list of floats, integers, or doubles
Can represent a location in space – [X, Y, Z] from origin
A magnitude and direction – from origin pointing to
location (direction) – Euclidian vector
Unit vector is = vector / magnitude(vector)
Can be much larger than homogeneous 3-tuple
Column vector
Row vector

 Sam Siewert 2
 Recall: Matrices (Arrays)
N-tuple in 2 to N dimensions – e.g., 3 x 3 array, 4 x 4 x 4
array that is homogeneous (one type)

2D intensity as a function of (X, Y), e.g., an image

2D Waveform – surface above plane (surface plot)

Array of coefficients for a vector (equation)

Used to rotate a vector or a matrix of scalars (image)

Can simply represent row & column data (spreadsheet)

 Sam Siewert 3
Simple Well-Determined Linear Algebra
For Square Matrices with unknown vectors and a RHS
– Use Cramer’s Rule! – much simpler than general matrix
inversion 2 equations, 2 unknowns
Coefficient matrix for Eq1 & Eq2 Can be re-written as vector/matrix

RHS = Right Hand Side

Unknown vector – find solution for x, y

– Could use substitution or matrix inversion, but Cramer’s rule is

simple and fast for a 2x2 (evaluate equations to check work!)
– x = = 2, y = = 0

Generic Summary of Method

 Sam Siewert 4
Cramer’s Rule for 3x3 Small System
Forget 4x4, …, nxn – use matrix inversion for large scale

Scale to 3x3 2x2 System

3x3 System

Compute 2x2 determinants

of numerator and
denominator to evaluate

Compute 3x3 determinants

of numerator and
denominator to evaluate
 Sam Siewert 5
 Basic Vector, Matrix Operations
Row vector addition: =

Column vector addition: + =

Row vector x Column vector multiplication: x = 0(0) + 1(1) + 2(2) = 5

Column vector x Row vector: x = =

1x3 x 3x1 = 1x1
Matrix addition: [C] = [A] + [B] = + =

These examples are standard vector matrix mathematical definitions

3x1 x 1x3 = 3x3
Consider carefully row major vs. column major programming languages and array
dimensions when programming

 Sam Siewert 6
 Basic Vector, Matrix Operations
Row vector addition: = e.g., add 2 vectors

Column vector addition: + = OR, e.g., + =

Row vector x Column vector multiplication: x = 0(0) + 1(1) + 2(2) = 5

https://academo.org/demos/3d-vector-plotter/

Column vector x Row vector: x = =

1x3 x 3x1 = 1x1
Matrix addition: [C] = [A] + [B] = + =
e.g., inner product produces scalar

These examples are standard vector matrix mathematical definitions

3x1 x 1x3 = 3x3
Consider carefully row major vs. column major programming languages and array
e.g., outer product produces matrix
dimensions when programming

 Sam Siewert 7
 Basic Vector, Matrix Operations
Matrix scaling: [C] = a x [B] = 2 x = e.g., Image
Enhancement
Scalar + Matrix: [C] = a + [B] = 2 + = Brightness
&
Contrast
Row vector x Matrix: x = =

Matrix x Column vector: x = =

Column vector x Matrix = NOT ALLOWED (columns in first != rows in second)

1x3 x 3x3 = 1x3 e.g., Translate
& Rotate
[C] = [A] x [B] = x = = a Vector

3x3 x 3x1 = 3x1 Linear System

of Equations

3x1 x 3x3 = 3x3

3x3 x 3x3 = 3x3

 Sam Siewert 8
 Vector, Matrix Operations
Addition/Subtraction, Scaling, Multiplication of Matrics
Given A & B, Compute C

[ ]
0 1 2
𝐴= 3 4 5
6 7 8

[ ]
0 1 2
B= 3 4 5
6 7 8

[ ]
0+ 0 1+1 2+ 2
3+ 3 4+4 5+5
6+ 6 7 +7 8+ 8

[ ]
0 +3+ 12 0+ 4 +14 0+5+ 16
0 +12+30 3 +16+ 35 6 +20+ 40
0+ 21+48 6 +28+ 56 12+ 35+64

Many [mat] x [vect] & [mat] x scalar problems: rotations, linear system (equations) C=
But what sort of problem requires [mat] x [mat]?
Economics for example – YouTube on Goods and Market Optimization
More problems – Markov chains, Page Rank, etc. (Final Program Ideas)
 Sam Siewert 9
 Review of Matrix Operations
Play with Vector Matrix Operations for Review - https://matrix.reshish.com/

X =

r1xc1 r2xc1 r3xc1

r1xc2 r2xc2 r3xc2

r1xc3 r2xc3 r3xc3

 Sam Siewert 10
 Geometric Vector/Matrix Operations
Dot-product: x

Cross-product: X = =
Dot & Cross
Product
Used in
Robotics

b a
bxa

Tensor product: =

https://academo.org/demos/3d-vector-plotter/ https://en.wikipedia.org/wiki/Cross_product

2x1 x 1x3 = 2x3

 Sam Siewert e.g., machine learning & quantum computing 11
 MATLAB – Vector/Matrix Mathematics
Treats everything like a Vector or Matrix
Interpreted Programming Language – for Math and Vector/Matrix,
Linear Algebra, Simulation, etc.
Now available at CSU if you are interested – CSU Page, Quick Start
MATLAB
Toolboxes
“ver” to list

Computer Vision
GPU Coder
Image Processing
Parallel Computing
DSP System
Statistics and Machine
Learning
…
Symbolic Math

Help Center

https://www.mathworks.com/help/matlab/matlab_prog/matlab-operators-and-special-characters.html
 Sam Siewert 12
 MATLAB Examples
Entering a Entering a
Matrix Row
vector

Entering a Entering a
Column Column
vector vector

Multiplying Multiplying
m1*m2 colv*rowv

m1*v1 rowv*m1
 Sam Siewert 13
Linear Systems – Vector Matrix Algebra
MATLAB code
Kaw and Kalu (Numerical
Methods with Applications)
– Pp. 194-196 – Basic overview
by example (rocket ascent)
– Pp. 224-244 – More Complete
coverage from basic operations
to matrix inversion Solving by simple substitution
– Pp. 204-223 – Series of
examples
Physics
Circuits
Mechanical
Simple 2 Equation 2
Unknown Example
– Generalize to N Equations, N
unknowns
– Matrix inversion more efficient
than simple substitution

 Sam Siewert 14
Example from Kaw & Kalu Ch. 4 – MATLAB
Sol’n for 3 Equations, 3 Unknowns
Enter polynomial equations as
matrix and coefficient vector
Equation LHS
coeff = [poly]-1 [vatt]
At t=7.5 sec, V=165.1036

V at time t
(measured)
Plug times into
Polynomial
Solve for coeff to verify
a, b, c V at time t

 Sam Siewert 15
 MATLAB and Solution
Key step is Matrix inversion of [poly] -1

We will learn matrix manipulation algorithms

(Part 2 after mid-term)
– Gauss Seidel method is iterative (Kaw & Kalu pp. Current Study Focus
291-305)
– LU decomposition (Kaw & Kalue pp. 274-290) – • Matrix manipulation
Matrix form of Gaussian elimination with partial • Matrix transformations
pivoting (Kaw & Kalu pp. 263-266, 267-268) • Vector/Matrix algebra
(basic)
MATLAB is an interpreted interface to these
After Exam #1, Solving
numerical methods
Systems of Equations
– Originally written in Fortran 77
– Now written in C/C++ • Gaussian elimination
• LU decomposition
Use MATLAB to check your work! • More advanced
Vector/Matrix algebra
 Sam Siewert 16
 MPI Examples of Vector / Matrix
https://www.ecst.csuchico.edu/~sbsiewert/csci551/code/
MPI_Examples/

MPI Vector Addition

1. MPI_Examples/vector_add.c
2. MPI_Examples/mpi_vector_add.c

MPI Vector / Matrix Multiplication

1. MPI_Examples/mpi_mat_vect_mult.c
2. MPI_Examples/mpi_mat_vect_time.c

 Sam Siewert 17
Some Applications of Large Vector / Matrix
Ops – Beyond Benchmarks
Machine Learning (e.g., Polynomial least squares)
– https://www.youtube.com/watch?v=Kqh7stbGakg

Image rotation (any angle)

– https://www.youtube.com/watch?v=2i3mFA53mr0
– https://www.youtube.com/watch?v=hU_2pZxmQVU

Image processing and transformation (as we covered –

DCT, PSF, etc.)

Finite Element Methods (Mesh methods)

– https://www.youtube.com/watch?v=GHjopp47vvQ

 Sam Siewert 18
 Linear Least Square Regression
A math model that deterministically finds best fit
– For linear regression, this is “linear least squares”
– https://en.wikipedia.org/wiki/Linear_least_squares#Example

Want best fit line y=mx + b

Given data we have 4 equations and 2 unknows:

Eq-1: 6 = m(1) + b
Eq-2: 5 = m(2) + b This system is
Eq-3: 7 = m(3) + b overdetermined
Eq-4: 10 = m(4) + b

y Rewrite accounting for residual error:

Eq-1: 6 = m(1) + b + r1, r1 = 6 - (b + m(1))
Eq-2: 5 = m(2) + b + r2, r2 = 5 – (b + m(2))
Eq-3: 7 = m(3) + b + r3, r3 = 7 – (b + m(3))
Eq-4: 10 = m(4) + b + r4, r4 = 10 – (b + m(4))
 Sam Siewert Try out Desmos for experimental math - here 19
 Linear Least Squares - Continued
Sum of squared residuals S, is a best “fit” when S is
minimized and use symbolab to solve n equations, n
unknowns Use: Symbolab
Accounting for residual error: Equation Calculator
Eq-1: r1 = 6 - (b + m(1))
Eq-2: r2 = 5 – (b + m(2))
Eq-3: r3 = 7 – (b + m(3))
Eq-4: r4 = 10 – (b + m(4))

S(b, m) = r12 + r22 + r32 + r42

= (6 – (b + m(1)))2 + (5 – (b + m(2))) 2 + (7 – (b +
m(3))) 2 + (10 – (b + m(4))) 2

S(b, m) = 4b2 + 30m2 + 20b(m) – 56b – 154m + 210

Therefore:
y=
y
8b + 20m = 56
20b + 60m = 154
 Sam Siewert 20
 What does this look like? – Curve Fit
Traversing the S(b, m) = MSE of residuals surface for
the minimum
– b = 0.0, …, 10.0, stepping by surface grid size (b can be Inf)
– m = 0.0, …, 10.0, stepping by surface grid size (m can be Inf)

Gradient Descent Learning of b and m

Recall that S(b, m) = MSE or (r12 + …, + rn)/n

Get datasets from Kaggle

 Sam Siewert 21
 Polynomial Regression
Nth degree polynomial used to
model X, Y model
β0 + β1x + β2x2 + β3x3 + … βnxn + ϵ

Multiple Linear Regression

– Treat x, x2, x3, …, xn as independent
variable regression models A cubic polynomial regression fit to a simulated
data set. The confidence band is a 95%
– β0 + β1xi + β2xi2 + β3xi3 + … βnxin + ϵi (i = simultaneous confidence band constructed
using the Scheffé approach. - Wikipedia
1,2, …, n)

Becomes a system of
linear equations

Requires matrix
inversion of coefficienst
 Sam Siewert 22
 MATLAB Polyfit
https://www.mathworks.com/help/matlab/ref/polyfit.html

Increasing degree in 2D space with 3D cost/fitness

function

Linear regression – potentially underfits

Polynomial regression – higher degree, possibly a better

model, but potentially overfits

Ideal model will generalize well

– Makes good predictions for test data (low MSE between test
data and model)
– Test data predictions can be compared to truth (measured)
 Sam Siewert 23
 3D Vector Rotation
Rotate coordinate system through Euler angles
– E.g., aircraft center of mass roll, pitch and yaw
– Center of rotation – e.g., center of mass Robotics - Corke p. 23
Orthonormal rotation (one plane at a time) • 1D rotation of vector
Rx(θ) = x = = (sine, cosine)
Ry(θ) = x = =
• 2D rotation in a plane
Rz(θ) = x = =
(XY)
3-Space Column Vectors rotated about a specific
• axis
Extend to 3D vector
with 3 planar rotations

• Rotation axis X, Y, Z

Rz(θ)

θ
θ Ry(θ) θ
θ
θ
Rx(θ)

 Sam Siewert 24
 Recall: General Rotation, Scaling, Translation
CSCI 551 C Starter Code – here
– JavaScript Examples – here
– Online Matrix Calculator – here Rotate 2D image by X degrees
– C/C++ Translation, Rotation, and Scaling
– here
– Basic C++ Examples from Open Source –
here Affine Example

Distortion Removal Example

Affine Transformations – deals with

multiple operations, shearing and
perspective viewpoint as well as rigid
transformations

Distortion and Warping – E.g.,

Fisheye, Barrel, and Pincushion (
MATLAB
Siewert, Camera
Sam B., et al. " Calibrator)
Slew-to-Cue Electro-Optical and Infrared Sensor Network for sma
ll UAS Detection, Tracking and Identification
." AIAA SciTech 2019 Forum. 2019.
 Sam Siewert 25
Options to Check Work – Beyond MATLAB
Please use Math Tools to check
Symbolab work, not to do work

Mathematica – Student Edition

(Alternative to MATLAB)
Maple – Student Options
Python - SageMath

Web-based Symbolab -
https://www.symbolab.com
Handy Math -
https://handymath.com/cgi-bin/m
atrix3d.cgi

OnlineMSchool -
https://onlinemschool.com/math/
 Sam Siewert
assistance/ 26