E1 216 COMPUTER VISION

LECTURE 02 : CAMERA GEOMETRY

Venu Madhav Govindu

Department of Electrical Engineering
Indian Institute of Science, Bengaluru

2021
From previous lecture:
• UG course approval
• Linear algebra, Probability, SP, Programming
• Timing ? Extra Classes
• Discuss model-algorithm distinction later
Szeliski 2nd Edition
 Projection Models

Szeliski 2nd Edition

 Camera Geometry

How do we capture light ?

Science for the Curious Photographer, Steve Seitz

 Camera Geometry

How do we capture light ?

Pinhole Camera

Science for the Curious Photographer, Steve Seitz

 Camera Geometry

How do we capture light ?

Pinhole Camera

Why ?

Science for the Curious Photographer, Steve Seitz

 What is a Camera ?

Camera = Pinhole

Science for the Curious Photographer, wikipedia

 What is a Camera ?

Camera = Pinhole

Powerful Mathematical Model

Science for the Curious Photographer, wikipedia

 Camera Geometry

Pinhole Camera Model

• What are the consequences of this model ?
• Imagine you project a 3D point onto the image plane
• Where did it come from ?
 Camera Geometry

Pinhole Camera Model

• What are the consequences of this model ?
• Imagine you project a 3D point onto the image plane
• Where did it come from ?
 Camera Geometry

Pinhole Camera Model

• What are the consequences of this model ?
• Imagine you project a 3D point onto the image plane
• Where did it come from ?
 Camera Geometry

Recovering 3D Geometry
• Consider two cameras (one is never enough)
• Take pictures
• Maps to points on image planes
• Know linear constraint on 3D point from left camera
• Use right camera constraint to intersect
 Camera Geometry

Recovering 3D Geometry
• Consider two cameras (one is never enough)
• Take pictures
• Maps to points on image planes
• Know linear constraint on 3D point from left camera
• Use right camera constraint to intersect
 Camera Geometry

Recovering 3D Geometry
• Consider two cameras (one is never enough)
• Take pictures
• Maps to points on image planes
• Know linear constraint on 3D point from left camera
• Use right camera constraint to intersect
 Camera Geometry

Recovering 3D Geometry
• Consider two cameras (one is never enough)
• Take pictures
• Maps to points on image planes
• Know linear constraint on 3D point from left camera
• Use right camera constraint to intersect
 Camera Geometry

Recovering 3D Geometry
• Consider two cameras (one is never enough)
• Take pictures
• Maps to points on image planes
• Know linear constraint on 3D point from left camera
• Use right camera constraint to intersect
 Camera Geometry

Many Considerations
• Do we know camera parameters ? (intrinsic calibration)
• Do we know orientations of cameras ? (extrinsic calibration)
• Match features (representation,matching,robustness)
• Do the backprojected rays intersect ? (structure estimation)
• Extend this principle to multiple images
• Non-trivial, but many important advances
• State-of-the-art can handle large datasets (> 104 images)
 Camera Geometry

Many Considerations
• Do we know camera parameters ? (intrinsic calibration)
• Do we know orientations of cameras ? (extrinsic calibration)
• Match features (representation,matching,robustness)
• Do the backprojected rays intersect ? (structure estimation)
• Extend this principle to multiple images
• Non-trivial, but many important advances
• State-of-the-art can handle large datasets (> 104 images)
 Camera Geometry

Many Considerations
• Do we know camera parameters ? (intrinsic calibration)
• Do we know orientations of cameras ? (extrinsic calibration)
• Match features (representation,matching,robustness)
• Do the backprojected rays intersect ? (structure estimation)
• Extend this principle to multiple images
• Non-trivial, but many important advances
• State-of-the-art can handle large datasets (> 104 images)
 Camera Geometry

Many Considerations
• Do we know camera parameters ? (intrinsic calibration)
• Do we know orientations of cameras ? (extrinsic calibration)
• Match features (representation,matching,robustness)
• Do the backprojected rays intersect ? (structure estimation)
• Extend this principle to multiple images
• Non-trivial, but many important advances
• State-of-the-art can handle large datasets (> 104 images)
 Camera Geometry

Many Considerations
• Do we know camera parameters ? (intrinsic calibration)
• Do we know orientations of cameras ? (extrinsic calibration)
• Match features (representation,matching,robustness)
• Do the backprojected rays intersect ? (structure estimation)
• Extend this principle to multiple images
• Non-trivial, but many important advances
• State-of-the-art can handle large datasets (> 104 images)
 Camera Geometry

Many Considerations
• Do we know camera parameters ? (intrinsic calibration)
• Do we know orientations of cameras ? (extrinsic calibration)
• Match features (representation,matching,robustness)
• Do the backprojected rays intersect ? (structure estimation)
• Extend this principle to multiple images
• Non-trivial, but many important advances
• State-of-the-art can handle large datasets (> 104 images)
 Camera Geometry

Many Considerations
• Do we know camera parameters ? (intrinsic calibration)
• Do we know orientations of cameras ? (extrinsic calibration)
• Match features (representation,matching,robustness)
• Do the backprojected rays intersect ? (structure estimation)
• Extend this principle to multiple images
• Non-trivial, but many important advances
• State-of-the-art can handle large datasets (> 104 images)
 What’s a Good Camera Model ?

Camera Systems
• Camera imaging surface - typically a rectangular plane
• Human retina is closer to a spherical surface
• Vastly different image plane geometries
• Fundamental 3D-2D imaging model is the same
• Spatial sampling is uniform for typical cameras
• Omnidirectional cameras
 Camera Model : Perspective Projection

• Very simple geometry

• Sufficiently powerful representation
• Virtual Image considered in front of focus
• Real cameras do deviate from this model
 Camera Model : Perspective Projection

Y
X X
y
x
x fY/Z
C Z C
p p Z
principal axis f
camera
centre image plane

• Coordinate system with origin at camera centre

• World coordinates of point P = (X , Y , Z )
• Image projection measured in local image coordinate system
• Image coordinates p = (x, y)

By simple similarity of triangles we have

x = fZX
y = fZY
 Camera Model : Perspective Projection

Changing focal length

• Keep camera fixed, change focal length
• What happens to the volume imaged ?

Science for the Curious Photographer; Forstyh and Ponce 2nd Edition.
 Camera Model : Perspective Projection
fX
x= Z
fY
y= Z

Implications
• Different points are scaled different according to depth
• Introduces non-linearities in the relationships
• Distant objects are smaller
• Cannot judge object size with a single image
 Perspective projection
• Cannot judge object size with a single image
• Judgement of size can be wrong!

From twitter.com/rainmaker1973
 Y

Y
X X
y
x
x fY/Z
C Z C
p p Z
principal axis f
camera
centre image plane

Two co-ordinate systems!

• Remember that we have two measurements of interest
• Measurements on the image plane
• Measurements in the 3D world
• Our interest is to relate the two
 Camera Model (contd.)

Consider perspective projection model

f
   
x X
=
y Z Y
Now let’s translate the frame of reference (or camera), new co-ordinates
 0 
f
 
x X + tx
0 =
y Z + tz Y + ty
 Camera Model (contd.)

Consider perspective projection model

f
   
x X
=
y Z Y
Now let’s translate the frame of reference (or camera), new co-ordinates

0 (X + tx )
x = f
(Z + tz )
0 (Y + ty )
y = f
(Z + tz )
 Camera Model (contd.)

Or if we were to rotate the camera by rotation matrix R

r11 r12 r13

 

R =  r21 r22 r23 

r31 r32 r33
The new 3D coordinates would be
 0  
r11 r12 r13
 
X X
 0  
 Y  = r21 r22 r23   Y 
Z
0
r31 r32 r33 Z
 Camera Model (contd.)

Therefore, the new image projections would look like

r11 X + r12 Y + r13 Z

x = f
r31 X + r32 Y + r33 Z
r21 X + r22 Y + r23 Z
y = f
r31 X + r32 Y + r33 Z

• Now if we apply an additional transformation, the two rotations

would get entangled
• End result of multiple transformations is very messy!
• Need a cleaner approach
 Homogeneous Representations

To arrive at a solution, we take recourse to geometry

Geometric approaches
• “Purist” view - co-ordinate free approach to geometry
• Classical theorems due to Euclid
• Since Descartes, there’s an algebraic view of geometric constructs
• Duality : Geometry ↔ Algebra
T
• Circle : Centre + Radius ↔ (p − p 0 ) (p − p 0 ) = r 2
 Homogeneous Representations

Consider a line y = mx + c
Rewrite as mx − y + c = 0
or generally as
ax+by+c = 0

Rewriting this we have

x
 

a b c  y =0
 
1
 Homogeneous Representation of a Line

p
z }| {
 x
a b c  y =0

| {z } 1
l

this results in a nice symmetric form

lT p = 0

This form has many advantages over y = mx + c form

 Homogeneous Representation of a Line

Consider the intersection of two lines

To solve for the point of intersection
y = m1 x + c1
y = m2 x + c2
(y−c1 )
Solve simultaneous equations by substitution, x = m1

m2
y = (y − c1 ) + c2
m1
m2 c1 m2
(1 − )y = c2 −
m1 m1
(c2 − c1mm12 )
y = =
(1 − mm21 )

Quite a mess!!
 Homogeneous Representation of a Line

In the homogeneous system of representation we have

l 1T p = 0
l 2T p = 0

Therefore, the co-ordinates of the intersection is given by

p = l1 × l2

Verify
• l 1 T (l 1 × l 2 ) = 0
• l 2 T (l 1 × l 2 ) = 0
• Much cleaner way of solving
 Homogeneous Representation of a Line
Consider the line through two given points
(X1,Y1)

Line : Y=mX+c

(X2,Y2)

Usual solution is messy

Instead, using homogenous coordinates, we get the dual representation

Line : l = p 1 × p 2

Easily verified that this satisfies the requirements

T
• (p 1 × p 2 ) p 1 =0
T
• (p 1 × p 2 ) p 2 =0
 Homogeneous Representation

The key relationship to note is that

p
z }| {
 x
a b c  y =0

| {z } 1
l

results in a nice symmetric (and homogeneous) form

lT p = 0

This form has many advantages over y = mx + c form

 Homogeneous Representation

In homogenous form everything upto unknown scalar

Homogeneous Inhomogeneous
Rn 7→ Rn+1 Rn 7→ Rn−1
 u 
u u
   
u w
 
7  v 
→  v  7→  v 
v w
1 w 1

Homogeneous Forms
• Embed in higher dimensions by appending a 1 (canonical)
• Homogeneous forms are equivalent upto scale
• Only ratios matter
• [u, v, w] = λ [u, v, w] , ∀λ 6= 0
• Notice [0, 0, 0] is not admissible
 Homogeneous Representation

In homogenous form everything upto unknown scalar

Homogeneous Inhomogeneous
Rn 7→ Rn+1 Rn 7→ Rn−1
 u 
u u
   
u w
 
7  v 
→  v  7→  v 
v w
1 w 1

Homogeneous Forms
• Embed in higher dimensions by appending a 1 (canonical)
• Homogeneous forms are equivalent upto scale
• Only ratios matter
• [u, v, w] = λ [u, v, w] , ∀λ 6= 0
• Notice [0, 0, 0] is not admissible
 Geometries in Computer Vision

• Geometry : Topological Space + Axioms

• Different set of axioms Different Geometries
• Euclidean (Distances and Angles)
• Affine (Parallelism)
• Projective (Straight Line)
• Non-linear (Riemannian Manifolds)

Stratification of transform space

Euclidean ⊂ Affine ⊂ Projective
 Euclidean Geometry

Axioms of incidence
• Familiar concepts from Euclidean geometry
• Length is a fundamental property of Euclidean Geometry
• Construction with straightedge and compass
• Axioms of Euclid

Following Hilbert state the axioms as

• For any two points A, B, a unique line passes through A, B
• Every line contains at least two points
• There exist three points not all on the same line
• Parallel axiom : For any line L and point P outside L, there is
exactly one line through P that does not meet L
Wikipedia

• Two points have a unique line through them (join)

• Two lines have a unique intersection point (meet)
• What happens when the lines are parallel ?
• What does it mean to say that they “intersect at ∞”?
• Question : Are all ∞ intersection points the same ?
• The answer lies in the geometry of projective space
• Take a detour via affine geometry
• Recall homogeneous representations
 Homogeneous Forms

Parallel Lines
T
• Recall line equation: l p = 0
T
• l and p upto scale factor l T p = (λl) (λ p) = 0
0

• Intersection of two lines p = l 1 × l 2

• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Homogeneous Forms

Parallel Lines
T
• Recall line equation: l p = 0
T
• l and p upto scale factor l T p = (λl) (λ p) = 0
0

• Intersection of two lines p = l 1 × l 2

• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Homogeneous Forms

Parallel Lines
T
• Recall line equation: l p = 0
T
• l and p upto scale factor l T p = (λl) (λ p) = 0
0

• Intersection of two lines p = l 1 × l 2

• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Homogeneous Forms

Parallel Lines
T
• Recall line equation: l p = 0
T
• l and p upto scale factor l T p = (λl) (λ p) = 0
0

• Intersection of two lines p = l 1 × l 2

• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Homogeneous Forms

Parallel Lines
T
• Recall line equation: l p = 0
T
• l and p upto scale factor l T p = (λl) (λ p) = 0
0

• Intersection of two lines p = l 1 × l 2

• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Homogeneous Forms

Parallel Lines
T
• Recall line equation: l p = 0
T
• l and p upto scale factor l T p = (λl) (λ p) = 0
0

• Intersection of two lines p = l 1 × l 2

• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Homogeneous Forms

Parallel Lines
T
• Recall line equation: l p = 0
T
• l and p upto scale factor l T p = (λl) (λ p) = 0
0

• Intersection of two lines p = l 1 × l 2

• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Homogeneous Forms

0 0
(c − c)b (c − c )a 0
 
p = l1 × l2 =
= [b, −a, 0]

What is the inhomogeneous form of p ?

Parallel Lines
• Recall line equation: l = p 1 × p 2
• l and p upto scale factor
• Intersection of two lines p = l 1 × l 2
• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Homogeneous Forms

0 0
(c − c)b (c − c )a 0
 
p = l1 × l2 =
= [b, −a, 0]

What is the inhomogeneous form of p ?

Distinct “points at infinity”

Parallel Lines
• Recall line equation: l = p 1 × p 2
• l and p upto scale factor
• Intersection of two lines p = l 1 × l 2
• When are lines parallel ?
• l1 = a b c
 
0 
• l2 = a b c


• Intersection point p ?
 Affine Geometry

Consider v 1 , v 2 ∈ R2
Linear combination: Span {v 1 , v 2 }
Affine combination: Line in R2

Affine Combinations
Linear Combinations
Consider vectors v 1 , · · · , v k ∈ Rn
Consider vectors v 1 , · · · , v k ∈ Rn
Affine Combination: λ1 v 1 + · · · λk v k
Linear Combination:
λ1 , · · · , λk ∈ R
λ1 v 1 + · · · + λk v k ∈ Rn
Restriction: λ1 + · · · + λk = 1
λ1 , · · · , λk ∈ R
Recall further restriction λi ≥ 0
 Affine Geometry

Affine Subspace
Vector Subspace • A ⊆ Rn
• A⊆R n
• No origin
• 0∈A • a ∈ A ⇒ λa ∈ A
• a ∈ A ⇒ λa ∈ A • a, b ∈ A ⇒ λa + (1 − λ)b ∈ A
• a, b ∈ A ⇒ a + b ∈ A • A − a is a vector space for any
• Points and vectors coincide a∈A
• Equipped with inner product • Vectors only as differences
• Distances and angles preserved (translations)
• Only parallelism is preserved
 Affine Geometry

Affine Subspaces
• Consider the 2D plane, but forget origin
• What can two independent observers agree upon ?
• Second observer assumes that p is the origin
• Adding two vectors a and b results in p + (a − p) + (b − p)
• When linear combination is λa + (1 − λ)b, observers agree
• Observers know the “affine structure” but not the “linear structure”
• Direction is a fundamental property here, not length

wikipedia
 Projective Geometry

• Represent the projective plane as P2

• Obtained by adding all ∞ points
• ∞ points form a ‘line at infinity’. Why ?
• Got rid of special case of parallel lines
• All lines have a unique intersection now
• So what is this space useful for ?
 Projective Geometry

(d) P2 ≡ S 2 (e) P2 ≡ R3 \{0}/ '

• Projective plane is topologically equivalent to unit sphere

• Associate with half-sphere to projective scale
• Where is the line at infinity on S 2 ?
• P2 is equivalent to R3 with origin removed, under equivalence
relationship of scale
 Homogeneous Form

Basic Definition
• n-dim real affine space is set of all points
(x1 , · · · , xn )  Rn
• Projective space Pn given by
• (x1 , · · · , xn , xn+1 )  Rn+1
• at least one xi is non-zero
• for λ 6= 0, all (λx1 , · · · , λxn , λxn+1 ) are equivalent
• Homogeneous coordinates obtained by (x1 , · · · , xn , 1)
 Homogeneous Form

• Let the homogeneous form be X = (X 1 , · · · , X n+1 )

• Let the inhomogeneous form be x = (x 1 , · · · , x n )
• Equivalence relationship : [x, 1] = (x 1 , · · · , x n , 1) ' X
Xi
• xi = X n+1
• Question : What is the homogeneous form for points at ∞ ?
 Camera = Camera Centre!
• Consider a centre of projection
• Establishes equivalence classes
• All points on ray are projectively equivalent (beads on wire)
• What happens when they line up ?
• Camera model
 Transformation Groups

Invariances to Group Action

• Define properties invariant to group of transformations
 Group Properties

• We will only consider finite dimensional groups

• n × n matrices
• For A,B and C in G,
• AB and BA in G (closure;right and left action)
• A(BC ) = (AB)C (associative)
• For every A there exists I s. t. AI = I A = A (identity)
• For every A there exists A−1 s. t. AA−1 = A−1 A (inverse)
• Group is commutative or Abelian if AB = BA
• What are the conditions on G ?
     
h11 h12 h13 h11 h12 h13 r 11 r 12 t 1
 h21 h22 h23   h21 h22 h23   r 21 r 22 t 2 
h31 h32 h33 0 0 1 0 0 1
Projective Affine Euclidean
 Euclidean Projective

0  0 
x x x x
       
h11 h12 h13 h11 h12 h13
0
h23   y 
0
 y  =  h21 h22  y  =  h21 h22 h23   y 
z z
0 0
z h31 h32 h33 z h31 h32 h33

Two interpretations
• Euclidean vs. Projective transformations
• H : R3 → R3 (9 dof)
• H : P2 → P2 (8 dof)
 Projective Geometry

Since in projective space

Pn , all (λx1 , · · · , λxn , λxn+1 )
are equivalent, we can linearise our imaging model
Recall that
1
   
x X
=
y Z Y
For now assume f = 1
then by embedding image and world points in projective spaces we have
   
x X
1
 y =  Y 
Z
1 Z
 Projective Geometry

We now have
   
x X
1
 y =  Y 
Z
1 Z
Recall, that scaled points are projectively equivalent, i.e.
   
x X
 y = Y 
1 Z

p=P

We have now managed to linearise the relationship

 Projective Geometry

Projective representations for both image and world points

 
 X
1 0 0 0 
  
x
 y = 0 Y
1 0 0 


 Z
1 0 0 1 0

1

Euclidean transformation of 3D points

 0  
r11 r12 r13
 
X tx X
 0  
 Y   r21 r22 r23  Y
ty   
 0 =
r31 r32 r33

 Z  tz   Z 
1 0 0 0 1 1
 Projective Geometry
General process of taking a picture
• Apply Euclidean motion to 3D points
• Project onto image plane
Combining two steps we get
0
 
X
1 0 0 0 
   
x 0
Y

 y  =  0 1 0 0 
 
0 
1 0 0 1 0  Z 
1
 r r12 r13
  
tx X
1 0 0 0  11
 
x
r21 r22 r23 ty   Y
=  0 1 0 0 

⇒ y   r31 r32 r33
 
tz   Z
1 0 0 1 0

0 0 0 1 1
 
 X
x   Y 
⇒ y  = R t  
 Z 
1
1
 Projective Geometry

r11 r12 r13

  
tx X
1 0 0 0
   
x  r21
 y  =  0 r22 r23 ty   Y
1 0 0 

 r31 r32 r33
 
tz   Z
1 0 0 1 0

0 0 0 1 1

Taking a Picture
• 3D Point in Homogeneous Form
• Rigid 3D Motion
• Ideal Pinhole Camera
• Image Projection
• P3 → P2
 Projective Geometry

  
r11 r12 r13 X

tx
1 0 0 0
   
x  r21
 y  =  0 1 0 0  r22 r23 ty   Y
 


 r31 r32 r33 tz   Z 
1 0 0 1 0 1
0 0 0 1

Taking a Picture
• 3D Point in Homogeneous Form
• Rigid 3D Motion
• Ideal Pinhole Camera
• Image Projection
• P3 → P2
 Projective Geometry

r11 r12 r13

  
tx

X
1 0 0 0
   
x  r21 r22 r23 ty 
 y  =  0 Y
1 0 0  
 
 r31 r32 r33
 
tz   Z

1 0 0 1 0

0 0 0 1 1

Taking a Picture
• 3D Point in Homogeneous Form
• Rigid 3D Motion
• Ideal Pinhole Camera
• Image Projection
• P3 → P2
 Projective Geometry

  r r12 r13 tx

X

1 0 0 0 11
  
x  r21
 y  =  0 1 0 0  r22 r23 ty   Y 
 r31 r32 r33
  
0 0 1 0 tz   Z
1

0 0 0 1 1

Taking a Picture
• 3D Point in Homogeneous Form
• Rigid 3D Motion
• Ideal Pinhole Camera
• Image Projection
• P3 → P2
 Projective Geometry

r11 r12 r13

  
  tx X
x 1 0 0 0
 
 y  =  0  r21 r22 r23 ty   Y
1 0 0 

 r31 r32 r33
 
1 tz   Z
0 0 1 0

0 0 0 1 1

Taking a Picture
• 3D Point in Homogeneous Form
• Rigid 3D Motion
• Ideal Pinhole Camera
• Ideal Projection
• P3 → P2
 Projective Geometry

r11 r12 r13

  
tx X
1 0 0 0
   
x  r21
 y =  0 r22 r23 ty   Y
1 0 0 

 r31 r32 r33
 
tz   Z
1 0 0 1 0

0 0 0 1 1

Taking a Picture
• 3D Point in Homogeneous Form
• Rigid 3D Motion
• Ideal Pinhole Camera
• Image Projection
• P3 → P2
 Projective Geometry

r11 r12 r13

  
tx X
1 0 0 0
   
x  r21
 y =  0 r22 r23 ty   Y
1 0 0 

 r31 r32 r33
 
tz   Z
1 0 0 1 0

0 0 0 1 1

Taking a Picture
• 3D Point in Homogeneous Form
• Rigid 3D Motion
• Ideal Pinhole Camera
• Image Projection
• P3 → P2