Motion Models
• Image Transformations to relate two images
• 3D Rigid motion
• Perspective & Orthographic Transformation
• Planar Scene Assumption
• Transformations
– Translation
– Rotation
– Rigid
– Affine
– Homography
– Pseudo Perspective
Global Flow
Lecture‐9
Global Motion
• Estimate motion using all pixels in the image.
• Parametric flow gives an equation, which describes
optical flow for each pixel.
– Affine
– Projective
• Global motion can be used to
– Remove camera (ego) motion (motion compensation)
– Object‐based segmentation
– generate mosaics
Contents
• Bergen et al method
– Affine transformation
• Mann & Piccard
– Homography (Projective)
– Pseudo Perspective
– Bi‐linear
• Image Warping
• Applications
– Mosaics
– COCOA system
Bergan et al
Affine
Affine
Image at t-1 Image at t
(0,0)
(0,0)
x (x,y)
x
(x’,y’)
(1,1) (1,1)
Affine
Bergan et al
•Affine
Bergan et al
Optical flow constraint eq
Bergan et al
min
Homework
Linear system
Basic Components
• Pyramid construction
• Motion estimation
• Image warping
• Coarse‐to‐fine refinement
Coarse‐to‐fine global flow estimation
u=1.25 pixels
u=2.5 pixels
u=5 pixels
image H u=10 pixels image I
Gaussian pyramid of image H Gaussian pyramid of image I
Coarse‐to‐fine global flow estimation
Compute Global Flow Iteratively
warp & upsample
Compute Global Flow Iteratively
.
.
.
image JH image I
Gaussian pyramid of image H Gaussian pyramid of image I
a0
Level=2 W M
2
a1
2
W
* + * W
a1
Level=1 W M
a2
2 2
+
W a2 W
* *
W M
Level=0
a3
f(x,y,t-1) + f(x,y,t-1) f(x,y,t)
a3
Estimation of Global Flow
Single Iteration
Compute A and B
Solve Aa B
Image ‘t’ Image ‘t+1’
Warp by a
Estimation of Global Flow
Iterative
Initial Estimate a a1 a2 b1 a3 a4 b2 T
Image ‘t’ Image ‘t+1’
Warp by a Compute A and B
Solve Aa B
Warp by a
Estimation of Global Flow
Iterative
Initial Estimate a a1 a2 b1 a3 a4 b2 T
Iterate
Image ‘t’ Image ‘t+1’
Warp by a Compute A and B
Solve Aa B
Update a
Image Warping
• Warping an image f into image h using some
transformation g,
• involves mapping intensity at each pixel (x,y)
in image f to a pixel (g(x),g(y)) in image h
such that
In case of affine transformation,
is transformed to as:
Displacement model
Instantaneous model
Image Warping (Bergan et al)
x
x
warp
Image Warping
•How about values in are not
integer.
•But image is sampled only at integer rows
and columns
• Instead of converting to and copying
at we can convert
integer values to and copy
at
Image Warping
x
x
warp
Image Warping
• But how about the values in are not
integer.
• Perform bilinear interpolation to compute
at non-integer values.
Warping
Warped image at t-1
Difference image before Difference image after
Video Mosaic
Video Mosaic
Video Mosaic
mosaic
Sprite
Mann & Picard
Projective
Projective Flow (weighted)
Optical Flow const.
equation
Projective transform
Projective Flow (weighted)
minimize
Homework
Projective Flow (weighted)
Projective Flow (unweighted)
Pseudo‐Perspective
Taylor Series
Bilinear
Taylor Series & remove
Square terms
Projective Flow (unweighted)
Minimize
Bilinear and Pseudo‐Perspective
bilinear
Pseudo perspective
Homework
Algorithm‐1
• Estimate “q” (using approximate model, e.g.
bilinear model).
• Relate “q” to “p”
– select four points S1, S2, S3, S4
– apply approximate model using “q” to compute (xk , yk )
– estimate exact “p”:
Determining Projective transformation using
point correspondences
• If point correspondences (x,y)<-->(x’,y’) are known
• a’s can be determined by least squares fit
Two rows for each point i
Determining Projective transformation using
point correspondences
Final Algorithm
• A Gaussian pyramid of three or four levels is
constructed for each frame in the sequence.
• The parameters “p” are estimated at the top
level of the pyramid, between the two lowest
resolution images, “g” and “h”, using
algorithm‐1.
Final Algorithm
• The estimated “p” is applied to the next
higher resolution image in the pyramid, to
make images at that level nearly congruent.
• The process continues down the pyramid until
the highest resolution image in the pyramid is
reached.
Video Mosaics
• Mosaic aligns different pieces of a scene
into a larger piece, and seamlessly blend
them.
– High resolution image from low resolution
images
– Increased filed of view
Steps in Generating A Mosaic
• Take pictures
• Pick reference image
• Determine transformation between
frames
• Warp all images to the same reference
view
Applications of Mosaics
• Virtual Environments
• Computer Games
• Movie Special Effects
• Video Compression
Steve Mann
Sequence of Images
Projective Mosaic
Affine Mosaic
Building
Wal‐Mart
Scientific American Frontiers
Scientific American Frontiers
Head‐mounted Camera at Restaurant
MIT Media Lab
COCOA: A System for Processing of
Aerial Videos
Saad Ali and Mubarak Shah, COCOA - Tracking in Aerial Imagery, SPIE
Airborne Intelligence, Surveillance, Reconnaissance (ISR) Systems and
Applications, Orlando, 2006.
COCOA – System Flow
Aerial Video
Telemetry*
COCOA
Ego Motion Motion Detection Object Tracking
Compensation
Accumulative Frame Kernel Tracking + Blob
Differencing + Background Tracking + Occlusion Handling
Feature based + Gradient Modeling + Object
Based Segmentation
Registered Images Motion Detection Tracks
Event Detection & Indexing
Registration Result ‐ I
Aerial Video - EO Mosaic
Alignment Mask
Registration Result ‐ II
Aerial Video - IR Mosaic
Alignment Mask
Detection Results
Tracking Results
References
• J. Bergen, P. Anandan, K. Hanna, and R. Hingorani,
“Hierarchical Model‐Based Motion Estimation”, ECCV‐92,
pp 237‐22.
• Video orbits of the projective group a simple approach to
featureless estimation of parameters S Mann, RW Picard
‐ Image Processing, IEEE Transactions on, 1997
• Saad Ali and Mubarak Shah, COCOA ‐ Tracking in Aerial
Imagery, SPIE Airborne Intelligence, Surveillance,
Reconnaissance (ISR) Systems and Applications, Orlando,
2006.
• R. Szeliski. “Video mosaics for virtual environments”, IEEE
Computer Graphics and Applications, pages,22‐30, March
1996.
