COMP499G/691G

COMPUTER VISION

LECTURE 1:
INTRODUCTION TO COMPUTER VISION

Copyright © Charalambos Poullis

Today’s Lecture
• Introductions
• What is Computer Vision?
• Computer Vision @ Concordia University
• Course administrative details
• Questions

Copyright © Charalambos Poullis

Teaching Assistant/Grader
TAs: Amin Karimi, Farhan Rahman Wasee
Grader: Qiao [Jocelyn] Chen
Office: EV3.260
Office hours: During lab sessions or by appointment

Copyright © Charalambos Poullis

Instructor
Dr. Charalambos [Charis] Poullis

Associate Professor in Computer Vision

Office: EV3.183
Office hours: 2-3pm every Monday
or by appointment

Copyright © Charalambos Poullis

Copyright © Charalambos Poullis
 CYPRUS

Copyright © Charalambos Poullis

 CYPRUS

Copyright © Charalambos Poullis

Copyright © Charalambos Poullis
 CYPRUS

Copyright © Charalambos Poullis

 CYPRUS

Copyright © Charalambos Poullis

 CYPRUS

Copyright © Charalambos Poullis

Copyright © Charalambos Poullis
Research Interests
• Fundamental research

Computer Vision

Computer Graphics
• Applied Research

Virtual Reality

Augmented Reality

www.theICTlab.org
Copyright © Charalambos Poullis
Recent work

Semantic Segmentation from Remote Sensor Data and the Exploitation of Latent Learning for Classification of Auxiliary
Tasks, B. Chaterjee, C. Poullis, CRV 2019

Copyright © Charalambos Poullis

Recent work

Delineation of Road Networks Using Deep Residual Neural Networks and Iterative Hough Transform, P. Xu, C. Poullis, ISVC
2019

Copyright © Charalambos Poullis

 Recent work

Inattentional Blindness for Redirected Walking Using Dynamic Foveated Rendering, Under Review 2020

Copyright © Charalambos Poullis

 The goal of computer vision
• To extract “meaning” from pixels

What we see
Source: S. Narasimhan
 The goal of computer vision
• To extract “meaning” from pixels

What we see What a computer sees

Source: S. Narasimhan
 The goal of computer vision
• To extract “meaning” from pixels

Humans are remarkably good at this…

Source: “80 million tiny images” by Torralba et al.
Can computers match (or beat) human
vision?
Can computers match (or beat) human
vision?
• Yes and no (but mostly no!)
• humans are much better at
“hard” things
• computers can be better at
“easy” things
Human perception has its shortcomings…
What kind of information can be
extracted from an image?
 What kind of information can be
extracted from an image?

Semantic information
 What kind of information can be
extracted from an image?

European
City

Outdoor scene
…

Semantic information
 What kind of information can be
extracted from an image?
tree tree
roof
sky chimney
building
building
window
door
trashcan car car
person
Outdoor scene
ground City European
…

Semantic information
 What kind of information can be
extracted from an image?
tree tree
roof
sky chimney
building
building
window
door
trashcan car car
person
Outdoor scene
ground City European
…

Semantic information Geometric information

Why study computer vision?
Why study computer vision?
• Vision is useful
Why study computer vision?
• Vision is useful
• Vision is interesting
Why study computer vision?
• Vision is useful
• Vision is interesting
• Vision is difficult
• Half of primate cerebral cortex is devoted to visual
processing
• Achieving human-level visual perception is probably
“AI-complete”
Why computer vision matters

Safety Health Security

Comfort Fun Access

Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
–“spend the summer linking a camera to a computer and getting the
computer to describe what it saw”
Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
–“spend the summer linking a camera to a computer and getting the
computer to describe what it saw”
• 1960’s: interpretation of synthetic worlds Guzman ‘68
Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
–“spend the summer linking a camera to a computer and getting the
computer to describe what it saw”
• 1960’s: interpretation of synthetic worlds Guzman ‘68
• 1970’s: some progress on interpreting selected
images

Ohta Kanade ‘78

Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
–“spend the summer linking a camera to a computer and getting the
computer to describe what it saw”
• 1960’s: interpretation of synthetic worlds Guzman ‘68
• 1970’s: some progress on interpreting selected
images
• 1980’s: ANNs come and go; shift toward geometry
and increased mathematical rigor
Ohta Kanade ‘78
Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
–“spend the summer linking a camera to a computer and getting the
computer to describe what it saw”
• 1960’s: interpretation of synthetic worlds Guzman ‘68
• 1970’s: some progress on interpreting selected
images
• 1980’s: ANNs come and go; shift toward geometry
and increased mathematical rigor
• 1990’s: face recognition; statistical analysis in vogue Ohta Kanade ‘78

Turk and Pentland ‘91

Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
–“spend the summer linking a camera to a computer and getting the
computer to describe what it saw”
• 1960’s: interpretation of synthetic worlds Guzman ‘68
• 1970’s: some progress on interpreting selected
images
• 1980’s: ANNs come and go; shift toward geometry
and increased mathematical rigor
• 1990’s: face recognition; statistical analysis in vogue Ohta Kanade ‘78
• 2000’s: broader recognition; large annotated
datasets available; video processing starts

Turk and Pentland ‘91

Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
–“spend the summer linking a camera to a computer and getting the
computer to describe what it saw”
• 1960’s: interpretation of synthetic worlds Guzman ‘68
• 1970’s: some progress on interpreting selected
images
• 1980’s: ANNs come and go; shift toward geometry
and increased mathematical rigor
• 1990’s: face recognition; statistical analysis in vogue Ohta Kanade ‘78
• 2000’s: broader recognition; large annotated
datasets available; video processing starts
• 2010’s: Deep learning with ConvNets

Turk and Pentland ‘91

Ridiculously brief history of computer vision
• 1966: Minsky assigns computer vision as an
undergrad summer project
–“spend the summer linking a camera to a computer and getting the
computer to describe what it saw”
• 1960’s: interpretation of synthetic worlds Guzman ‘68
• 1970’s: some progress on interpreting selected
images
• 1980’s: ANNs come and go; shift toward geometry
and increased mathematical rigor
• 1990’s: face recognition; statistical analysis in vogue Ohta Kanade ‘78
• 2000’s: broader recognition; large annotated
datasets available; video processing starts
• 2010’s: Deep learning with ConvNets
• 2030’s: robot uprising?
Turk and Pentland ‘91
Origins of computer vision

L. G. Roberts,
Machine Perception of Three Dime
nsional Solids
, Ph.D. thesis, MIT Department of
Electrical Engineering, 1963.
Successes of computer vision to date
Optical character recognition (OCR)

Digit recognition
yann.lecun.com

Source: S. Seitz, N. Snavely

Optical character recognition (OCR)

Digit recognition License plate readers

yann.lecun.com http://en.wikipedia.org/wiki/Automatic_number_plate_recognition

Source: S. Seitz, N. Snavely

Optical character recognition (OCR)

Digit recognition License plate readers

yann.lecun.com http://en.wikipedia.org/wiki/Automatic_number_plate_recognition

Automatic check processing Source: S. Seitz, N. Snavely

Optical character recognition (OCR)

Digit recognition License plate readers

yann.lecun.com http://en.wikipedia.org/wiki/Automatic_number_plate_recognition

Sudoku grabber
http://sudokugrab.blogspot.com/

https://www.youtube.com/watch?v=oI
mMJ6p6mKE
Automatic check processing Source: S. Seitz, N. Snavely
Biometrics

Face recognition systems now

Fingerprint scanners on
beginning to appear more widely
many new laptops,
http://www.sensiblevision.com/
other devices

Source: S. Seitz
Biometrics

How the Afghan Girl was Identified by Her Iris Patterns

Source: S. Seitz
Face detection

Many consumer digital cameras now detect faces

Source: S. Seitz
Smile detection

Sony Cyber-shot® T70 Digital Still Camera Source: S. Seitz

Face detection for privacy protection
Technology gone wild…
Technology gone wild…
Technology gone wild…
Technology gone wild…
Face recognition: Apple iPhoto software

http://www.apple.com/ilife/iphoto/
Visual search: Google search by image
 Google self-driving cars

https://www.youtube.com/watch?v=bDOnn0-4Nq8

• Google’s self-driving car passes 300,000 miles (Forbes, 8/15/2012)

• Nissan pledges affordable self-driving car models by 2020
(CNET, 8/27/2013)
 Automotive safety

Mobileye: Vision systems in high-end BMW, GM, Volvo models

• Pedestrian collision warning https://www.youtube.com/watch?v=HXpiyLUEOOY
• Forward collision warning
• Lane departure warning
• Headway monitoring and warning Source: A. Shashua, S. Seitz
Vision-based interaction: Xbox Kinect
3D Reconstruction: Kinect Fusion

YouTube Video
 Occipital: 3D structure sensor for iPad

http://www.kickstarter.com/projects/occi
pital/structure-sensor-capture-the-world
-in-3d

http://structure.io/#home-about-vid
3D reconstruction from photo collections

Q. Shan, R. Adams, B. Curless, Y. Furukawa, and S. Seitz,

The Visual Turing Test for Scene Reconstruction, 3DV 2013
YouTube Video
Object recognition
Google Goggles
Bing Vision
Special effects: shape and motion capture

Source: S. Seitz
Vision in space

NASA'S Mars Exploration Rover Spirit captured this westward view from atop
a low plateau where Spirit spent the closing months of 2007.

Vision systems (JPL) used for several tasks

• Panorama stitching
• 3D terrain modeling
• Obstacle detection, position tracking
• For more, read “Computer Vision on Mars” by Matthies et al.
Industrial robots

Vision-guided robots position nut runners on wheels

Mobile robots

NASA’s Mars Spirit Rover

http://www.robocup.org/
http://en.wikipedia.org/wiki/Spirit_rover

Saxena et al. 2008

STAIR at Stanford
Medical imaging

3D imaging Image guided surgery

MRI, CT Grimson et al., MIT
Why is computer vision difficult?
Challenges: viewpoint variation

Michelangelo 1475-1564 slide credit: Fei-Fei, Fergus & Torralba

 Challenges: viewpoint variation

slide credit: Fei Fei Li, Andrej Karpathy, Justin Johnson

Challenges: illumination

image credit: J. Koenderink

 Challenges: illumination

slide credit: Fei Fei Li, Andrej Karpathy, Justin Johnson

Challenges: scale

slide credit: Fei-Fei, Fergus & Torralba

Challenges: deformation

Xu, Beihong 1943

slide credit: Fei-Fei, Fergus & Torralba
 Challenges: deformation

slide credit: Fei Fei Li, Andrej Karpathy, Justin Johnson

Challenges: object intra-class variation

slide credit: Fei-Fei, Fergus & Torralba

 Challenges: object intra-class variation

slide credit: Fei Fei Li, Andrej Karpathy, Justin Johnson

 Challenges: occlusion

slide credit: Fei Fei Li, Andrej Karpathy, Justin Johnson

 Challenges:clutter

slide credit: Fei Fei Li, Andrej Karpathy, Justin Johnson

Challenges: occlusion, clutter

Image source: National Geographic

Challenges: Motion
Challenges: ambiguity

Source: Rob Fergus and Antonio Torralba

Challenges: ambiguity

Source: Rob Fergus and Antonio Torralba

Challenges: ambiguity

slide credit: Fei-Fei, Fergus & Torralba

 Challenges: ambiguity
• Many different 3D scenes could have given rise to a
particular 2D picture
 Review: Intro to computer vision
• State-of-the-art applications
• Challenges of vision
– Viewpoint and lighting variation
– Intra-class variations: size, shape, deformation, etc.
– Nuisances: motion, blur, noise, etc.
– Intrinsic ambiguity
 Challenges or opportunities?
• Images are confusing, but they also reveal the structure of
the world through numerous cues
• Our job is to interpret the cues!
Depth cues: Linear perspective
Depth cues: Parallax
Depth cues: Parallax
Shape cues: Texture gradient
Shape and lighting cues: Shading
Grouping cues: Similarity (color, texture,
proximity)
Grouping cues: “Common fate”

Image credit: Arthus-Bertrand (via F. Durand)

 The computer vision industry
• A list of companies here:

http://www.cs.ubc.ca/spider/lowe/vision.html
 Robotics
Computer Vision

Machine
Scope of Human Computer
Interaction
Learning COMP 425/6341
Image Processing Medical Imaging
Graphics
Feature Matching
Computational Recognition
Neuroscience
Photography

Optics
Computer Vision and Nearby Fields
• Computer Graphics: Models to Images
• Comp. Photography: Images to Images
• Computer Vision: Images to Models
Course Website
http://www.poullis.org/courses/2020/Winter/COMP425-6341

▪lecture notes are posted the day of the class

▪bring to class and keep notes

▪The lecture notes are password protected

▪user: computer_vision
▪pass: lecture_notes2020

Copyright © Charalambos Poullis

Syllabus - Course Outline
Posted on the website

Copyright © Charalambos Poullis

Pre-requisites
• Data structures
• A good working knowledge of C and C++
programming
• Linear algebra
• Vector Calculus

• No prior knowledge of image processing, computer

vision is assumed

Copyright © Charalambos Poullis

Grading
• Programming assignments (10%+15%)
• Project (20%)
• Quiz 1 (25%)
• Quiz 2 (30%)

• Late submission policy

• Extra 3 days at the cost of 2 points [out of 10]
• No extra credit will be awarded

Copyright © Charalambos Poullis

Academic Integrity Policy
• Feel free to discuss assignments with each other,
but coding must be done individually

• Feel free to incorporate code or tips you find on the

Web, provided this doesn’t make the assignment
trivial and you explicitly acknowledge your sources

Copyright © Charalambos Poullis

Submissions
• Electronic Assignment Submission
• https://fis.encs.concordia.ca/eas/

• Do not get confused: the dates on EAS are the late

submission deadlines

Copyright © Charalambos Poullis

Suggested Textbooks

http://szeliski.org/Book/
Copyright © Charalambos Poullis
Labs
• Start on Tuesday, 14th January 2020

• Tutorials in OpenCV
• OpenCV API

Copyright © Charalambos Poullis

Slides Acknowledgements
The slides are a combination of multiple resources and
materials generously made publicly available by

R. Szeliski, A. Efros,
S. Lazebnik,
D. Forsyth, B. Freeman, F. Durand,
J. Ponce, M. Pollefeys, L. Fei-Fei,
J. Koenderink, D. Lowe, A. Torralba,
S. Seitz, K. Grauman, R. Fergus,
J. Hays
Copyright © Charalambos Poullis
 Assignments and Project
• #1: Demosaicing
• #2: Feature Detection and Matching
• Project: Image Stitching
 Demosaicing
• Implement the conversion of the Bayer pixel pattern to an
RGB representation where each pixel has red, green and blue
color channels
 Feature Detection and Matching
• Implement a feature detection and matching algorithm
• Feed feature matches to a structure-from-motion system

Image credit: S. Szeliski Image credit: J. Hays

 Image Stitching
• Implement a feature-based alignment and image stitching
algorithm

Image credit: S. Szeliski

Copyright © Charalambos Poullis