C280 Computer Vision
C280,ComputerVision
Prof.TrevorDarrell
[email protected]
Today
Administrivia
Whatisvision?
ElementaryImageformation
l
f
i
Schedule
Introductions
Prerequisites
Thiscourseisappropriateasafirstcourseforgraduate
pp p
g
studentswithaEECSbackground,whichshouldhave
preparedthestudentswiththeseessentialprerequisites:
Data
Datastructures
structures
AgoodworkingknowledgeofMATLABprogramming(orwillingness
andtimetopickitupquickly!)
Linearalgebra
i
l b
Vectorcalculus
Thecoursedoesnotassumepriorimagingexperience,
p
g g p
,
computervision,imageprocessing,orgraphics
Grading
Therewillbethreeequalcomponentstothecoursegrade
q
p
g
Fiveproblemsets
Atakehomeexam
Finalproject(includingevaluationofproposaldocument,inclass
Final project (including evaluation of proposal document in class
presentation,andfinalreport)
Inaddition,strongclassparticipationcanoffsetnegative
performanceinanyoneoftheabovecomponents.
Text
The
TheprimarycoursetextwillbeRickSzeliski
primary course text will be Rick Szeliskiss
draftComputerVision:Algorithmsand
Applications; wewilluseanonlinecopyofthe
we will use an online copy of the
June7th draft.
ThesecondarytextisForsythandPonce,
The secondary text is Forsyth and Ponce
ComputerVision:AModernApproach.
Primary Text
PrimaryText
Secondary Text
SecondaryText
Matlab
ProblemsetsandprojectswillinvolveMatlab programming(youarefree
tousealternativepackages).Matlab runsonalltheInstructionalWindows
andUNIXsystems.Instructionsandtoolkitsaredescribedin
http://inst.eecs.berkeley.edu/cgibin/pub.cgi?file=matlab.help.
CS280studentscanusetheirexistingEECSWindowsaccountsinEECS
instructionallabs,andtheycanrequestnewaccounts(fornonmajors)or
additionalaccesstoInstructionalresourcesbyfollowingtheinstructions
aboutnamedaccountsin
b
d
i
http://inst.eecs.berkeley.edu/connecting.html#accounts.Theycanlogon
remotelyandrunitonsomeofourservers:
http://inst eecs berkeley edu/connecting html#labs
http://inst.eecs.berkeley.edu/connecting.html#labs
Problem sets
Problemsets
Pset0 BasicImageManipulationinMatlab
Pset0
Basic Image Manipulation in Matlab
Pset1 FilteringandFeatures
Pset2
2 GeometryandCalibration
G
d C lib i
Pset3 Recognition
Pset4 StereoandMotion
Can discuss but must submit individual work
Candiscuss,butmustsubmitindividualwork
Take home
Takehome
Limitedtime:3days
Limited
time: 3 days
Coverseverythingthroughhandoutdate
Littleprogramming
i l
i
*Nodiscussionorcollaborationallowed*
Final project
Finalproject
Significant
Significantnovelimplementationoftechnique
novel implementation of technique
relatedtocoursecontent
Teamsof2encouraged(documentrole!)
Teams of 2 encouraged (document role!)
Orjournallengthreviewarticle(noteams)
Threecomponents:
proposaldocument(nomorethan5pages)
inclassresultspresentation(10minutes)
finalwriteup(nomorethan15pages)
p(
p g )
Class Participation
ClassParticipation
Classparticipationincludes
showingup
beingabletoarticulatekeypointsfromlast
g
yp
lecture
havingreadassignedsectionsandbeingableto
fillintheblankduringthelecture
Iwontcoldcall,butwillsolicitvolunteers
,
Stronginclassparticipationcanoffsetpoor
performance in one of the other grade
performanceinoneoftheothergrade
components.
Course goals.(broadly speaking)
Coursegoals.(broadlyspeaking)
principlesofimageformation
principles
of image formation
convolutionandimagepyramids
l lf
localfeatureanalysis
l i
multiviewgeometry
imagewarpingandstitching
structure from motion
structurefrommotion
visualrecognition
i
imagebasedrendering
b d
d i
Whatiscomputervision?
Done?
Whatiscomputervision?
Automaticunderstandingofimagesandvideo
Computing
Computingpropertiesofthe3Dworldfromvisualdata
properties of the 3D world from visual data
(measurement)
Algorithms
Algorithmsandrepresentationstoallowamachineto
and representations to allow a machine to
recognizeobjects,people,scenes,andactivities.
(perceptionandinterpretation)
Visionformeasurement
Realtimestereo
Structurefrommotion
Multiviewstereofor
communityphotocollections
NASAMarsRover
Pollefeysetal.
Goeseleetal.
Slidecredit:L.Lazebnik
Visionforperception,interpretation
amusementpark
t
k
sky
TheWicked
The
Wicked
Twister
C d P i t
CedarPoint
Ferris
wheel
ride
LakeErie
ride
12E
water
ride
id
tree
tree
Objects
Activities
Scenes
Locations
Text/writing
Faces
Gestures
Motions
Emotions
peoplewaitinginline
peop
e a g
e
peoplesittingonride
umbrellas
tree
deck
maxair
carousel
bench
tree
pedestrians
Relateddisciplines
Artificial
intelligence
Graphics
p
IImage
processing
Computer
vision
Al ith
Algorithms
Machine
learning
Cognitive
C
iti
science
Visionandgraphics
Images
Vision
Model
Graphics
Inverse problems: analysis and synthesis
Inverseproblems:analysisandsynthesis.
Whyvision?
Asimagesourcesmultiply,sodoapplications
Relievehumansofboring,easytasks
Relieve humans of boring easy tasks
Enhancehumanabilities:humancomputer
interaction visualization
interaction,visualization
Perceptionforrobotics/autonomousagents
Organizeandgiveaccesstovisualcontent
Organize and give access to visual content
Whyvision?
Imagesandvideoareeverywhere!
Images and video are everywhere!
Personalphotoalbums
Surveillanceandsecurity
Movies,news,sports
Medicalandscientificimages
Slidecredit;L.Lazebnik
Again, what is computer vision?
Again,whatiscomputervision?
Mathematics
Mathematicsofgeometryofimage
of geometry of image
formation?
Statisticsofthenaturalworld?
Statistics of the natural world?
Modelsforneuroscience?
Engineeringmethodsformatchingimages?
ScienceFiction?
Vision Demo?
VisionDemo?
Terminator 2
werenotquitethereyet.
Everypicturetellsastory
Goalofcomputervisionistowritecomputerprograms
thatcaninterpretimages
Cancomputersmatch(orbeat)humanvision?
Yesandno(butmostlyno!)
(
)
humansaremuchbetterathardthings
computerscanbebetterateasythings
Humanperceptionhasitsshortcomings
p
p
g
Sinha and Poggio, Nature, 1996
Copyright A.Kitaoka 2003
Current state of the art
Currentstateoftheart
The
Thenextslidesshowsomeexamplesofwhat
next slides show some examples of what
currentvisionsystemscando
Earthviewers(3Dmodeling)
Image from Microsofts Virtual Earth
((see also: Google
g Earth))
Photosynth
http://labs live com/photosynth/
http://labs.live.com/photosynth/
Based on Photo Tourism technology developed
by Noah Snavely,
Snavely Steve Seitz
Seitz, and Rick Szeliski
PhotoTourismoverview
Scene
reconstruction
Inputphotographs
PhotoExplorer
Relativecamerapositions
andorientations
Pointcloud
Sparsecorrespondence
Systemforinteractivebrowsingandexploringlargecollectionsofphotosofascene.
Computesviewpointofeachphotoaswellasasparse3dmodelofthescene.
Photo Tourism overview
PhotoTourismoverview
Optical character recognition (OCR)
Opticalcharacterrecognition(OCR)
Technology to convert scanned docs to text
If you have a scanner, it probably came with OCR software
Digit recognition, AT&T labs
p
y
http://www.research.att.com/~yann/
License plate readers
http://en.wikipedia.org/wiki/Automatic_number_plate_recognition
Face detection
Facedetection
Manynewdigitalcamerasnowdetectfaces
Canon,Sony,Fuji,
,
y, j ,
Smile detection?
Smiledetection?
SonyCybershotT70DigitalStillCamera
Objectrecognition(insupermarkets)
LaneHawkbyEvolutionRobotics
Asmartcameraisflushmountedinthecheckoutlane,continuouslywatching
foritems.Whenanitemisdetectedandrecognized,thecashierverifiesthe
q
quantityofitemsthatwerefoundunderthebasket,andcontinuestoclosethe
y
transaction.Theitemcanremainunderthebasket,andwithLaneHawk,youare
assuredtogetpaidforit
Facerecognition
Who is she?
Visionbasedbiometrics
How the Afghan Girl was Identified by Her Iris Patterns Read the story
Loginwithoutapassword
Fingerprint scanners on
many new laptops
laptops,
other devices
Face recognition systems now
beginning to appear more widely
htt //
http://www.sensiblevision.com/
ibl i i
/
Objectrecognition(inmobile
phones)
Thisisbecomingreal:
This is becoming real:
MicrosoftResearch
Point&Find,Nokia
Point & Find Nokia
SnapTell.com(nowamazon)
Snaptell
http://snaptell.com/demos/DemoLarge.htm
NokiaPointandTell
http://conversations.nokia.com/home/2008/09/pointandfin1.html
Specialeffects:shapecapture
The Matrix movies, ESC Entertainment, XYZRGB, NRC
Specialeffects:motioncapture
Pirates of the Carribean, Industrial Light and Magic
Click here for interactive demo
Sports
Sportvision first down line
Nice explanation on www.howstuffworks.com
www howstuffworks com
Smartcars
Mobileye
VisionsystemscurrentlyinhighendBMW,GM,Volvomodels
By2010:70%ofcarmanufacturers.
By 2010: 70% of car manufacturers
Videodemo
Slide content courtesy of Amnon Shashua
Smartcars
Mobileye
VisionsystemscurrentlyinhighendBMW,GM,Volvomodels
By2010:70%ofcarmanufacturers.
By 2010: 70% of car manufacturers
Videodemo
Slide content courtesy of Amnon Shashua
Visionbasedinteraction(and
games)
Digimask: put your face on a 3D avatar.
Nintendo Wii has camera-based IR
tracking built in. See Lee
Lees
s work at
CMU on clever tricks on using it to
create a multi-touch display!
Game turns moviegoers into Human Joysticks, CNET
Camera tracking a crowd, based on this work.
Vision in space
Visioninspace
NASA'S Mars Exploration Rover Spirit captured this westward view from atop
a low plateau where Spirit spent the closing months of 2007.
Vision systems
y
((JPL)) used for several tasks
Panorama stitching
3D terrain modeling
Obstacle detection
detection, position tracking
For more, read Computer Vision on Mars by Matthies et al.
Robotics
NASAs Mars Spirit Rover
http://en.wikipedia.org/wiki/Spirit_rover
http://www.robocup.org/
Medical imaging
Medicalimaging
3D imaging
MRI, CT
Image guided surgery
Grimson et al., MIT
Currentstateoftheart
Youjustsawexamplesofcurrentsystems.
Manyofthesearelessthan5yearsold
Many of these are less than 5 years old
Thisisaveryactiveresearcharea,andrapidlychanging
Manynewappsinthenext5years
Tolearnmoreaboutvisionapplicationsandcompanies
DavidLowemaintainsanexcellentoverviewofvision
companies
http://www.cs.ubc.ca/spider/lowe/vision.html
Syllabus / Schedule (see handout)
Syllabus/Schedule(seehandout)
http://tinyurl.com/UCBC280CAL
ImageFormation
g
ImageFiltering
Pyramids&Regularization
FeatureDetectionand
Matching
Geometric Alignment
GeometricAlignment
Calibration
Geometric Image Stitching
GeometricImageStitching
PhotometricImage
Stitching
Recognition
Stereo
OpticFlow
DenseMotionModels
ShapefromSilhouettes
ShapefromShadingand
Texture
SurfaceModels
Segmentation
SFM
IBR&HDR
And now, who are you?
Andnow,whoareyou?
And
Andwhatdoyouexpecttogetoutofthis
what do you expect to get out of this
class?
Previousexperienceinvision,learning,
Previous experience in vision learning
graphics?
Researchagenda?
R
h
d ?
(Projecttopics?)
Letssgetstarted:Imageformation
Let
get started: Image formation
How
Howareobjectsintheworldcapturedinan
are objects in the world captured in an
image?
Physicalparametersofimage
formation
Geometric
Typeofprojection
Camerapose
p
Optical
Sensorslenstype
focallength,fieldofview,aperture
Photometric
Type,direction,intensityoflightreachingsensor
T
di ti
i t it f li ht
hi
Surfacesreflectanceproperties
Imageformation
Letsdesignacamera
Idea1:putapieceoffilminfrontofanobject
Idea 1 p t a piece of film in front of an object
Dowegetareasonableimage?
SlidebySteveSeitz
Pinholecamera
Addabarriertoblockoffmostoftherays
Thisreducesblurring
Theopeningisknownastheaperture
Howdoesthistransformtheimage?
H d
hi
f
h i
?
SlidebySteveSeitz
Pinholecamera
Pinholecameraisasimplemodeltoapproximate
imaging process perspective projection
imagingprocess,perspectiveprojection.
Image
plane
Virtual
image
g
pinhole
Ifwetreatpinholeasapoint,onlyonerayfrom
any given point can enter the camera
anygivenpointcanenterthecamera.
FigfromForsythandPonce
Cameraobscura
IInLatin,means
L ti
darkroom
"ReinerusGemmaFrisius,observedaneclipseofthesunatLouvainonJanuary24,1544,
andlaterheusedthisillustrationoftheeventinhisbookDeRadioAstronomicaet
Geometrica,1545.Itisthoughttobethefirstpublishedillustrationofacameraobscura..."
Hammond,JohnH.,TheCameraObscura,AChronicle
http://www.acmi.net.au/AIC/CAMERA_OBSCURA.html
Cameraobscura
JettyatMargateEngland,1898.
Anattractioninthelate19th
century
Around1870s
http://brightbytes.com/cosite/collection2.html
AdaptedfromR.Duraiswami
Cameraobscuraathome
Sketchfromhttp://www.funsci.com/fun3_en/sky/sky.htm
http://blog.makezine.com/archive/2006/02/how_to_room_siz
ed_camera_obscu.html
Perspectiveeffects
Perspectiveeffects
Farawayobjectsappearsmaller
ForsythandPonce
Perspectiveeffects
Perspectiveeffects
Parallellinesinthesceneintersectintheimage
Convergeinimageonhorizonline
Converge in image on horizon line
Imageplane
(virtual)
pinhole
Scene
Slide Credits
SlideCredits
Slides14
Slides 1421
21,55
5566:
66:KristenGrauman
Kristen Grauman
Slides2340,4352:SteveSeitz
andothers,asmarked
d h
k d
Next time
Nexttime
Continue
ContinuewithImageFormation
with Image Formation
Readingsfortoday:Szeliski,Ch.1
Readingsfornextlecture:Szeliski
di
f
l
S li ki 2
2.12.3.1,
23
ForsythandPonce1.1,1.4(optional).
Pset 0releasedtomorrow,duefollowing
Friday
