Introduction to Vision and Perception

Sensors and Devices

Mirza Farhan Bin Tarek
Lecturer, Dept. of Computer Science and Engineering
BRAC University
1
 Sensors
• Sensors are physical devices that provides information about the world. This
process is called Sensing or Perception.
• Examples:
• Accelerometer
• Gyroscope
• Camera
• Temperature and Humidity Sensor
• Ultrasonic Sensor
• IR Sensor
• LiDAR Sensor

2
Types of Digital Imaging Sensors

CCD Sensor CMOS Sensor

3
 Charge-coupled Device (CCD)
• First developed in 1969 by Willard Boyle and George E. Smith at AT&T Bell
Labs in USA
• A charge-coupled device or CCD is a highly-sensitive photon detector
which can detect light between the wavelength of 400nm to 1000nm.
• The chip is an array of light sensitive small areas called pixels. (There could be
several millions of pixels in a single chip)

4
 Charge-coupled Device (CCD): How it works??
1. Charge Collection
When photons strike the pixels, they will release a number of electrons
(proportional to the intensity of the scene at the pixel) which are then captured
by electric fields and retained at the pixel.
2. Readout
• Each row of pixels or CCD Columns are then shifted using vertical conveyor
belts(Implemented using shift registers) to the final row (Readout Register)
• In the read-out register, the electrodes are arranged in such a way that the
charge can be transferred horizontally along the readout register.
• At the time of read-out, the collected charge is transferred one at a time to
an output amplifier which converts the charge to voltage.
3. Image Construction
Finally, using an ADC (Analog to Digital Converter) and other electronic
devices in the camera, the voltage is converted to digital form. This process
continuously runs until all pixels' charge have been amplified and output.

5
 Working Principle of CMOS Sensors
• CMOS sensors are built with fabrication
tech similar to ICs.
• Hence, amplifiers, converters can be
integrated into the circuit.
• Compared to CCD, the charge to
voltage conversion and the
amplification is performed inside the
pixels. So the processing speed of the
CMOS sensors are higher than CCD.
• The pixels are finally read row by row
using pixel select switches

https://www.youtube.com/watch?v=FKJFIzDfUNE&ab_channel=ALLABOUTELECTRONICS 6
Comparison of CCD and CMOS Sensors

CCD CMOS
Circuit Complexity Complex Compact

Power Supply High Low

Processing Speed Low High

Photon Sensitivity High Low

7
 Ultrasonic Sensor
• Converts electrical energy into acoustic wave,
which is an ultrasonic wave travelling at above
18kHz frequency.
• HC-SR04 operates at 40kHz
• a microcontroller is used for communication with HC-SR04 Ultrasonic Sensor
an ultrasonic sensor. (Source: Digikey)

• Applications
• Measure wind speed and direction Ultrasonic sensor measuring
• Navigation of UAV height during drone’s flight.
(Source: RadioLink)
• Measure tank depth

8
 Ultrasonic Sensor: How It Works??
1. a microcontroller is used for communication
with an ultrasonic sensor.
2. To begin measuring the distance, the
microcontroller sends a trigger signal to the
ultrasonic sensor. The duty cycle of this trigger
signal is 10µS for the HC-SR04 ultrasonic sensor.
3. When triggered, the ultrasonic sensor generates
eight acoustic (ultrasonic) wave bursts and
initiates a time counter.
4. As soon as the reflected (echo) signal is received,
the timer stops. The output of the ultrasonic
sensor is a high pulse with the same duration as
the time difference between transmitted
ultrasonic bursts and the received echo signal.
9
 Ultrasonic Sensor: How It Works??
𝑒𝑐ℎ𝑜 𝑝𝑢𝑙𝑠𝑒 𝑤𝑖𝑑𝑡ℎ (𝜇𝑆)
1.
Distance (cm) =
58
𝑒𝑐ℎ𝑜 𝑝𝑢𝑙𝑠𝑒 𝑤𝑖𝑑𝑡ℎ (𝜇𝑆)
2. Distance (inch) =
148
𝑡𝑖𝑚𝑒 𝑡𝑎𝑘𝑒𝑛 ∗𝑠𝑝𝑒𝑒𝑑 𝑜𝑓 𝑠𝑜𝑢𝑛𝑑
3. Distance =
2

10
 Infrared Sensors
• An electronic device that can detect and
measure infrared (IR) radiation in the
environment
• Anything that emits heat (everything that
has a temperature above around five
degrees Kelvin) gives off infrared radiation
• Applications
• TV Remote
• Motion Sensing
• Proximity Sensing
• Gas and Flame Detection and Monitoring
11
 Infrared Sensors: Types
• Passive IR Sensor (PIR)
• Does not have any IR sources but can detect energy emitted from the obstacles or objects
in the field of view
• Two types: Thermal and Quantum
• Thermal IR sensor use IR energy as the source of heat and are not dependent on wavelength.
Examples include thermocouples, bolometers etc.
• Quantum IR sensors, on the other hand, have higher detection capability and its photosensitivity
is wavelength dependent.

HC-SR501 PIR Motion Sensor Module

12
 Infrared Sensors: Types
• Active IR Sensor
• They can emit IR energy using infrared LED or infrared laser diode and receive the reflected
energy using an infrared detector e.g. photodiode, phototransistors

IR Emitter LED IR Receiver Module

13
Infrared Sensors: How Active Sensing Works

14
 LiDAR
• Stands for Light Detection And
Ranging
• Similar to Radar and Sonar
• Uses laser instead of radio wave or
sound wave
• Applications include:
• High Resolution Mapping
• Laser Guidance Systems
• Control and Navigation of Robots and
Autonomous Cars

15
 Components of a LiDAR Scanner
• Laser
• 600-1000 nm Lasers are the most prevalent for non-scientific usage such as
agricultural mapping
• One common alternative, 1550 nm lasers
• eye-safe at relatively high power levels
• wavelength is not strongly absorbed by the eye,
• However, detector technology is less advanced
• and so used at longer ranges with lower accuracies.
• Laser can be emitted using a phased antenna array

16
 Components of a LiDAR Scanner
• Scanners and Optics
• Different scanners and mirrors are used for changing detection resolution or range.
• Beam splitters can be used for capturing the return signal
• Photodetector and receiver electronics
• Solid State Photodetectors
• Photomultipliers (They convert photon into electrical signal)
• Position and navigation systems
• GPS or IMU (Inertial Measurement Unit)
• Used to determine the position and orientation of sensors especially when the LiDAR unit is on
airplane or satellite
• Sensors for signal detection and sending. They can be built using hybrid CCD/CMOS
fabrication technology.

17
 Basic Working Principles
• Each time the laser is pulsed:
• Laser generates an optical pulse(Up to
200,000+ pulses/second)
• After reflecting off an object, the pulse
returns to the receiver sensor
• High-speed counter measures the time of
flight from the start pulse to the return
pulse
• Time measurement is converted to a
distance
• An onboard computer records each laser’s
reflection point, translating this rapidly
updating “point cloud” into an animated 3D
representation of its surroundings.

18
 Point Cloud Identification

https://cutt.ly/ql5wZX2
19
 Point Cloud Identification

https://cutt.ly/ql5wZX2
20
 Point Cloud Identification

https://cutt.ly/ql5wZX2
21
Autonomous Navigation using Lidar

22
• To see LiDAR in action
• https://www.youtube.com/watch?v=JC94Y063x58&ab_channel=GeospatialWorld
• https://www.youtube.com/watch?v=EYbhNSUnIdU&ab_channel=NEONScience

23
 Choosing The Proximity and Range Sensors
• Depends on applications!!
• Proximity Sensors
• 1-D range finders
• Distance is measured from observer to object in a straight line. Used in obstacle detection and
avoidance
• IR sensors, Ultrasonic sensors etc.
• 2-D range finders
• Mainly includes laser scanners that can scan 2D plane and calculate the range measurement
• 2D static laser scanners can scan in limited angles
• 2D rotational laser scanners can scan in 360 degrees
• Used in object detection and avoidance, navigation

24
 Choosing The Proximity and Range Sensors
• 3-D Static Range Finders and RGB-D cameras
• Have to be mounted on robots
• RGB-D cameras use IR projection and reflection to calculate 3d range
• Can give 3D range within a specific angle

3-D
1-D 2-D

Courtesy: https://robocademy.com/2020/04/10/how-to-choose-a-sensor-for-your-robot/
25
Introduction to Imaging Devices

26
Digital Imaging Devices

27
How Digital Cameras Work

1. This is the insides of a webcam. We can see that a lens is planted

directly onto an electronic chip

2. This is the image sensor (CCD or CMOS).You

should observe that the chip has a dark green part
in the middle which is the light sensitive part.
The outer parts are connections with the main

3. Closeup look at the sensor

28
 How Digital Cameras Work (Contd)
• Step 1: Light enters the camera
through the aperture
• Step 2: The image sensor splits the
image into a number of pixels
(could be millions, depends on the
size of the sensor)
• Step 3: The sensor, after measuring
the brightness and the color of the
image, then stores the color and
brightness info in the memory card
as binary numbers.

29
 Choosing The Right Camera Module
• Look at the camera interfacing available for your development board
• MIPI CSI – 2 (MIPI Camera Serial Interface 2)
• Default camera interface in RPi and NJN. Has high performance, cost-effective and easy-to-use
• On the RPi, the MIPI CSI-2 interface takes the form of two physical connectors: 15-pin connector
on standard Raspberry Pi models and a 22-pin connector on Pi zero and the Jetson Nano uses a
15-pin connector
• USB Camera
• Typically driver free (USB Video Class Cameras or UVC Cameras)
• Mostly used type is webcam. They can be plug and play
• Other UVC cameras available for embedded uses. Arducam offers several UVC board cameras as
an implement for our current embedded MIPI camera modules. These cameras have more case-
specific features like Wide Dynamic Range (WDR), low light enhancement and so on

https://www.arducam.com/choose-camera-modules-raspberry-pi-jetson-nano-guide/ 30
 Choosing The Right Camera Module
• Monochrome or Color Sensor
• Color Sensor with Rolling Shutter
• Color sensors are image sensors with color filters
• Color sensors typically implement Rolling shutter. This is a method of capturing image
where the image sensor is gradually exposed from top to bottom
• Cheaper to build and easy to add more pixels
• Cannot capture scenes with fast moving objects properly. Causes distortion in these
cases.
• Monochrome Sensor with Global Shutter
• a global shutter camera exposes all pixels at the same time. The shutter opens and
closes in a short period. Therefore, it can take better pictures of fast-moving objects
without blurring the details.
• For taking videos in higher frame rates and fast-moving objects, global
shutter monochrome cameras can be good option. For capturing detailed still
photos, high-resolution rolling shutter color cameras are better.
• Have to choose between manual focus or autofocus camera.

https://www.arducam.com/choose-camera-modules-raspberry-pi-jetson-nano-guide/ 31
 Choosing The Right Camera Module
• KNOW YOUR APPLICATIONS AND REQUIREMENTS
• NEED TO CAPTURE CRISP DETAILED IMAGES?
• OR MOVING IMAGES?
• HOW FAR IS THE OBJECT?? HOW LARGE IS THE AREA OF SHOOTING?
• LIGTING CONDITION??

https://www.arducam.com/choose-camera-modules-raspberry-pi-jetson-nano-guide/ 32
33