CS7.

404: Digital Image Processing

Monsoon 2023: Image Segmentation

Anoop M. Namboodiri
Biometrics and Secure ID Lab, CVIT,
IIIT Hyderabad
 Three “Urges” on seeing a Picture*
1. To group proximate and similar parts of the image into
meaningful “regions”.
Called segmentation in computer vision.
2. To connect to memory to recollect previously seen
“objects”.
Called recognition in computer vision.
3. To measure quantitative aspects such as number and sizes
of objects, distances to/between them, etc.
Called reconstruction in computer vision.
*Jitendra Malik; Mysore Park, Dec. 2011
 Urge to Group

• We don’t see individual pixels (like the computer does!).

• We see groups of pixels together.
• What is the basis for “correct” grouping?
 Urge to Group

• Group similar pixels together as objects.

• Group semantically meaningful pixels together as objects.
• Is appearance similarity the same as semantic similarity?
 Segmentation
• Dividing an image into semantically meaningful regions.
 Types of Segmentation
• Classification-based
• Label pixels based on region properties
• Label each pixel based on object models
• Region-based
• Region growing and splitting
• Boundary-based
• Find edges in the image and use them as region boundary
• Motion-based
• Group pixels that have consistent motion (e.g., move in the same
direction)
 Segmentation by Pixel Classification
Two Primary Challenges:
1. How to use object / background properties to decide on
pixel label?
• e.g., Ducks are white and yellow, while background is green and
brown
2. How to ensure that regions are continuous regions?
• Avoid fragmentation of object regions
Thresholding
Decide each pixel to be part of an
object or background depending on its
gray value

Original Thresholded (T=95)

 Types of Thresholding
• Global
• A single threshold is used for the whole image
• How to determine the threshold?
• Adaptive (Local)
• Decide the threshold for every pixel depending on its neighborhood
• How to define the threshold function?
 Histogram
• A count of pixels of each graylevel (or range of graylevels) in an image

3000

2500

Pixel count
2000

1500

T = 125
1000

500

0
0 50 100 150 200 250 300

Gray-level
Grayscale Image Histogram
Thresholded Image

Original Thresholded (T=125)

 Automatic Thresholding

1. Select an initial estimate of T

2. Segment the image using T. Compute the mean gray values of the
two regions, µ1 and µ2

3. Set the new threshold T=(µ1+µ2)/2

4. Repeat 2 and 3 until T stabilizes

Assumptions: normal distribution, low noise

Thresholding in Real-World
 Extensions
• Multiple Thresholds
• Find multiple peaks and valleys in the gray level histogram

• Multi-spectral Thresholding
• In color images, one could use different thresholds for each of
the color channels

One might set all the background pixels to black, while leave the
foreground at the original value so that the information is not lost.
Multiple Thresholds

1000

900

800
97 170
700

600

500

400

300

200

100

0
0 50 100 150 200 250 300

Histogram
Original Thresholded
Multi-spectral Thresholding
1500

1000

500
140

0
0 50 100 150 200 250 300

1600

1400

1200

1000

800

600
150
400

200

0
0 50 100 150 200 250 300

0002

Original 0081

0061
Thresholded
0041

0021

130
0001

008

006

004

002

0
003 052 002 051 001 05 0

Histograms
 Adaptive Thresholding
• Adaptive thresholding changes the threshold dynamically
over the image. This can accommodate strong illumination
gradients and shadows

Original Single Threshold

http://www.dai.ed.ac.uk/HIPR2
 Adaptive Thresholding
• Set the threshold as mean of pixels (gray values) in a
neighborhood (say 7x7)

Original Adaptive Threshold

http://www.dai.ed.ac.uk/HIPR2
 Adaptive Thresholding
• Thresholding using Mean-C
Set cxc image regions of uniform graylevel to background
• Chow and Kaneko
1. Apply the mean operator (low pass filter)
2. Subtract original image from the “mean”mage
3. Threshold image in step 2
4. Invert the result

C.K. Chow and T. Kaneko Automatic Boundary Detection of the Left Ventricle
from Cineangiograms, Comp. Biomed. Res.(5), 1972, pp. 388-410.
 Adaptive Thresholding
Chow & Kaneko Thresholding:

Original Low-pass filtered Difference

Adaptive Thresholding Results

Chow & Kaneko Thresholding Mean-C (10) Thresholding

 Otsu’s Method: Insight
• Variance = A measure of region homogeneity
• Regions with high homogeneity will have a
low variance.
Otsu’s algorithm: Find the threshold that minimizes
intra-class variance.

1. Consider all possible thresholds T

2. For each threshold t in T
1. Compute the variance for Class-1
pixels (intensities < t)
2. Compute the variance for Class-2
pixels (intensities >= t) Intra-class variance
 Otsu’s Method
1. Compute histogram and probabilities of each intensity
2. Set up initial ωi(0) and μi(0) and
3. Step through all possible thresholds t = 1..tmax
1. Update ωi and μi
2. Compute σ2b(t)
4. Final threshold corresponds to the maximum σ2b(t)
5. Compute two maxima (and respective thresholds t1 and t2)
using > and >= (first and last maxima)
6. Desired threshold = (t1 + t2)/2

𝜎!2" 𝑘 = 𝑃# 𝑘 (𝑚# 𝑘 − 𝑚$ )" +𝑃" 𝑘 (𝑚" 𝑘 − 𝑚$ )"

 Otsu’s Method 𝜎!" 𝑘 = 𝑃# 𝑘 (𝑚# 𝑘 − 𝑚$ )" +𝑃" 𝑘 (𝑚" 𝑘 − 𝑚$ )"

• 𝑃! 𝑘 is probability of 𝐶! occurring
(
𝑃# 𝑘 = * 𝑝% , 𝑘 = 0,1,2, … , 𝑘
*+# %&'

𝑃" 𝑘 = * 𝑝% = 1 − 𝑃# 𝑘 , 𝑘 = 0,1,2, … , 𝑘
%&()#

• 𝑚! 𝑘 and 𝑚" 𝑘 are means of 𝐶! and 𝐶"

∑(%&' 𝑖 𝑝% ∑*+#
%&()# 𝑖 𝑝%
𝑚# 𝑘 = 𝑚" 𝑘 =
𝑃# 𝑘 𝑃" 𝑘
Otsu’s Method

T = 181
Otsu’s Method
Handling Noise
Otsu’s method: Main Limitation
 Filtering out Shadows: Homomorphic Filtering
• One can also think of the image containing a high-frequency reflectance
component and a low-frequency illumination component.
𝑓 𝑥, 𝑦 = 𝑖 𝑥, 𝑦 . 𝑟(𝑥, 𝑦)
 Homomorphic Filtering
• Separate illumination (low frequency) and reflectance (high
frequency) components and remove the low frequencies to
suppress the effects of illumination.
• Circular symmetric filter in the frequency domain:
𝐻(𝑢, 𝑣)
𝛾,

𝛾*

𝐷(𝑢, 𝑣)
Homomorphic Filtering
 Optimal Thresholding
• The graylevel histogram is approximated using a mixture of two
gaussian distributions and set the threshold to minimize the
segmentation error
3500

3000

2500

2000

1500

1000

500

0
20 40 60 80 100 120 140 160 180 200 220

Grayscale Image Histogram

 Gaussian Mixture Estimation by EM
(#$%! )'
! .
• Obj: 𝑁 𝜇! , 𝜎! = 𝑒 '(! '
,! "-

(#$%' )'
! .
• Bkg: 𝑁 𝜇" , 𝜎" = , 𝑒 '(' '
' "-

• Initialize µ1, s1, µ2, and s2.

• E-Step: Computed the expected pixel label assignments.
This could be either hard or soft assignment.
• M-Step: Computed Maximum-(Log)Likelihood estimates of the parameters:
µ1,s1,µ2,s2
• Repeat the E and M steps until convergence
Optimal Thresholding

0.035

0.03
k=2
0.025

0.02

0.015

0.01

0.005

0
0 50 100 150 200 250

Histogram with bimodal fit Thresholded (T=94)

Is Intensity Histogram Sufficient?
 Segmentation by Pixel Classification
Two Primary Challenges:
1. How to use object / background properties to decide on
pixel label?
• e.g., Ducks are white and yellow, while background is green and
brown
2. How to ensure that regions are continuous regions?
• Avoid fragmentation of object regions
Questions?