See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/305906269

Survey on sketch based image retrieval methods

Conference Paper · March 2016

DOI: 10.1109/ICCPCT.2016.7530359

CITATIONS READS

9 906

2 authors, including:

Indu Murali
Indian Institute of Information Technology and Management - Kerala
3 PUBLICATIONS   27 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Cyber Resilience View project

All content following this page was uploaded by Indu Murali on 24 January 2018.

The user has requested enhancement of the downloaded file.

 2016 International Conference on Circuit, Power and Computing Technologies [ICCPCT]

Survey on sketch based image retrieval methods

M Indu Kavitha K V
M.Tech Student Assistant Professor
Department of Computer Science and Engineering Department of Computer Science and Engineering
SCT College of Engineering SCT College of Engineering
Trivandrum, India Trivandrum, India
[email protected] [email protected]

Abstract— In many applications critical roles are played by report based on image analysis. There are various purposes for
face sketch-photo synthesis, such as law enforcement and digital image processing which can be categorised into; visualization
entertainment. Several image based retrieval systems are text -Observe the objects that are not visible, image sharpen and
based, content based and sketch based. Image retrieval methods restoring - to create a better image, image retrieval - lookout
which use sketch content as input are referred to as Sketch Based for the image of interest, measurement of pattern – various
Image Retrieval systems. In several ways an image can be objects in an image is measured, image recognition –
retrieved from the database using user queries as input. One of distinguish the objects in an image.
the efficient methods and popular methods for retrieval is as
Sketch Based Image Retrieval which is not necessary to have a Due to the various advancements in image technology in
high skill to draw the query sketch. Paper reviews the various the field of computer science, image retrieval has become a
sketch based image retrieval methods used in image processing popular area. Retrieval systems of images were used for
and a comparison of all these methods is also done. The retrieval retrieving, searching and browsing images from large
system using sketches can be effective and essential in our day to databases of digital images [2]. A specialized data search is
day life such as Medical diagnosis, digital library, search engines, used to find images. To search for images, a user may provide
crime prevention, photo sharing sites, geographical information, query terms such as image file/link, keyword or click on some
and sensing remote systems. image, and the system will return images "similar" to the
query. The search criteria similarities used could be meta tags,
Keywords— sketch-photo; TBIR; CBIR; SBIR; patch;
dictionaries; sparse representation.
colour distribution in images, region/shape attributes, etc.
II. OBJECTIVE OF SKETCH BASED IMAGE RETRIEVAL METHODS
I. INTRODUCTION
Face is a prime part of human body through which an
Manipulation of digital images through a digital computer individual can be identified. It plays a major role in bearing
is familiarized as digital image processing [1]. Its a subfield of identity and emotions of an individual. Every individual has
signals and systems but focus primarily on images. DIP distinctive facial features other than identical twins.
focuses on developing a system to perform processing on an
image. The input is a digital image and that image is processed Major applications of face recognition include security
using efficient algorithms to produce processed output image. systems, verification of credit card, culprit identification etc.
example: Adobe Photoshop. On applying signal processing Automatic retrieval of photos of suspects from police culprit
methods, image is treated as a two dimensional signal by the database help them to narrow down potential suspects quickly.
image processing system. A core research area for image In cases where photo image of the suspect is not available, the
processing comes in fields of computer science, business, and best substitute is to draw a sketch based on the recall of an
engineering disciplines. eyewitness. Thus automatically searching through a photo
database using a query sketch drawing is very useful. Thus
Data is represented in two dimension. Digital image helping the police to locate a group of potential suspects, and
contains pixels arranged in rectangular array with a certain also help the witness and artist to revise the sketch drawing of
width & height. For representing brightness of an image, the the culprit interactively based on the similar photos retrieved.
content of each pixel is represented by one or more bits. Image It can also be used in other fields where photo is not available
processing can be used to solve identification problems, such but the details of the photo can be described. This method
as forensic medicine or creating weather maps from satellite remarkably reduces the difference between photo and sketch.
such as refining a picture from video source.
For more than twenty years human face recognition has
Image processing basically has three steps. First the image been studying. Since faces are complex, multi-dimensional
is imported with optical scanner or using digital photography. visual stimuli developing a computational model of face
Analyzing and manipulating the image includes compression recognition is very difficult. Face recognition is a computer
of data, enhancement of image and spotting the patterns that vision task, in which many early vision techniques can be
are not visible to human eyes like satellite photographs. involved.
Finally the output digital image can be an altered image or

978-1-5090-1277-0/16/$31.00 ©2016 IEEE

 2016 International Conference on Circuit, Power and Computing Technologies [ICCPCT]
III. IMAGE RETRIEVAL SYSTEMS C. Sketch based image retrieval (SBIR) system
Since early 90’s image retrieval has become a very It is a relevant means of querying large image databases
[2]. Researches mainly focus on solving the gap between
attractive research area. Both commercial and research image
retrieval systems have been built till date. Most image sketch and problems relating to image matching. In large scale
retrieval systems support any one of the following options: database visual features SBIR technique is used to find query
image. Using different types of descriptors many researchers
A. Text-based image retrieval (TBIR) system tried to extract the features of images.
It uses traditional database techniques to manage images.
Using the text associated with an image determines the image
IV. REVIEW OF VARIOUS SKETCH BASED IMAGE RETRIEVAL
content. Example: Google, Yahoo Image Search engines. The
METHODS
search engines are fast and robust but at times fail to retrieve
relevant images [3]. There are various disadvantages such as For most of the surveillance purposes such as search for
for large database manual annotation is impossible, Polysemy wanted culprits, suspected terrorists, and missing children;
problem (more than one object can be referred by the same face retrieval is a widely used biometric technique. Face is the
word) and also nearby text also will not describe the image. key focus in the society which plays a crucial role in bearing
an individual’s identity and emotions. Facial recognition and
B. Content based image retrieval (CBIR) system retrieval system uses human faces to attempt and identify an
This technique uses user’s interest to search images from individual or verify a person’s claimed identity. It’s a form of
large image databases. It is an alternative to TBIR. To computer vision. Reduce the difference between two
represent and index an image, various visual contents of an modalities is the key objective of this method. For making the
recognition process easier, photo and sketch should be made
image is used such as color, shape, texture, and spatial layout.
to same mode. Various applications in the field of multimedia
From the database images visual contents are extracted and
management, security, smart card, banking etc.
portrayed using multi-dimensional feature vectors as
illustrated in Fig.1 [4]. A feature database is formed by A. Principal component analysis (PCA)
different feature vectors of the images in the database. To In this architecture for the training phase conversion of
retrieve images, users provide example images or sketched database photo into sketch by using PCA algorithm is done
figures. As an internal representation, the system changes primarily. That is faces from database is selected one by one
these examples into its feature vectors. Using an indexing and face image is extracted by removing their background
scheme the similarities or distances between the feature details. Then face image is converted into sketch, by using
vectors of the sketch and the images in the database are PCA sketch and is then converted into Eigen Face by
calculated and performed the retrieval. For searching an image considering the intensity of image [6].
from the large database a systematic way is provided by
indexing scheme. To produce more meaningful retrieval In the same way, average of all Eigen face is found out for
the entire images in the database. Now convert artist sketch
result, recent retrieval systems add user’s relevance feedback
into Eigen face & mix it with average of all faces from
to modify the retrieval process. databases. Then by comparing, two faces with maximum
match will give the output. Here sketch is converted into
Eigen face that is then compared with the sketch generated
from photo which is obtained from the criminal record of
police. The same is illustrated in Fig. 2.
The face region is divided sequentially into overlapping
patches. During synthesis of sketch, a photo patch from the
face to be synthesized, a similar photo patch from the training
set is found. Using its corresponding sketch patch in the
training set, estimates the sketch patch to be synthesized.
Assumes if two photo patches are similar, their sketch patches
also will be similar.
In the initial PCA method it is assumed that the source
Fig.1: Content based image retrieval system.
sketch and the target face photo share the same projection
coefficient. Using PCA procedure coefficients are obtained.
CBIR system basically has four main sections in system
There are basically two types of photo input: target photo and
realization: collection of data, builds up feature database, training photos .The coefficients are obtained by projecting the
search in the database, orderly arrangement and deals with the input photos on the target photos. The target sketch is obtained
result of retrieval [5]. Various features of CBIR system by the linear combination of the training weights provided by
include automatic extraction of visual features such as color, the projection coefficients
shape information and texture of images.
 2016 International Conference on Circuit, Power and Computing Technologies [ICCPCT]
It is widely used for nonlinear dimension reduction of high-
dimensional data and image analysis [10], [11], [12]. The
fundamental idea of LLE is to compute neighbour-preserving
mapping between a high-dimensional original data space and a
low-dimensional feature space, on the basis of simple
geometric intuition that each data and its neighbours lie on or
close to a locally patch.
A patch-based strategy is used and dividing the photo and
sketch images into N small overlapping image patches. For
each photo image patch first fit it with its K nearest
neighbours from training samples and calculate the
Fig.2: Face recognition with eigen face. reconstruction weights. Then its corresponding sketch patch
can be estimated from training sketch samples by preserving
This is further improved by separating shape from the the local geometry. Face sketch recognition is used to measure
texture. Here the target sketch is obtained by fusing the shape similarities between probe sketch and pseudo-sketches from
and texture using the Eigen transformation. photos. In sketch recognition, for better describing nonlinear
B. Multiscale Markov random field (MRF) variations caused by distortions and blurs in the real sketches
drawn by artists and pseudo-sketches, KNDA based nonlinear
To synthesize local face structures in different scales this discriminative classifier is used. A nonlinear version of LDA
approach is used [7]. Here training set consists of photo-sketch is KNDA. It is widely used in pattern recognition [13], [14],
pairs. The faces to be studied should be in frontal pose, with a [15].
normal lighting, neutral expression and without occlusion. As
learning global face structure is complicated, simple structured When comparing to PCA the experimental results show
local patches are considered. A Face region is sequentially that the recognition accuracy and performance of KNDA is
divided into overlapping patches. During sketch synthesis, to better. PCA has the worst performance, but it is optimal for
process a photo patch of the face, similar photo patch from the reconstruction but not for discriminating one class from the
training set is need to be found out and its corresponding other.
sketch patch is generated. The assumption is, if two patches D. Multi dictionary sparse representation framework
are similar then their corresponding sketch patch will also be
similar. In addition, smoothness is required between adjacent Sketches or photos generated using LLE method produces
patches of the synthesized sketch. The scale of face structure lower definition and blurred outputs that consequently reduces
is decided by its patch size. To learn face structures at the visual quality and recognition rate across the
different scales MRF model is widely used. All face photos in heterogeneous images. So in order to improve the quality of
the gallery are transformed to sketches using sketch synthesis the synthesized images a novel multi dictionary based sparse
algorithms and to the synthesized sketches a query sketch is based sketch-photo synthesis model is constructed.
matched. A query sketch is transformed to a photo and the
This method also consists of N sketch-photo pairs as
synthesized photo is then matched with the existing real
training set. Since the method works at several patch level,
photos in the gallery.
each sketch and photo is partitioned into even patches with
This approach is based on local patches. It does not require some overlapping. Using clustering method such as k means
a face photo constructed by PCA from training set. It can approach, these patches are clustered into classes. The sketch
construct more complicated face structures such as hair. A patch intensity subtracting the mean intensity of this patch acts
sketch patch not only requires information from neighbouring as the feature for clustering, this determines the cluster to
patches but also from patches that are far away by belief which that patch has to be clustered. K nearest neighbours is
propagation [8]. It can synthesize sketch patches over different found with its corresponding optimized weights [16]. Fuse
scales. Here sketch synthesis algorithm may not work properly these synthesized patches into an initial sketch with
if the input photo is taken under different conditions than the overlapping areas averaged. Here two feature dictionary of
photos in the training set. sketch path and photo patch is there. A joint learning strategy
is applied to each cluster for two dictionaries [17]. For any
C. Local linear embedding (LLE) testing photo patch a sparse representation coefficient vector is
There are various problem encountering if a photo image is computed by projecting the feature of patch to the photo patch
recovered from a sketch [9]. For a sketch image often main feature dictionary learned from the cluster. The features are
facial features, and many detailed features are ignored. A used in the initial estimate stage and image enhancement stage
different approach is to generate a pseudo-sketch from a photo was different, clustering algorithm is used twice for two stages
image which is being used in this scheme. To setting up a which improve the results.
mapping relation between a photo and a sketch the method of From the experimental results it has got that, this method
pseudo-sketch generation is equivalent. Focus of this method has higher definition and richer detail information. For a
is on sketches of plain style. It is inevitable to have some sketch of size 64x64 it takes about 2.5minutes to synthesis a
distortions when sketches are drawn by artists. Additionally, sketch running on a 3GHz CPU computer. Sketch-photo
the weight combination of local nearest neighbours brings recognition rate is better when compared to other methods.
some blurs to the pseudo-sketches. LLE is a learning method.
 2016 International Conference on Circuit, Power and Computing Technologies [ICCPCT]
The mean opinion score (MOS) in the field of multimedia is approaches to come up with an efficient retrieval system
also higher than the other methods. together with the limitations or challenges.
V. COMPARISON OF VARIOUS SKETCH BASED IMAGE REFERENCES
RETRIEVAL METHODS [1] http://wwwelec.inaoep.mx/~jmram/Digital_Image_Processing_GONZA
LEZ.pdf.
[2] Huda Abdulaali Abdulbaqi, Ghazali Sulong,Soukaena Hassan Hashem,
In this review paper various image retrieval methods based “A sketch based image retrieval: a review of literature,” Journal of
on query sketch is discussed and a comparison of these Theoretical and Applied Information Technology, 2014.
methods are discussed in TABLEI. [3] GulfishanFirdose Ahmed, RajuBarskar,”A Study on Different Image
Retrieval Techniques in Image Processing,” International Journal of soft
Computing and Engineering (IJSCE), ISSN: 2231-2307,Volume-1,
TABLE I. COMPARISON OF VARIOUS RETRIEVAL METHODS Issue-4, September 2011.
[4] Dr. Fuhui Long, Dr. Hongjiang Zhang and Prof. David Dagan Feng
Method Feature/ Advantages Disadvantages “Fundamentals of content-based image retrieval” Springer,2003.
Methodology [5] NidhiSinghai, Prof. Shishir K. Shandilya,”A Survey On: Content Based
PCA Mapping between -Simple and Very sensitive to Image Retrieval Systems,” International Journal of Computer
a photo and its efficient scale, so a low- Applications (0975 – 8887) Volume 4 – No.2, July 2010.
corresponding -Low dimensional level pre- [6] Amit R. Sharma and Prakash. R. Devale, “An application to human face
sketch via subspace processing is photo-sketch synthesis and recognition,” International Journal of
Linear representation. needed. Advances in Engineering & Technology, May 2012.
transformation. [7] X. Wang and X. Tang, “Face photo-sketch synthesis and recognition,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 11, pp. 1955–1967,
Multiscale The relationship Boundaries can Unsupervised Nov. 2009.
MRF of adjacent local be identified parameter
patches and the easily. estimation of the [8] J.S. Yedidia, W.T. Freeman, and Y. Weiss, “Understanding Belief
face structures at MRF is difficult. Propagation and Its Generalizations,” Exploring Artificial Intelligence in
different scales. the New Millennium, Morgan Kaufmann, 2003.
[9] Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, “A nonlinear approach for
LLE Nonlinear process Relationship Need more face sketch synthesis and recognition,” in Proc. IEEE Comput. Soc.
of face sketch between face training samples. Conf. Comput. Vis. Pattern Recognit., Jun. 2005, pp. 1005–1010.
synthesis. photos and face- [10] S. T. Roweis and L. K. Saul, “Nonlinear dimensionality reduction by
sketches can be locally linear embedding”, Science, 290(5500), pp.2323-2326, 2000.
easily estimated.
[11] D. Ridder and R. Duin, “Locally Linear Embedding For Classification,
Multi Using LLE - Enhance the Multiple Tech-report”, PH-2002-01, Imaging Science & Technology Department,
dictionary method and multi- quality of the sketches cannot Delft University of Technology, Netherlands.
sparse dictionary sparse initial image be retrieved. [12] H. Chang, D. Yeung, and Y. Xiong, “Super- Resolution Through
representati representation -Sketch- Neigbhor Embedding”, Proc. of. Int. Conf. Computer Vision and Pattern
on model. photo recognition Recognition, 2004.
framework rate is better. [13] Q. S. Liu, R. Huang, H. Q. Lu, S. D. Ma, “Face Recognition Using
Kernel Based Fisher Discriminant Analysis”, Proc. of. Int. Conf.
Automatic Face and Gesture Recognition, 2002.
[14] M. H. Yang, Kernel Eigenfaces vs. Kernel Fisherfaces: “Face
VI. CONCLUSION Recognition Using Kernel Methods”, Proc. of. Int. Conf. Automatic
A study of various image retrieval work is done in this Face and Gesture Recognition, pp. 215-220, 2002.
survey. A wide variety of researches is taking place in the [15] Y. Wu, T. S. Huang, and K. Toyama, “Self- Supervised Learning for
Object based on Kernel Discriminant-EM Algorithm”, Proc. of Int.
field of image retrieval. Each work has its own technique, Conf. Computer Vision, 2001.
contribution and limitations. As a review paper, it cannot
[16] Nannan Wang, Xinbo Gao, “Face sketch photo synthesis under multi
incorporate each and every detail of individual works, dictionary sparse representation framework”, sixth international Conf.
however this paper attempts to deal with an elaborate review on image ang graphics,2011.
of the most usual traditional and modern image retrieval [17] J. Yang, J. Wright, T. Huang and Y. Ma,” Image super resolution via
systems. This review mainly focuses on the methods or sparse representation,” IEEE Trans. Image processing, 2010, vol. 19, no.
11,pp.2861-2873.

View publication stats