A relevant image search engine with late fusion
luis gonzalo garcìa leiva2011, Proceedings of the 16th international conference on Intelligent user interfaces
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
A fundamental problem in image retrieval is how to improve the text-based retrieval systems, which is known as "bridging the semantic gap". The reliance on visual similarity for judging semantic similarity may be problematic due to the semantic gap between low-level content and higher-level concepts. One way to overcome this problem and increase thus retrieval performance is to consider user feedback in an interactive scenario. In our approach, a user starts a query and is then presented with a set of (hopefully) relevant images; selecting from these images those which are more relevant to her. Then the system refines its results after each iteration, using late fusion methods, and allowing the user to dynamically tune the amount of textual and visual information that will be used to retrieve similar images. We describe how does our approach fit in a real-world setting, discussing also an evaluation of results.
Related papers
Towards Intelligible Query Processing in Relevance Feedback-Based Image Retrieval Systems
Tools in Artificial Intelligence, 2008
We propose in this paper the specification of an image retrieval architecture based on a relevance feedback framework which operates on high-level image descriptions instead of their extracted low-level features. This framework features a conceptual model which integrates visual semantics as well as symbolic relational characterizations and operates on image objects, abstractions of visual entities within a physical image. Also, it manipulates a rich query language, consisting of both boolean and quantification operators, which therefore leads to optimized user interaction and increased retrieval performance. Let us first introduce the context of our research. In order to cope with the storing and retrieval of ever-growing digital image collections, the first retrieval systems (cf. [Smeulders et al. 00] for a review of the state-of-the-art), known as content-based, propose fully automatic processing methods based on low-level signal features (color, texture, shape...). Although they allow the fast processing of queries, they do not make it possible to search for images based on their semantic content and consider for example red apples or Ferraris as being the same entities simply because they have the same color distribution. Failing to relate low-level features to semantic characterization (also known as the semantic gap) has slowed down the development of such solutions since, as shown in [Hollink 04], taking into account aspects related to the image content is of prime importance for efficient retrieval. Also, users are more skilled in defining their information needs using language-based descriptors and would therefore rather be given the possibility to differentiate between red roses and red cars. In order to overcome the semantic gap, a class of frameworks within the framework of the European Fermi project proposed to model the image semantic and signal contents following a sharp process of human-assisted indexing [Mechkour 95] [Meghini et al. 01]. These approaches, based on elaborate knowledge-based representation models, provide satisfactory results in terms of retrieval quality but are not easily usable on large collections of images because of the necessary human intervention required for indexing. Automated systems which attempt to deal with the semantics/signal integration (e.g. iFind [Lu et al. 00] and the prototype presented in [Zhou & Huang 02]) propose solutions based on textual annotations to characterize semantics and on a relevance feedback (RF) scheme operating on low-level features. RF techniques are based on an interaction with a user www.intechopen.com providing judgment on displayed images as to whether and to what extent they are relevant or irrelevant to his need. For each loop of the interaction, these images are learnt and the system tries to display images close in similarity to the ones targeted by the user. As any learning process, it requires an important number of training images to achieve reasonable performance. The user is therefore solicited through several tedious and time-consuming loops to provide feedback for the system in real time, which penalizes user interaction and involves costly computations over the whole set of images. Moreover, starting from a textual query on semantics, these state-of-the art systems are only able to manage opaque RF (i.e. a user selects relevant and/or non-relevant documents and is then proposed a revised ranking without being given the possibility to 'understand' how his initial query was transformed) since it operates on extracted low-level features. Finally, these systems do not take into account the relational spatial information between visual entities, which affects the quality of the retrieval results. Our RF process is a specific case of state-of-the-art RF frameworks reducing the user's burden since it involves a unique loop returning the relevant images. Moreover, as opposed to the opacity of state-of-the-art RF frameworks, it holds the advantage of being transparent (i.e. the system displays the query generated from the selected documents) and penetrable (i.e. the modification of the generated query is possible before processing), which increases the quality of retrieval results. Through the use of a symbolic representation, the user is indeed able to visualize and comprehend the intelligible query being processed. We manage transparent and penetrable interactions by considering a conceptual representation of images and model their conveyed visual semantics and relational information through a high-level and expressive representation formalism. Given a user's feedback (i.e. judgment or relevance or irrelevance), our RF process, operating on both visual semantics and relational spatial characterization, is therefore able to first generate and then display a query for eventual further modifications operated by the user. It enforces computational efficiency by generating a symbolic query instead of dealing with costly learning algorithms and optimizes user interaction by displaying this 'readable' symbolic query instead of operating on hidden low-level features. As opposed to state-of-the-art loosely-coupled solutions penalizing user interaction and retrieval performance with an opaque RF framework operating on low-level features, our architecture combines a keyword-based module with a transparent and penetrable RF process which refines the retrieval results of the first. Moreover, we offer a rich query language consisting of several Boolean operators. At the core of our work is the notion of image objects (IOs), abstract structures representing visual entities within an image. Their specification is an attempt to operate beyond simple low-level signal features since IOs convey the semantic and relational information. In the remainder, we first detail the processes allowing to abstract the extracted low-level features to high-level relational description in section 2. Section 3 deals with the visual semantic characterization. We specify in section 4 the image model and develop its conceptual instantiation integrating visual semantics and relational (spatial) features. Section 5 is dedicated to the presentation of the RF framework. Taking into account spatial relations between semantically-defined visual entities is crucial in the framework of an image retrieval system since it enriches the index structures and www.intechopen.com
Fusion Techniques for Combining Textual and Visual Information Retrieval
This chapter describes several approaches for information fusion that have been used in ImageCLEF over the past seven years. In this context, the fusion of information is mainly meant to combine textual and visual retrieval. Data fusion techniques from 116 papers (62% of ImageCLEF working notes) are categorized, described and discussed. It was observed that three general approaches were used for retrieval that can be categorized based on the system level chosen for combining modalities: 1) at the input of the system with inter–media query expansion, 2) internally to the system with early fusion and 3) at the output of the system with late fusion which is by far the most widely used fusion strategy.
Learning and inferring a semantic space from user's relevance feedback for image retrieval
2002
As current methods for content-based retrieval are incapable of capturing the semantics of images, we experiment with using spectral methods to infer a semantic space from user's relevance feedback, so that our system will gradually improve its retrieval performance through accumulated user interactions. In addition to the long-term learning process, we also model the traditional approaches to query refinement using relevance feedback as a short-term learning process. The proposed shortand long-term learning frameworks have been integrated into an image retrieval system. Experimental results on a large collection of images have shown the effectiveness and robustness of our proposed algorithms.
Multimedia Information Retrieval Based on Late Semantic Fusion Approaches: Experiments on a Wikipedia Image Collection
IEEE Transactions on Multimedia, 2000
Main goal of this work is to show the improvement of using a textual pre-filtering combined with an image re-ranking in a Multimedia Information Retrieval task. The defined three stepbased retrieval processes and a well-selected combination of visual and textual techniques help the developed Multimedia Information Retrieval System to overcome the semantic gap in a given query. In the paper, five different late semantic fusion approaches are discussed and experimented in a realistic scenario for multimedia retrieval like the one provided by the publicly available ImageCLEF Wikipedia Collection.
Relevance Feedback and Learning in Content-Based Image Search
2003
A major bottleneck in content-based image retrieval (CBIR) systems or search engines is the large gap between low-level image features used to index images and high-level semantic contents of images. One solution to this bottleneck is to apply relevance feedback to refine the query or similarity measures in image search process. In this paper, we first address the key issues involved in relevance feedback of CBIR systems and present a brief overview of a set of commonly used relevance feedback algorithms. Almost all of the previously proposed methods fall well into such framework. We present a framework of relevance feedback and semantic learning in CBIR. In this framework, low-level features and keyword annotations are integrated in image retrieval and in feedback processes to improve the retrieval performance. We have also extended framework to a content-based web image search engine in which hosting web pages are used to collect relevant annotations for images and users' feedback logs are used to refine annotations. A prototype system has developed to evaluate our proposed schemes, and our experimental results indicated that our approach outperforms traditional CBIR system and relevance feedback approaches.
Learning a semantic space from user's relevance feedback for image retrieval
IEEE Transactions on Circuits and Systems for Video Technology, 2003
As current methods for content-based retrieval are incapable of capturing the semantics of images, we experiment with using spectral methods to infer a semantic space from user's relevance feedback, so that our system will gradually improve its retrieval performance through accumulated user interactions. In addition to the long-term learning process, we also model the traditional approaches to query refinement using relevance feedback as a short-term learning process. The proposed shortand long-term learning frameworks have been integrated into an image retrieval system. Experimental results on a large collection of images have shown the effectiveness and robustness of our proposed algorithms.
A Semantic Feedback Framework for Image Retrieval
International Journal of Computer and Electrical Engineering, 2010
In this paper, a new system of fuzzy relevance feedback for image retrieval is introduced. In conventional CBIR systems, the users are restricted to make a binary labeling on the retrieval results, while this determination is difficult for rich images in semantic. In the proposed system, we accumulate user interactions using a soft feedback model to construct Fuzzy Transaction Repository (FTR). The repository remembers the user's intent and, therefore, in terms of the semantic meanings, provides a better representation of each image in the database. To best exploit the benefits of user feedback, we improved the proposed system, so that the repository remembers the user's intent in a suitable manner (as structure-based fuzzy transaction repository) and provides an accurate representation for each image in the database. The semantic similarity between the query and each database image can then be computed using the current feedback and the semantic values in the FTR. Furthermore, feature re-weighting is applied to the session-term feedback in order to learn the weight of low-level features. These two similarity measures are normalized and combined together to form the overall similarity measure. Our experimental results show that the average precision of the proposed systems exceeds 83% after three iterations.
RELEVANCE FEEDBACK: A PROPOSED METHOD FOR REDUCING SEMANTIC GAP IN CONTENT-BASED IMAGE RETRIEVAL USING IMAGES WITH RELEVANCE LEVEL
2013
Content Based Image Retrieval (CBIR) systems attempt to allow users to perform searches in large image repositories. Content-Based Image Retrieval (CBIR) has become one of the most progressive research areas in the past few years. In content Based Image Retrieval, images are retrieved based on color, texture and shape (low level perception). There is a gap between user semantics (high level perception/concepts) and low level perception is called ‘Semantic Gap’. Relevance Feedback (Relevance Feedback) learns association between high level semantics and low level features. While these research efforts establish the basis of CBIR, the usefulness of the proposed approaches is limited. Specifically, these efforts have relatively ignored two distinct characteristics of CBIR systems are semantic gap and human perception of visual content respectively. In this paper, we propose different aspects of the system such as first, we analyze the nature of the Relevance Feedback problem in a continuous representation space in the context of image retrieval. Secondly, a Relevance Feedback based interactive retrieval approach, which effectively takes into account the above two characteristics in CBIR. During the retrieval process, the user's high level query and perception subjectivity are captured by dynamically updated weights based on the user's feedback and finally, the proposed system where user can view/understands the relevance level of the retrieved result of images to his/her given query image. The proposed approach greatly reduces the user's effort of composing a query and captures the user's information need more specifically. We can reduce the user intervention in the CBIR retrieval system.
Unifying keywords and visual contents in image retrieval
IEEE Multimedia, 2002
We're interested in using keywords and visual content together in image retrieval. We used a seamless joint querying and relevance feedback scheme based on keywords and lowlevel visual content, incorporating keyword similarities. We developed an algorithm for a learned word similarity matrix and conducted experiments that validated our approach.
Scalable relevance feedback using click-through data for web image retrieval
2006
Relevance feedback (RF) has been extensively studied in the content-based image retrieval community.
Using Implicit Relevane Feedback to Advance Web Image Search
2006
Although relevance feedback has been extensively studied in content-based image retrieval in the academic area, no commercial web image search engine has employed the idea. There are several obstacles for Web image search engines in applying relevance feedback. To overcome these obstacles, we proposed an efficient implicit relevance feedback mechanism. The proposed mechanism shows advantage over traditional relevance feedback methods in the following three aspects. Firstly, instead of enforcing the users to make explicit judgment on the results, our method regards user's click-through data as implicit relevance feedback which release burden from users. Secondly, a hierarchical image search results clustering algorithm is proposed to semantically organize the search results. Using the clustering results as features, our relevance feedback scheme could catch and reflect users' search intention precisely. Lastly, unlike traditional relevance feedback user interface which hardily substitutes subsequent results for previous ones, our method employed friendly recommendation rather than substitution to let the user narrow down on the refined images. To evaluate the implicit relevance feedback mechanism, comprehensive user studies were performed.
Context-based image similarity queries
2006
In this paper an effective context-based approach for interactive similarity queries is presented. By exploiting the notion of image "context", it is possible to associate different meanings to the same query image. This is indeed necessary to model complex query concepts that, due to their nature, cannot be effectively represented without contextualizing the target image. The context model is simple yet effective and consists of a set of significant images (possibly not relevant to the query) that describe the semantic meaning the user is interested in. When feedback is present, the query context assumes a dynamic nature, changing over time depending on the actual retrieved images judged as relevant by the user for her current search task. Moreover, the proposed approach is able to complement the role of relevance feedback by persistently maintaining the query parameters determined through user interaction over time and ensuring search efficiency. Experimental results on a database of about 10,000 images show the high quality contribution of the proposed approach.
Late Semantic Fusion Approach for the Retrieval of Multimedia Data
—In Multimedia information retrieval late semantic fusion is used to combine textual pre-filtering with an image re-ranking. Three steps are used for retrieval processes. Visual and textual techniques are combined to help the developed Multimedia Information Retrieval System to minimize the semantic gap for given query. In the paper, different late semantic fusion approaches i.e. Product, Enrich, MaxMerge and FilterN are used and for experiments publicly available ImageCLEF Wikipedia Collection is used.
Interactive visual information retrieval
2000
The need to retrieve visual information from large image and video collections is shared by many application domains. This paper describes the main features of our image search engine of Quicklook. Quicklook allows the user to query image and video databases with the aid of example images or a user-made sketch, and progressively refine the system's response by indicating the relevance, or nonrelevance of the retrieved items.
Interactive content-based image retrieval using relevance feedback
2002
Database search engines are generally used in a one-shot fashion in which a user provides query information to the system and, in return, the system provides a number of database instances to the user. A relevance feedback system allows the user to indicate to the system which of these instances are desirable, or relevant, and which are not. Based on this feedback, the system modifies its retrieval mechanism in an attempt to return a more desirable instance set to the user.
Interactive Visual and Semantic Image Retrieval
Intelligent Systems Reference Library, 2013
Information Fusion for Combining Visual and Textual Image Retrieval
Pattern Recognition (ICPR), …, 2010
In this paper, classical approaches such as maximum combinations (combMAX), sum combinations (comb-SUM) and the product of the maximum and a non-zero number (combMNZ) were employed and the trade-off between two fusion effects (chorus and dark horse effects) was studied based on the sum of n maximums. Various normalization strategies were tried out. The fusion algorithms are evaluated using the best four visual and textual runs of the ImageCLEF medical image retrieval task 2008 and 2009. The results show that fused runs outperform the best original runs and multimodality fusion statistically outperforms single modality fusion. The logarithmic rank penalization shows to be the most stable normalization. The dark horse effect is in competition with the chorus effect and each of them can produce best fusion performance depending on the nature of the input data.
Learning From User Feedback In Image Retrieval Systems
Advances in neural information processing …, 1999
Relevance feedback: a power tool for interactive content-based image retrieval
IEEE Transactions on Circuits and Systems for Video Technology, 1998
Based Image Retrieval CBIR has become one of the most active research areas in the past few years. Many visual feature representations have been explored and many systems built. While these research e orts establish the basis of CBIR, the usefulness of the proposed approaches is limited. Speci cally, these e orts have relatively ignored two distinct characteristics of CBIR systems: 1 the gap between high level concepts and low level features; 2 subjectivity o f h uman perception of visual content. This paper proposes a relevance feedback based interactive retrieval approach, which e ectively takes into account the above t wo c haracteristics in CBIR. During the retrieval process, the user's high level query and perception subjectivity are captured by dynamically updated weights based on the user's feedback. The experimental results over more than 70,000 images show that the proposed approach greatly reduces the user's e ort of composing a query and captures the user's information need more precisely. Keywords| Content-Based Image Retrieval, interactive multimedia processing, relevance feedback
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.