Semantic video indexing
Arun Hampapur1999, ACM SIGMOD Record
Sign up for access to the world's latest research
checkGet notified about relevant papers
checkSave papers to use in your research
checkJoin the discussion with peers
checkTrack your impact
Abstract
Providing concept level access to video data requires, video management systems tailored to the domain of the data. Effective indexing and retrieval for high-level access mandates the use of domain knowledge. This paper proposes an approach based on the use of knowledge models to building domain specific video information systems. The key issues in such systems are identified and discussed.
Related papers
Using knowledge representation languages for video annotation and retrieval
2006
Effective usage of multimedia digital libraries has to deal with the problem of building efficient content annotation and retrieval tools. In particular in video domain, different techniques for manual and automatic annotation and retrieval have been proposed. Despite the existence of well-defined and extensive standards for video content description, such as MPEG-7, these languages are not explicitly designed for automatic annotation and retrieval purpose. Usage of linguistic ontologies for video annotation and retrieval is a common practice to classify video elements by establishing relationships between video contents and linguistic terms that specify domain concepts at different abstraction levels. The main issue related to the use of description languages such as MPEG-7 or linguistic ontologies is due to the fact that linguistic terms are appropriate to distinguish event and object categories but they are inadequate when they must describe specific or complex patterns of events or video entities. In this paper we propose the usage of knowledge representation languages to define ontologies enriched with visual information that can be used effectively for video annotation and retrieval. Difference between content description languages and knowledge representation languages are shown, the advantages of using enriched ontologies both for the annotation and the retrieval process are presented in terms of enhanced user experience in browsing and querying video digital libraries.
Ontology-based video retrieval
International Journal of Digital Culture and Electronic Tourism, 2009
The paper presents an ontological approach for enabling semantic-aware information retrieval and browsing framework facilitating the user access to its preferred contents. Through the ontologies the system will express key entities and relationships describing learning material in a formal machineprocessable representation. An ontology-based knowledge representation could be used for content analysis and concept recognition, for reasoning processes and for enabling user-friendly and intelligent multimedia content search and retrieval.
Semantic modeling and knowledge representation in multimedia databases
IEEE Transactions on Knowledge and Data Engineering, 1999
In this paper, we present the current state of the art in semantic data modeling of multimedia data. Semantic conceptualization can be performed at several levels of information granularity, leading to multilevel indexing and searching mechanisms. Various models at different levels of granularity are compared. At the finest level of granularity, multimedia data can be indexed based on image contents, such as identification of objects and faces. At a coarser level of granularity, indexing of multimedia data can be focused on events and episodes, which are higher level abstractions. In light of the above, we also examine modeling and indexing techniques of multimedia documents.
Semantic Reasoning based Video Database Systems
Lecture Notes in Computer Science, 2000
A constraint of existing content-based video data models is that each modeled semantic description must be associated with time intervals exactly within which it happens and semantics not related to any time interval are not considered. Consequently, users are provided with limited query capabilities. This paper is aimed at developing a novel model with two innovations: (1) Semantic contents not having related time information can be modeled as ones that do (2) Not only the temporal feature of semantic descriptions, but also the temporal relationships among themselves are components of the model. The query system is by means of reasoning on those relationships. To support users' access, a video algebra and a video calculus as formal query languages, which are based on semantic relationship reasoning, are also presented.
Interoperability Support for Ontology-Based Video Retrieval Applications
Lecture Notes in Computer Science, 2004
Domain ontologies are very useful for indexing, query specification, retrieval and filtering, user interfaces, even information extraction from audiovisual material. The dominant emerging language standard for the description of domain ontologies is OWL. We describe here a methodology and software that we have developed for the interoperability of OWL with the complete MPEG-7 MDS so that domain ontologies described in OWL can be transparently integrated with the MPEG-7 MDS metadata. This allows applications that recognize and use the MPEG-7 MDS constructs to make use of domain ontologies for applications like indexing, retrieval, filtering etc. resulting in more effective user retrieval and interaction with audiovisual material.
A generic framework for semantic video indexing based on visual concepts/contexts detection
Multimedia Tools and Applications, 2014
Providing a semantic access to video data requires the development of concept detectors. However, semantic concepts detection is a hard task due to the large intra-class and the small inter-class variability of content. Moreover, semantic concepts co-occur together in various contexts and their occurrence may vary from one to another. Thus, it is interesting to exploit this knowledge in order to achieve satisfactory performances. In this paper we present a generic semantic video indexing scheme, called SVI_REGIMVid. It is based on three levels of analysis. The first level (level1) focuses on low-level processing such as video shot boundary/ key-frame detection, annotation tools, key-points detection and visual features extraction tools. The second level (level2) aims to build the semantic models for supervised learning of concepts/contexts. The third level (level3) enriches the semantic interpretation of concepts/ contexts by exploiting fuzzy knowledge. The obtained experimental results are promising for a semantic concept/context detection process.
Knowledge representation for semantic multimedia content analysis and reasoning
Proc. of the European …, 2004
A metadata classification schema for semantic content analysis of videos
2002
Simple ancillary metadata, such as those encompassed by the 15 elements of the Dublin Core, may be sufficient and entirely appropriate for basic coarse-granularity cross-domain resource discovery. However, they are insufficient and inappropriate for content description of complex data types such as videos, which require more detailed relational models. We propose a metadata classification schema for the characterization of items and events in videos that permits subsequent query by content. Following MPEG-7 nomenclature, metadata intrinsic to the information content of the video are defined as either structural or semantic, where structural metadata are numerical feature primitives produced by analysing the colour, shape, texture, structure and motion within the video frames, whereas semantic metadata describe the locations and timings of individual items and particular actions or events in the video, and are thus of higher information value. In this paper, the semantic metadata required to describe the visual information content of videos are defined and classified into four distinct classes: Media Entities; Content Items; Events;
Domain knowledge based queries for multimedia data retrieval
2007
This paper describes an approach for semantic description and retrieval of multimedia data described by means of MPEG-7. This standard uses XML schema to define the descriptions. Therefore, it lacks ability to represent the data semantics in a formal and concise way and it does not allow integration and use of domain specific knowledge. Moreover, inference mechanisms are not provided and hence the extraction of implicit information is not (always) possible. To address these issues, we propose to add a conceptual layer on top of MPEG-7 metadata layer, where the domain knowledge is represented using a formal language. A set of mapping rules is proposed. They serve as a bridge between the two layers. Querying MPEG-7 descriptions using XML query languages such as XPath or XQuery requires to know MPEG-7 syntax and documents structure. To provide a flexible query formulation, we exploit the conceptual layer vocabulary to express user queries. A user query, making reference to terms specified at the conceptual level, is rewritten into an XQuery expression over MPEG-7 descriptions.
A rule-based video database system architecture
Information Sciences, 2002
We propose a novel architecture for a video database system incorporating both spatio-temporal and semantic (keyword, event/activity and category-based) query facilities. The originality of our approach stems from the fact that we intend to provide full support for spatio-temporal, relative object-motion and similarity-based objecttrajectory queries by a rule-based system utilizing a knowledge-base while using an object-relational database to answer semantic-based queries. Our method of extracting and modeling spatio-temporal relations is also a unique one such that we segment video clips into shots using spatial relationships between objects in video frames rather than applying a traditional scene detection algorithm. The technique we use is simple, yet novel and powerful in terms of effectiveness and user query satisfaction: video clips are segmented into shots whenever the current set of relations between objects changes and the video frames, where these changes occur, are chosen as keyframes. The directional, topological and third-dimension relations used for shots are those of the keyframes selected to represent the shots and this information is kept, along with frame numbers of the keyframes, in a knowledge-base as Prolog facts. The system has a comprehensive set of inference rules to reduce the number of facts stored in the knowledge-base because a considerable number of facts, which otherwise would have to be stored explicitly, can be derived by rules with some extra effort.
Related topics
Related papers
Semantic representation of multimedia content: Knowledge representation and semantic indexing
Multimedia Tools and Applications, 2008
In this paper we present a framework for unified, personalized access to heterogeneous multimedia content in distributed repositories. Focusing on semantic analysis of multimedia documents, metadata, user queries and user profiles, it contributes to the bridging of the gap between the semantic nature of user queries and raw multimedia documents. The proposed approach utilizes as input visual content analysis results, as well as analyzes and exploits associated textual annotation, in order to extract the underlying semantics, construct a semantic index and classify documents to topics, based on a unified knowledge and semantics representation model. It may then accept user queries, and, carrying out semantic interpretation and expansion, retrieve documents from the index and rank them according to user preferences, similarly to text retrieval. All processes are based on a novel semantic processing methodology, employing fuzzy algebra and principles of taxonomic knowledge representation. The first part of this work presented in this paper deals with data and knowledge models, manipulation of multimedia content annotations and semantic indexing, while the second part will continue on the use of the extracted semantic information for personalized retrieval.
Constructing and Utilizing Video Ontology for Accurate and Fast Retrieval
International Journal of Multimedia Data Engineering and Management, 2011
This paper examines video retrieval based on Query-By-Example (QBE) approach, where shots relevant to a query are retrieved from large-scale video data based on their similarity to example shots. This involves two crucial problems: The first is that similarity in features does not necessarily imply similarity in semantic content. The second problem is an expensive computational cost to compute the similarity of a huge number of shots to example shots. The authors have developed a method that can filter a large number of shots irrelevant to a query, based on a video ontology that is knowledge base about concepts displayed in a shot. The method utilizes various concept relationships (e.g., generalization/specialization, sibling, part-of, and co-occurrence) defined in the video ontology. In addition, although the video ontology assumes that shots are accurately annotated with concepts, accurate annotation is difficult due to the diversity of forms and appearances of the concepts. Demps...
Concept-oriented indexing of video databases: toward semantic sensitive retrieval and browsing
IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, 2004
Digital video now plays an important role in medical education, health care, telemedicine and other medical applications. Several content-based video retrieval (CBVR) systems have been proposed in the past, but they still suffer from the following challenging problems: semantic gap, semantic video concept modeling, semantic video classification, and concept-oriented video database indexing and access. In this paper, we propose a novel framework to make some advances toward the final goal to solve these problems. Specifically, the framework includes: 1) a semantic-sensitive video content representation framework by using principal video shots to enhance the quality of features; 2) semantic video concept interpretation by using flexible mixture model to bridge the semantic gap; 3) a novel semantic video-classifier training framework by integrating feature selection, parameter estimation, and model selection seamlessly in a single algorithm; and 4) a concept-oriented video database org...
Intelligent Content Based Access to Network Video Databases
2001
Video database and video on demand represent important applications of the evolving Global Information Infrastructure. Conventional database management systems (DBMS) are not well-suited for video querying and retrieval because they are designed to operate on alphanumeric data where selection conditions are easier to specify. Defining similarity between video data is difficult since the similarity involves semantics of the video, which is inherent in the data itself. We are planning to integrate two approaches to modeling, indexing and querying video data. The first one is based on textual annotations and the second is based on object motion and digital video processing techniques, in order to support content based and intelligent access. In addition to this novel data model, we also plan to develop a s ystem prototype that can synergistically use multiple media modes to retrieve data using fuzzy information fusion.
Semantic Web for Content Based Video Retrieval
2009 IEEE International Conference on Semantic Computing, 2009
This paper aims to provide a semantic web based video search engine. Currently, we do not have scalable integration platforms to represent extracted features from videos, so that they could be indexed and searched. The task of indexing extracted features from videos is a difficult challenge, due to the diverse nature of the features and the temporal dimensions of videos. We present a semantic web based framework for automatic feature extraction, storage, indexing and retrieval of videos. Videos are represented as interconnected set of semantic resources. Also, we suggest a new ranking algorithm for finding related resources which could be used in a semantic web based search engine.
Knowledge-Driven Video Information Retrieval with LOD: From Semi-Structured to Structured Video Metadata
In parallel with the tremendously increasing number of video contents on the Web, many technical specifications and standards have been introduced to store technical details and describe the content of, and add subtitles to, online videos. Some of these specifications are based on unstructured data with limited machine-processability, data reuse, and interoperability, while others are XML-based, representing semi-structured data. While low-level video features can be derived automatically, high-level features are mainly related to a particular knowledge domain and heavily rely on human experience, judgment, and background. One of the approaches to solve this problem is to map standard, often semi-structured, vocabularies, such as that of MPEG-7, to machine-interpretable ontology. Another approach is to introduce new multimedia ontologies. While video contents can be annotated efficiently with terms defined by structured LOD datasets, such as DBpedia, ontology standardization would be desired in the video production and distribution domains. This paper compares the state-of-the-art video annotations in terms of descriptor level and machine-readability, highlights the limitations of the different approaches, and makes suggestions towards standard video annotations.
Smart VideoText: a video data model based on conceptual graphs
2002
Abstract. An intelligent annotation-based video data model called Smart VideoText is introduced. It utilizes the conceptual graph knowledge representation formalism to capture the semantic associations among the concepts described in text annotations of video data. The aim is to achieve more effective query, retrieval, and browsing capabilities based on the semantic content of video data. Finally, a generic and modular video database architecture based on the Smart VideoText data model is described.
A semi-automatic semantic annotation tool for video databases
2002
Advances in compression techniques, decreasing cost of storage, and high-speed transmission have facilitated the way video is created, stored and distributed. As a consequence, video is now being used in many application areas. The increase in the amount of video data deployed and used in today's applications not only caused video to draw more attention as a multimedia data type, but also led to the requirement of efficient management of video data. Management of video data paved the way for new research areas, such as indexing and retrieval of videos with respect to their spatio-temporal, visual and semantic contents. In this paper, semantic content of video is studied, where video metadata, activities, actions and objects of interest are considered within the context of video semantic content. A data model is proposed to model video semantic content, which is extracted from video data by a video annotation tool. The work in this paper constitutes a part of a video database system to provide support for semantic queries.
Knowledge assisted analysis and categorization for semantic video retrieval
2004
In this paper we discuss the use of knowledge for the analysis and semantic retrieval of video. We follow a fuzzy relational approach to knowledge representation, based on which we define and extract the context of either a multimedia document or a user query. During indexing, the context of the document is utilized for the detection of objects and for automatic thematic categorization.
A Framework for Semantic Video Content Indexing Using Textual Information
2018 IEEE Second International Conference on Data Stream Mining & Processing (DSMP), 2018
Loading Preview
Sorry, preview is currently unavailable. You can download the paper by clicking the button above.