Skip to main content

Simon Dobrisek

University of Ljubljana, Faculty of Electrical Engineering, Faculty Member

Followers

107

Following

4

Co-authors

4

Public Views

Simon Dobrišek is an Associate Professor of Electrical Engineering, currently working as the Head of the Laboratory Machine Intelligence of the Faculty of Electrical Engineering, University of Ljubljana. His research interests center on artificial intelligence, pattern recognition, biometrics, smart surveillance systems, and spoken language technologies. He is also focused on interdisciplinary research that cuts across the field of cognitive science, linguistics and phonetics, as well as the protection of privacy in the use of information and communication technologies. In the last ten years he has been involved as a key team member in eight national and international ICT projects, and as a project manager and principal researcher in two EU FP7 projects in the field of biometrics, speech technologies, pattern recognition, machine learning, and smart surveillance technologies.

less

Harvard University

Vrije Universiteit Brussel

David Pierre Leibovitz

Carleton University

Jorhat Engineering College Jorhat Assam India

Joaquim Llisterri

Universitat Autònoma de Barcelona

University of Cologne

T. Florian Jaeger

University of Rochester

Iran University of Science & Technology

University of Edinburgh

Armando Marques-Guedes

UNL - New University of Lisbon

InterestsView All (12)

Uploads

Papers by Simon Dobrisek

A slovenian spoken dialog system for air flight inquiries

6th European Conference on Speech Communication and Technology (Eurospeech 1999)

In this paper we give a n o v erview of a Slovenian spoken dialog system, developed within a join... more In this paper we give a n o v erview of a Slovenian spoken dialog system, developed within a joint project in multilingual speech recognition and understanding. The aim of the project is the development o f a n information retrieval system, capable of having a dialog with a user. The paper presents the work on the development of the Slovenian spoken dialog system. Such a system has to be able to handle spontaneous speech, and to provide the user with correct information. The information system being developed for Slovenian speech is used for air ight information retrieval. The system has to answer questions about air ight connections and their time and date. In the paper we present the developed modules of the Slovenian system and show some results with respect to word accuracy, semantic accuracy and dialog success rate.

Dynamic speaker localization based on a novel lightweight R–CNN model

Neural Computing and Applications

Validation of Speech Data for Training Automatic Speech Recognition Systems

2022 30th European Signal Processing Conference (EUSIPCO)

Finite-state super transducers for compact language resource representation in edge voice-AI

Systems Science & Control Engineering

A sequential minimization algorithm for finite-state pronunciation lexicon models

Interspeech 2009, 2009

The paper first presents a large-vocabulary automatic speechrecognition system that is being deve... more The paper first presents a large-vocabulary automatic speechrecognition system that is being developed for the Slovenian language. The concept of a single-pass token-passing algorithm for the fast speech decoding that can be used with the designed multi-level system structure is discussed. From the algorithmic point of view, the main component of the system is a finitestate pronunciation lexicon model. This component has crucial impact on the overall performance of the system and we developed a sequential minimization algorithm that very efficiently reduces the size and algorithmic complexity of the lexicon model. Our finite-state lexicon model is represented as a state-emitting finite-state transducer. The presented experiments show that the sequential minimization algorithm easily outperforms (up to 60%) the conventional algorithms that were developed for the static global optimization of the transitionemitting finite-state transducers. These algorithms are delivered as part of the AT&T FSM library TM and the OpenFST library.

Time- and acoustic-mediated alignment algorithms for speech recognition evaluation

Interspeech 2011, 2011

The paper investigates the time-and acoustic-mediated alignment algorithms that can be used for b... more The paper investigates the time-and acoustic-mediated alignment algorithms that can be used for better speech recognition evaluation. The edit-cost function, which weights the cost of speech unit matches, substitutions, deletions and insertions, is defined as a function of timed symbols or even as a function of speech signal segments. The algorithms are compared using several classical statistical measures of different types that are derived from speech recognition confusion matrices and are normally used to measure the agreement between different classifications of the same set of objects. These measures provide a reasonable indication that the investigated algorithms provide more relevant speech recognition error statistics than the algorithms that are commonly used for this purpose.

3D Moving Sound Source Localization via Conventional Microphones

Elektronika ir Elektrotechnika, 2017

In this paper, we present an approach to sound localization for moving sound sources. We use four... more In this paper, we present an approach to sound localization for moving sound sources. We use four conventional microphones and a multi-channel sound recording device for capturing sound signals. A database with different types of moving sound source scenarios is created and one of the scenarios is analysed in this paper. This scenario involves walking of a person and creating sounds simultaneously at certain positions in a room, and our aim is to determine the location of the person as three dimensions. We use time difference of arrival for 3D sound source localization. The time delays between sounds channels are obtained by excitation source information based time-delay estimation algorithm. In addition, signal processing methods are used for increasing the success of moving sound source localization. Savitzky-Golay and threshold filters are utilized to pre-filter the recorded sounds and removing the background noise signals, respectively. The non-linear equation of the sound localization is solved by using the modified Levenberg-Marquardt Algorithm (LMA) on the Time Difference of Arrival (TDOA) process. The results for the 3D localization of the moving sound source are obtained by measuring the differences between the real and estimated positions of the person.

Speech Segmentation Aspects of Phone Transition Acoustical Modelling

Lecture Notes in Computer Science, 1999

The paper presents our experiences with the phone transition acoustical models. The phone transit... more The paper presents our experiences with the phone transition acoustical models. The phone transition models were compared to the traditional context dependent phone models. We put special attention on the speech signal segmentation analysis to provide a better insight i n to certain segmentation e ects when using the di erent acoustical models. Experiments with the HMM-based models were performed using the HTK toolkit, which w as extended to provide proper state parameter tying for the phone transition models. All the model parameters were estimated on the GOPOLIS speech database. The annotation confusions concerning two-phone speech units are also discussed.

Criteria for the Evaluation of Automated Speech-Recognition Scoring Algorithms

Variations of the basic string-alignment algorithm are commonly used for the detection and classi... more Variations of the basic string-alignment algorithm are commonly used for the detection and classification of speech-recognition errors. In this procedure, reference and system-output hypothesis speech transcriptions are first aligned using the string-alignment algorithm that is based on primitive edit operations. The edit operations needed to transform one transcription into the other are then tallied as speech-recognition errors. The algorithms normally detect approximately the same total number of errors; however, they can produce different error classifications. This paper investigates these differences and several criterion functions that can be used for the comparison and evaluation of the algorithms for the detection and classification of speech-recognition errors. The proposed criterion functions were used for the experimental evaluation of the standard algorithms that are implemented as part of the CUED HTK and the NIST SCTK, and were used for the detection and classification of phone-recognition errors from the TIMIT speech database.

Spoken language resources at LUKS of the University of Ljubljana

International Journal of Speech Technology, 2003

This document provides a description of the database of spoken flight information enquiries in Sl... more This document provides a description of the database of spoken flight information enquiries in Slovene. The database is intended primarily for the development of acoustic speech models that are the key components of various speech technologies, such as speech recognizers and speech synthesizers, as well as biometric speaker-recognition systems, etc.

Towards the optimal minimization of a pronunciation dictionary model

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2010

This paper presents the results of our efforts to obtain the minimum possible finite-state repres... more This paper presents the results of our efforts to obtain the minimum possible finite-state representation of a pronunciation dictionary. Finite-state transducers are widely used to encode word pronunciations and our experiments revealed that the conventional redundancy-reduction algorithms developed within this framework yield suboptimal solutions. We found that the incremental construction and redundancy reduction of acyclic finite-state transducers creates considerably smaller models (up to 60%) than the conventional, non-incremental (batch) algorithms implemented in the OpenFST toolkit.

Language Model Representations for the GOPOLIS Database

Lecture Notes in Computer Science, 1999

The formation of a domain-oriented sentence corpus by sentence pattern rules is described. The sa... more The formation of a domain-oriented sentence corpus by sentence pattern rules is described. The same rules were transformed into word networks to serve as a language model within a HTK based speech recognition system. The performance of the word network language model was compared to the one of the bigram model.

FST-Based Pronunciation Lexicon Compression for Speech Engines

International Journal of Advanced Robotic Systems, 2012

Finite-state transducers are frequently used for pronunciation lexicon representations in speech ... more Finite-state transducers are frequently used for pronunciation lexicon representations in speech engines, in which memory and processing resources are scarce. This paper proposes two possibilities for further reducing the memory footprint of finite-state transducers representing pronunciation lexicons. First, different alignments of grapheme and allophone transcriptions are studied and a reduction in the number of states of up to 30% is reported. Second, a combination of grapheme-to-allophone rules with a finite-state transducer is proposed, which yields a 65% smaller finite-state transducer than conventional approaches.

An Edit-Distance Model for the Approximate Matching of Timed Strings

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009

An edit-distance model that can be used for the approximate matching of contiguous and non-contig... more An edit-distance model that can be used for the approximate matching of contiguous and non-contiguous timed strings is presented. The model extends the concept of the weighted string-edit distance by introducing timed edit operations and by making the edit costs time dependent. Special attention is paid to the timed null symbols that are associated with the timed insertions and deletions. The usefulness of the presented model is demonstrated on the classification of phone-recognition errors using the TIMIT speech database.

Homer II—man–machine interface to internet for blind and visually impaired people

Computer Communications, 2003

HOMER II is a voice-driven text-to-speech system developed for blind or visually impaired persons... more HOMER II is a voice-driven text-to-speech system developed for blind or visually impaired persons for reading Slovenian texts. Users can obtain texts from the Internet site of the Association of Slovenian Blind and Visually Impaired Persons Societies from their Electronic Information System where they can find daily newspapers, some novels and other information. The system consists of four main modules. The first module enables Internet communication, retrieves text to a local disc and converts it to a standard form. The input interface manages the keyboard entry and/or speaker independent speech recognition. The output interface performs speech synthesis of a given text and in addition prints the same text magnified to the screen. The user dialog is responsible for the user friendly communication and controls other tasks of the system. Homer II was ported from Linux to the MS Windows 9x/ME/NT/2000 operating systems. For the best performance it uses multi-threading and other advantages of the 32-bit environment. Further versions of the HOMER system with even more advanced dialogue modules and some basic World Wide Web browsing functionality will represent an important tool in the distance learning and teaching process for the impaired persons using academic networks.

Simultaneous regression and feature learning for facial landmarking

Face alignment (or facial landmarking) is an important task in many face-related applications, ra... more Face alignment (or facial landmarking) is an important task in many face-related applications, ranging from registration, tracking and animation to higher-level classification problems such as face, expression or attribute recognition. While several solutions have been presented in the literature for this task so far, reliably locating salient facial features across a wide range of posses still remains challenging. To address this issue, we propose in this paper a novel method for automatic facial landmark localization in 3D face data designed specifically to address appearance variability caused by significant pose variations. Our method builds on recent cascaded-regression-based methods to facial landmarking and uses a gating mechanism to incorporate multiple linear cascaded regression models each trained for a limited range of poses into a single powerful landmarking model capable of processing arbitrary posed input data. We develop two distinct approaches around the proposed gat...

ChildNet: Structural Kinship Face Synthesis Model With Appearance Control Mechanisms

IEEE Access

Kinship face synthesis is an increasingly popular topic within the computer vision community, par... more Kinship face synthesis is an increasingly popular topic within the computer vision community, particularly the task of predicting the child appearance using parental images. Previous work has been limited in terms of model capacity and inadequate training data, which comprised of low-resolution and tightly cropped images, leading to lower synthesis quality. In this paper, we propose ChildNet, a method for kinship face synthesis that leverages the facial image generation capabilities of a state-of-the-art Generative Adversarial Network (GAN), and resolves the aforementioned problems. ChildNet is designed within the GAN latent space and is able to predict a child appearance that bears high resemblance to real parents' children. To ensure fine-grained control, we propose an age and gender manipulation module that allows precise manipulation of the child synthesis result. ChildNet is capable of generating multiple child images per parent pair input, while providing a way to control the image generation variability. Additionally, we introduce a mechanism to control the dominant parent image. Finally, to facilitate the task of kinship face synthesis, we introduce a new kinship dataset, called Next of Kin. This dataset contains 3690 high-resolution face images with a diverse range of ethnicities and ages. We evaluate ChildNet in comprehensive experiments against three competing kinship face synthesis models, using two kinship datasets. The experiments demonstrate the superior performance of ChildNet in terms of identity similarity, while exhibiting high perceptual image quality. The source code for the model is publicly available at: .

High Resolution Face Editing with Masked GAN Latent Code Optimization

arXiv (Cornell University), Mar 20, 2021

Arched eyebrows Blond hair Big nose Wearing lipstick Fig. 1: This paper introduces MaskFaceGAN, a... more Arched eyebrows Blond hair Big nose Wearing lipstick Fig. 1: This paper introduces MaskFaceGAN, a novel approach to face attribute editing capable of generating high-resolution, artefact-free and photo-realistic editing results through a carefully designed optimization procedure applied over the StyleGAN2 latent space. The presented (1024 × 1024) examples show editing results for four target attributes. Best viewed zoomed-in.

FICE: Text-Conditioned Fashion Image Editing With Guided GAN Inversion

arXiv (Cornell University), Jan 5, 2023

Fashion-image editing represents a challenging computer vision task, where the goal is to incorpo... more Fashion-image editing represents a challenging computer vision task, where the goal is to incorporate selected apparel into a given input image. Most existing techniques deal with this task by first selecting an example image of the desired apparel and then transferring the clothing onto the target person. Such techniques are commonly referred to as Virtual Try-On methods. Conversely, in this paper, we consider editing fashion images with text descriptions. Such an approach has several advantages over example-based virtual try-on techniques, e.g.: (i) it does not require an image of the target fashion item, and (ii) it allows the expression of a wide variety of visual concepts through the use of natural language. Existing image-editing methods that work with language-based inputs are heavily constrained by their requirement for training sets with rich attribute annotations or they are only able to handle simple text descriptions. To address these constraints, we propose a novel text-conditioned editing model, called FICE (Fashion Image CLIP Editing), capable of handling a wide variety of semantically diverse text descriptions to guide the editing procedure. Specifically with FICE, we augment the common GAN inversion process by including semantic, poserelated, and image-level constraints when generating the desired images. We leverage the capabilities of the pretrained CLIP model to enforce the targeted semantics, due to its impressive image-text association capabilities. We furthermore propose a latent-code regularization technique that provides the means to better control the fidelity of the synthesized images and ensures that images are generated from latent codes coming from a well-defined part of the latent space. We validate FICE through comprehensive experiments on a combination of VITON images and Fashion-Gen text descriptions and in comparison with several state-of-the-art text-conditioned image editing approaches. Experimental results demonstrate FICE generates highly realistic fashion images and leads to stronger editing performance than existing competing approaches.

Making the Most of Single Sensor Information: A Novel Fusion Approach for 3D Face Recognition Using Region Covariance Descriptors and Gaussian Mixture Models

Sensors, 2022

Most commercially successful face recognition systems combine information from multiple sensors (... more Most commercially successful face recognition systems combine information from multiple sensors (2D and 3D, visible light and infrared, etc.) to achieve reliable recognition in various environments. When only a single sensor is available, the robustness as well as efficacy of the recognition process suffer. In this paper, we focus on face recognition using images captured by a single 3D sensor and propose a method based on the use of region covariance matrixes and Gaussian mixture models (GMMs). All steps of the proposed framework are automated, and no metadata, such as pre-annotated eye, nose, or mouth positions is required, while only a very simple clustering-based face detection is performed. The framework computes a set of region covariance descriptors from local regions of different face image representations and then uses the unscented transform to derive low-dimensional feature vectors, which are finally modeled by GMMs. In the last step, a support vector machine classificati...

A slovenian spoken dialog system for air flight inquiries

6th European Conference on Speech Communication and Technology (Eurospeech 1999)

In this paper we give a n o v erview of a Slovenian spoken dialog system, developed within a join... more In this paper we give a n o v erview of a Slovenian spoken dialog system, developed within a joint project in multilingual speech recognition and understanding. The aim of the project is the development o f a n information retrieval system, capable of having a dialog with a user. The paper presents the work on the development of the Slovenian spoken dialog system. Such a system has to be able to handle spontaneous speech, and to provide the user with correct information. The information system being developed for Slovenian speech is used for air ight information retrieval. The system has to answer questions about air ight connections and their time and date. In the paper we present the developed modules of the Slovenian system and show some results with respect to word accuracy, semantic accuracy and dialog success rate.

Dynamic speaker localization based on a novel lightweight R–CNN model

Neural Computing and Applications

Validation of Speech Data for Training Automatic Speech Recognition Systems

2022 30th European Signal Processing Conference (EUSIPCO)

Finite-state super transducers for compact language resource representation in edge voice-AI

Systems Science & Control Engineering

A sequential minimization algorithm for finite-state pronunciation lexicon models

Interspeech 2009, 2009

The paper first presents a large-vocabulary automatic speechrecognition system that is being deve... more The paper first presents a large-vocabulary automatic speechrecognition system that is being developed for the Slovenian language. The concept of a single-pass token-passing algorithm for the fast speech decoding that can be used with the designed multi-level system structure is discussed. From the algorithmic point of view, the main component of the system is a finitestate pronunciation lexicon model. This component has crucial impact on the overall performance of the system and we developed a sequential minimization algorithm that very efficiently reduces the size and algorithmic complexity of the lexicon model. Our finite-state lexicon model is represented as a state-emitting finite-state transducer. The presented experiments show that the sequential minimization algorithm easily outperforms (up to 60%) the conventional algorithms that were developed for the static global optimization of the transitionemitting finite-state transducers. These algorithms are delivered as part of the AT&T FSM library TM and the OpenFST library.

Time- and acoustic-mediated alignment algorithms for speech recognition evaluation

Interspeech 2011, 2011

The paper investigates the time-and acoustic-mediated alignment algorithms that can be used for b... more The paper investigates the time-and acoustic-mediated alignment algorithms that can be used for better speech recognition evaluation. The edit-cost function, which weights the cost of speech unit matches, substitutions, deletions and insertions, is defined as a function of timed symbols or even as a function of speech signal segments. The algorithms are compared using several classical statistical measures of different types that are derived from speech recognition confusion matrices and are normally used to measure the agreement between different classifications of the same set of objects. These measures provide a reasonable indication that the investigated algorithms provide more relevant speech recognition error statistics than the algorithms that are commonly used for this purpose.

3D Moving Sound Source Localization via Conventional Microphones

Elektronika ir Elektrotechnika, 2017

In this paper, we present an approach to sound localization for moving sound sources. We use four... more In this paper, we present an approach to sound localization for moving sound sources. We use four conventional microphones and a multi-channel sound recording device for capturing sound signals. A database with different types of moving sound source scenarios is created and one of the scenarios is analysed in this paper. This scenario involves walking of a person and creating sounds simultaneously at certain positions in a room, and our aim is to determine the location of the person as three dimensions. We use time difference of arrival for 3D sound source localization. The time delays between sounds channels are obtained by excitation source information based time-delay estimation algorithm. In addition, signal processing methods are used for increasing the success of moving sound source localization. Savitzky-Golay and threshold filters are utilized to pre-filter the recorded sounds and removing the background noise signals, respectively. The non-linear equation of the sound localization is solved by using the modified Levenberg-Marquardt Algorithm (LMA) on the Time Difference of Arrival (TDOA) process. The results for the 3D localization of the moving sound source are obtained by measuring the differences between the real and estimated positions of the person.

Speech Segmentation Aspects of Phone Transition Acoustical Modelling

Lecture Notes in Computer Science, 1999

The paper presents our experiences with the phone transition acoustical models. The phone transit... more The paper presents our experiences with the phone transition acoustical models. The phone transition models were compared to the traditional context dependent phone models. We put special attention on the speech signal segmentation analysis to provide a better insight i n to certain segmentation e ects when using the di erent acoustical models. Experiments with the HMM-based models were performed using the HTK toolkit, which w as extended to provide proper state parameter tying for the phone transition models. All the model parameters were estimated on the GOPOLIS speech database. The annotation confusions concerning two-phone speech units are also discussed.

Criteria for the Evaluation of Automated Speech-Recognition Scoring Algorithms

Variations of the basic string-alignment algorithm are commonly used for the detection and classi... more Variations of the basic string-alignment algorithm are commonly used for the detection and classification of speech-recognition errors. In this procedure, reference and system-output hypothesis speech transcriptions are first aligned using the string-alignment algorithm that is based on primitive edit operations. The edit operations needed to transform one transcription into the other are then tallied as speech-recognition errors. The algorithms normally detect approximately the same total number of errors; however, they can produce different error classifications. This paper investigates these differences and several criterion functions that can be used for the comparison and evaluation of the algorithms for the detection and classification of speech-recognition errors. The proposed criterion functions were used for the experimental evaluation of the standard algorithms that are implemented as part of the CUED HTK and the NIST SCTK, and were used for the detection and classification of phone-recognition errors from the TIMIT speech database.

Spoken language resources at LUKS of the University of Ljubljana

International Journal of Speech Technology, 2003

This document provides a description of the database of spoken flight information enquiries in Sl... more This document provides a description of the database of spoken flight information enquiries in Slovene. The database is intended primarily for the development of acoustic speech models that are the key components of various speech technologies, such as speech recognizers and speech synthesizers, as well as biometric speaker-recognition systems, etc.

Towards the optimal minimization of a pronunciation dictionary model

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2010

This paper presents the results of our efforts to obtain the minimum possible finite-state repres... more This paper presents the results of our efforts to obtain the minimum possible finite-state representation of a pronunciation dictionary. Finite-state transducers are widely used to encode word pronunciations and our experiments revealed that the conventional redundancy-reduction algorithms developed within this framework yield suboptimal solutions. We found that the incremental construction and redundancy reduction of acyclic finite-state transducers creates considerably smaller models (up to 60%) than the conventional, non-incremental (batch) algorithms implemented in the OpenFST toolkit.

Language Model Representations for the GOPOLIS Database

Lecture Notes in Computer Science, 1999

The formation of a domain-oriented sentence corpus by sentence pattern rules is described. The sa... more The formation of a domain-oriented sentence corpus by sentence pattern rules is described. The same rules were transformed into word networks to serve as a language model within a HTK based speech recognition system. The performance of the word network language model was compared to the one of the bigram model.

FST-Based Pronunciation Lexicon Compression for Speech Engines

International Journal of Advanced Robotic Systems, 2012

Finite-state transducers are frequently used for pronunciation lexicon representations in speech ... more Finite-state transducers are frequently used for pronunciation lexicon representations in speech engines, in which memory and processing resources are scarce. This paper proposes two possibilities for further reducing the memory footprint of finite-state transducers representing pronunciation lexicons. First, different alignments of grapheme and allophone transcriptions are studied and a reduction in the number of states of up to 30% is reported. Second, a combination of grapheme-to-allophone rules with a finite-state transducer is proposed, which yields a 65% smaller finite-state transducer than conventional approaches.

An Edit-Distance Model for the Approximate Matching of Timed Strings

IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009

An edit-distance model that can be used for the approximate matching of contiguous and non-contig... more An edit-distance model that can be used for the approximate matching of contiguous and non-contiguous timed strings is presented. The model extends the concept of the weighted string-edit distance by introducing timed edit operations and by making the edit costs time dependent. Special attention is paid to the timed null symbols that are associated with the timed insertions and deletions. The usefulness of the presented model is demonstrated on the classification of phone-recognition errors using the TIMIT speech database.

Homer II—man–machine interface to internet for blind and visually impaired people

Computer Communications, 2003

HOMER II is a voice-driven text-to-speech system developed for blind or visually impaired persons... more HOMER II is a voice-driven text-to-speech system developed for blind or visually impaired persons for reading Slovenian texts. Users can obtain texts from the Internet site of the Association of Slovenian Blind and Visually Impaired Persons Societies from their Electronic Information System where they can find daily newspapers, some novels and other information. The system consists of four main modules. The first module enables Internet communication, retrieves text to a local disc and converts it to a standard form. The input interface manages the keyboard entry and/or speaker independent speech recognition. The output interface performs speech synthesis of a given text and in addition prints the same text magnified to the screen. The user dialog is responsible for the user friendly communication and controls other tasks of the system. Homer II was ported from Linux to the MS Windows 9x/ME/NT/2000 operating systems. For the best performance it uses multi-threading and other advantages of the 32-bit environment. Further versions of the HOMER system with even more advanced dialogue modules and some basic World Wide Web browsing functionality will represent an important tool in the distance learning and teaching process for the impaired persons using academic networks.

Simultaneous regression and feature learning for facial landmarking

Face alignment (or facial landmarking) is an important task in many face-related applications, ra... more Face alignment (or facial landmarking) is an important task in many face-related applications, ranging from registration, tracking and animation to higher-level classification problems such as face, expression or attribute recognition. While several solutions have been presented in the literature for this task so far, reliably locating salient facial features across a wide range of posses still remains challenging. To address this issue, we propose in this paper a novel method for automatic facial landmark localization in 3D face data designed specifically to address appearance variability caused by significant pose variations. Our method builds on recent cascaded-regression-based methods to facial landmarking and uses a gating mechanism to incorporate multiple linear cascaded regression models each trained for a limited range of poses into a single powerful landmarking model capable of processing arbitrary posed input data. We develop two distinct approaches around the proposed gat...

ChildNet: Structural Kinship Face Synthesis Model With Appearance Control Mechanisms

IEEE Access

Kinship face synthesis is an increasingly popular topic within the computer vision community, par... more Kinship face synthesis is an increasingly popular topic within the computer vision community, particularly the task of predicting the child appearance using parental images. Previous work has been limited in terms of model capacity and inadequate training data, which comprised of low-resolution and tightly cropped images, leading to lower synthesis quality. In this paper, we propose ChildNet, a method for kinship face synthesis that leverages the facial image generation capabilities of a state-of-the-art Generative Adversarial Network (GAN), and resolves the aforementioned problems. ChildNet is designed within the GAN latent space and is able to predict a child appearance that bears high resemblance to real parents' children. To ensure fine-grained control, we propose an age and gender manipulation module that allows precise manipulation of the child synthesis result. ChildNet is capable of generating multiple child images per parent pair input, while providing a way to control the image generation variability. Additionally, we introduce a mechanism to control the dominant parent image. Finally, to facilitate the task of kinship face synthesis, we introduce a new kinship dataset, called Next of Kin. This dataset contains 3690 high-resolution face images with a diverse range of ethnicities and ages. We evaluate ChildNet in comprehensive experiments against three competing kinship face synthesis models, using two kinship datasets. The experiments demonstrate the superior performance of ChildNet in terms of identity similarity, while exhibiting high perceptual image quality. The source code for the model is publicly available at: .

High Resolution Face Editing with Masked GAN Latent Code Optimization

arXiv (Cornell University), Mar 20, 2021

Arched eyebrows Blond hair Big nose Wearing lipstick Fig. 1: This paper introduces MaskFaceGAN, a... more Arched eyebrows Blond hair Big nose Wearing lipstick Fig. 1: This paper introduces MaskFaceGAN, a novel approach to face attribute editing capable of generating high-resolution, artefact-free and photo-realistic editing results through a carefully designed optimization procedure applied over the StyleGAN2 latent space. The presented (1024 × 1024) examples show editing results for four target attributes. Best viewed zoomed-in.

FICE: Text-Conditioned Fashion Image Editing With Guided GAN Inversion

arXiv (Cornell University), Jan 5, 2023

Fashion-image editing represents a challenging computer vision task, where the goal is to incorpo... more Fashion-image editing represents a challenging computer vision task, where the goal is to incorporate selected apparel into a given input image. Most existing techniques deal with this task by first selecting an example image of the desired apparel and then transferring the clothing onto the target person. Such techniques are commonly referred to as Virtual Try-On methods. Conversely, in this paper, we consider editing fashion images with text descriptions. Such an approach has several advantages over example-based virtual try-on techniques, e.g.: (i) it does not require an image of the target fashion item, and (ii) it allows the expression of a wide variety of visual concepts through the use of natural language. Existing image-editing methods that work with language-based inputs are heavily constrained by their requirement for training sets with rich attribute annotations or they are only able to handle simple text descriptions. To address these constraints, we propose a novel text-conditioned editing model, called FICE (Fashion Image CLIP Editing), capable of handling a wide variety of semantically diverse text descriptions to guide the editing procedure. Specifically with FICE, we augment the common GAN inversion process by including semantic, poserelated, and image-level constraints when generating the desired images. We leverage the capabilities of the pretrained CLIP model to enforce the targeted semantics, due to its impressive image-text association capabilities. We furthermore propose a latent-code regularization technique that provides the means to better control the fidelity of the synthesized images and ensures that images are generated from latent codes coming from a well-defined part of the latent space. We validate FICE through comprehensive experiments on a combination of VITON images and Fashion-Gen text descriptions and in comparison with several state-of-the-art text-conditioned image editing approaches. Experimental results demonstrate FICE generates highly realistic fashion images and leads to stronger editing performance than existing competing approaches.

Making the Most of Single Sensor Information: A Novel Fusion Approach for 3D Face Recognition Using Region Covariance Descriptors and Gaussian Mixture Models

Sensors, 2022

Most commercially successful face recognition systems combine information from multiple sensors (... more Most commercially successful face recognition systems combine information from multiple sensors (2D and 3D, visible light and infrared, etc.) to achieve reliable recognition in various environments. When only a single sensor is available, the robustness as well as efficacy of the recognition process suffer. In this paper, we focus on face recognition using images captured by a single 3D sensor and propose a method based on the use of region covariance matrixes and Gaussian mixture models (GMMs). All steps of the proposed framework are automated, and no metadata, such as pre-annotated eye, nose, or mouth positions is required, while only a very simple clustering-based face detection is performed. The framework computes a set of region covariance descriptors from local regions of different face image representations and then uses the unscented transform to derive low-dimensional feature vectors, which are finally modeled by GMMs. In the last step, a support vector machine classificati...