Skip to main content

Filiz Bunyak

Followers

25

Following

13

Co-authors

13

Public Views

University of Cambridge

Fusun Balik Sanli

Yildiz Technical University

Lebanese University

University of Technology Sydney

PALIMOTE JUSTICE

RIVERS STATE POLYTECHNIC

Rose-Hulman Institute of Technology

Punjab Technical University

Mohamad Ivan Fanany

University of Indonesia

Musabe Jean Bosco

Chongqing University of Posts and Telecommunications

Software Competence Center Hagenberg

Interests

Uploads

Papers by Filiz Bunyak

Microvascular Pachymeningeal Lymphatic Structures: Association with Blood Vessels and Cranial Bones

The FASEB Journal

Phase transitions during fruiting body formation in Myxococcus

The formation of a collectively moving group benefits individuals within a popula-tion in a varie... more The formation of a collectively moving group benefits individuals within a popula-tion in a variety of ways such as ultra-sensitivity to perturbation, collective modes of feeding, and protection from environmental stress. While some collective groups use a single organizing principle, others can dynamically shift the behavior of the group by modifying the interaction rules at the individual level. The surface-dwelling bac-terium Myxococcus xanthus forms dynamic collective groups both to feed on prey and to aggregate during times of starvation. The latter behavior, termed fruiting-body for-mation, involves a complex, coordinated series of density changes that ultimately lead to three-dimensional aggregates comprising hundreds of thousands of cells and spores. This multi-step developmental process most likely involves several different single-celled behaviors as the population condenses from a loose, two-dimensional sheet to a three-dimensional mound. Here, we use high-resolution micr...

Fast Integral Histogram Computations on GPU for Real-Time Video Analytics

ArXiv, 2017

In many Multimedia content analytics frameworks feature likelihood maps represented as histograms... more In many Multimedia content analytics frameworks feature likelihood maps represented as histograms play a critical role in the overall algorithm. Integral histograms provide an efficient computational framework for extracting multi-scale histogram-based regional descriptors in constant time which are considered as the principle building blocks of many video content analytics frameworks. We evaluate four different mappings of the integral histogram computation onto Graphics Processing Units (GPUs) using different kernel optimization strategies. Our kernels perform cumulative sums on row and column histograms in a cross-weave or wavefront scan order, use different data organization and scheduling methods that is shown to critically affect utilization of GPU resources (cores and shared memory). Tiling the 3-D array into smaller regular data blocks significantly speeds up the efficiency of the computation compared to a strip-based organization. The tiled integral histogram using a diagon...

Segmentation of Breast Cancer Tissue Microarrays for Computer-Aided Diagnosis in Pathology

Computer-Aided Diagnosis (CAD) systems for pathologists can act as an intelligent digital assista... more Computer-Aided Diagnosis (CAD) systems for pathologists can act as an intelligent digital assistant supporting automated grading and morphometric-based discovery of tissue features that are important in cancer diagnosis and patient prognosis. Automated image segmentation is an essential component of computer-based grading in CAD. We describe a novel tissue segmentation algorithm using local feature-based active contours in a globally convex formulation. Preliminary results using the Stanford Tissue MicroArray database shows promising stromal/epithelial superpixel segmentation.

Facial Component Segmentation Using Convolutional Neural Network

Facial components are important for many face image analysis applications. Facial component segme... more Facial components are important for many face image analysis applications. Facial component segmentation is a challenging task due to variations in illumination conditions, pose, scale, skin color etc. Deep learning is a novel branch of machine learning, very efficient in solving complex problems. In this study, we developed a deep Convolutional Neural Network (CNN) to automatically segment facial components in face images. The network has been trained with face images in Radboud face database. Training labels have been created using Face++ SDK. The developed CNN produces a segmentation mask where mouth, eyes, and eyebrows components of the face are marked as foreground. We have focused on these components because they can include very important information for facial image analysis studies such as facial expression recognition. The segmentation success rate of the study is 98.01 according to average accuracy.

Level set-based histology image segmentation with region-based comparison

Automated histological grading of tissue biopsies for clinical cancer care is a challenging probl... more Automated histological grading of tissue biopsies for clinical cancer care is a challenging problem that requires sophisticated algorithms for image segmentation, tissue architecture characterization, global texture feature extraction, and high-dimensional clustering and classification algorithms. Currently there are no automatic image-based grading systems for establishing the pathology of cancer tissues. A primary step in computational histology is accurate image segmentation to detect significant regions such as nuclei, lumen and epithelial cytoplasm which together make up a gland structure. We describe a new approach for tissue segmentation using fuzzy spatial clustering and level set active contours. The proposed technique shows improvement in segmentation accuracy and outperform the classical clustering and level set methods, when compared to ground truth segmentation.

Persistent target tracking using likelihood fusion in wide-area and full motion video sequences

2012 15th International Conference on Information Fusion, 2012

Vehicle tracking using airborne wide-area motion imagery (WAMI) for monitoring urban environments... more Vehicle tracking using airborne wide-area motion imagery (WAMI) for monitoring urban environments is very challenging for current state-of-the-art tracking algorithms, compared to object tracking in full motion video (FMV). Characteristics that constrain performance in WAMI to relatively short tracks range from the limitations of the camera sensor array including low frame rate and georegistration inaccuracies, to small target support size, presence of numerous shadows and occlusions from buildings, continuously changing vantage point of the platform, presence of distractors and clutter among other confounding factors. We describe our Likelihood of Features Tracking (LoFT) system that is based on fusing multiple sources of information about the target and its environment akin to a track-before-detect approach. LoFT uses image-based feature likelihood maps derived from a template-based target model, object and motion saliency, track prediction and management, combined with a novel ad...

Semi-automatic System for Rapid Annotation of Moving Objects in Surveillance Videos using Deep Detection and Multi-object Tracking Techniques

2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), 2020

With the recent advances in video sensor technologies, emergence of new applications associated w... more With the recent advances in video sensor technologies, emergence of new applications associated with these technologies, and demand for automated video analytics have increased the need for ground-truth annotations. Researchers attempt to explore different methodologies and algorithms on different challenging datasets. Ground-truth annotations are needed for quantitative evaluation and comparison of video analysis methods, and training machine learning approaches. Ground Truth Generator (GTG) is our semi-automatic solution for rapid moving object annotation. GTG allows annotation of high densities of moving objects in a video with considerably fewer human interactions, and greatly reduces annotation effort and time. For each individual moving object in a video GTG generates bounding boxes, object classes, and motion trajectories. GTG integrates a fully-automated detection, tracking, and fusion pipeline with an interactive visualization, annotation, and editing interface. Our propose...

Motion U-Net: Multi-cue Encoder-Decoder Network for Motion Segmentation

2020 25th International Conference on Pattern Recognition (ICPR), 2021

Detection of moving objects is a critical component of many computer vision tasks. Recently, deep... more Detection of moving objects is a critical component of many computer vision tasks. Recently, deep learning architectures have been developed for supervised learning based moving object change detection. Some top performing architectures, like FgSegNet are single frame spatial appearance cue-based detection and tend to overfit to the training videos. We propose a novel compact multi-cue autoencoder deep architecture, Motion U-Net (MU-Net) for robust moving object detection that generalizes much better than FgSegNet and requires nearly 30 times fewer weight parameters. Motion and change cues are estimated using a multi-modal background subtraction module combined with flux tensor motion estimation. MU-Net was trained and evaluated on the CDnet-2014 change detection challenge video sequences and had an overall F-measure of 0.9369. We used the unseen SBI-2015 video dataset to assess generalization capacity where MU-Net had an F-measure of 0.7625 while FgSegNet_v2 was 0.3519, less than h...

Robust target tracking using adaptive color feature and likelihood fusion

Designing a robust and accurate object tracker is important in many computer vision applications.... more Designing a robust and accurate object tracker is important in many computer vision applications. The problem becomes more complicated when additional factors like changing appearance, illumination, and scale are introduced in the sequence. Recently, trackers that are based on the correlation filter method like Sum of Template and Pixel-wise Learners (STAPLE)1 have shown state-of-the-art short-term tracking performance. STAPLE consists two major modules: learning correlation filter on HOG features and representing color information using RGB histogram. In this paper, we propose an improved STAPLE (iSTAPLE) tracker by adding the Color Names (CN)2 to the correlation part of the tracker. CN complements HOG feature because using only HOG can lead to tracking failures in some cases where occlusion or deformation is present. As the color information could be a confusing factor and unreliable in tracking due to the rapid illumination changes, Bhattacharyya distance is used to measure the c...

Geospatial content summarization of UAV aerial imagery using mosaicking

In this study, we devise a method for summarizing geospatial content of aerial imagery using mosa... more In this study, we devise a method for summarizing geospatial content of aerial imagery using mosaicking. Mosaicking has been a popular method for combining large sequence of image information and providing a wide field of view in computer vision. We propose a feature based registration method which brings the complexity and homogeneity nature of maize data under consideration and formulates a two step registration method which reduces error accumulation acquired through traditional frame-to-frame registration. Experimental result shows good quality mosaic generation without any metadata information available for long sequence of aerial video. This geospatial summarization of crop field data can be helpful for phenotypic analysis, monitoring of field, checking movement and growth of plants, change in field dynamics, etc.

Improving Nuclei Classification Performance in H&E Stained Tissue Images Using Fully Convolutional Regression Network and Convolutional Neural Network

Detection and classification of nuclei in histopathology images is an important step in the resea... more Detection and classification of nuclei in histopathology images is an important step in the research of understanding tumor microenvironment and evaluating cancer progression and prognosis. The task is challenging due to imaging factors such as varying cell morphologies, batch-to-batch variations in staining, and sample preparation. We present a two-stage deep learning pipeline that combines a Fully Convolutional Regression Network (FCRN) that performs nuclei localization with a Convolution Neural Network (CNN) that performs nuclei classification. Instead of using hand-crafted features, the system learns the visual features needed for detection and classification of nuclei making the process robust to the aforementioned variations. The performance of the proposed system has been quantitatively evaluated on images of hematoxylin and eosin (H&E) stained colon cancer tissues and compared to the previous studies using the same data set. The proposed deep learning system produces promisi...

Multi-object Tracking Cascade with Multi-Step Data Association and Occlusion Handling

2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)

Multi-object tracking is a fundamental computer vision task with a wide variety of real-life appl... more Multi-object tracking is a fundamental computer vision task with a wide variety of real-life applications ranging from surveillance and monitoring to biomedical video analysis. Multi-object tracking is a challenging task due to complications caused by object appearance changes, complex object dynamics, clutter in the environment, and partial or full occlusions. In this paper, we propose a time-efficient detection-based multi-object tracking system using a threestep cascaded data association scheme that combines a fast spatial distance only short-term data association, a robust tracklet linking step using discriminative object appearance models, and an explicit occlusion handling unit relying not only on tracked objects' motion patterns but also on environmental constraints such as presence of potential occluders in the scene. Our experiments on UA-DETRAC multiobject tracking benchmark dataset consisting of challenging real-world traffic videos show promising results against state-of-the-art trackers.

DeepDDK: A Deep Learning based Oral-Diadochokinesis Analysis Software

2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI)

Oromotor dysfunction caused by neurological disorders can result in significant speech and swallo... more Oromotor dysfunction caused by neurological disorders can result in significant speech and swallowing impairments. Current diagnostic methods to assess oromotor function are subjective and rely on perceptual judgments by clinicians. In particular, the widely used oral-diadochokinesis (oral-DDK) test, which requires rapid, alternate repetitions of speech-based syllables, is conducted and interpreted differently among clinicians. It is therefore prone to inaccuracy, which results in poor test reliability and poor clinical application. In this paper, we present a deep learning based software to extract quantitative data from the oral DDK signal, thereby transforming it into an objective diagnostic and treatment monitoring tool. The proposed software consists of two main modules: a fully automated syllable detection module and an interactive visualization and editing module that allows inspection and correction of automated syllable units. The DeepDDK software was evaluated on speech files corresponding to 9 different DDK syllables (e.g., “Pa”, “Ta”, “Ka”). The experimental results show robustness of both syllable detection and localization across different types of DDK speech tasks.

Image Analysis for Ophthalmology: Segmentation and Quantification of Retinal Vascular Systems

Ocular Fluid Dynamics

The retina is directly connected to the central nervous system and the vascular circulation, whic... more The retina is directly connected to the central nervous system and the vascular circulation, which uniquely enables three-dimensional retinal tissue structures and blood flow dynamics to be imaged and visualized from the exterior using non-invasive imaging modalities. Rapid advances in the types of diagnostic imaging modalities, combined with image processing, computer vision, artificial intelligence, and machine learning algorithms for quantitative image analytics are opening up a host of new possibilities for early diagnosis and treatment of a broad range of eye and systemic diseases with clinical impact. Incorporating patient-specific imaging to estimate geometric structures of vessel morphology and boundary conditions as input to the mathematical and computational fluid-dynamics modeling frameworks described in earlier chapters will enable new ways to predict treatment outcomes and model physiological effects at the systemic level. This chapter describes a set of widely used retinal imaging modalities, including fundoscopy, fluorescein angiography (FA), and optical coherence tomography (OCT), along with emerging modalities to measure retinal blood flow dynamics like optical coherence tomography angiography (OCTA) and laser speckle flowgraphy (LSFG). We use vessel segmentation and quantification as a prototypical ophthalmology image analysis pipeline that can be applied across imaging modalities, to describe processing techniques for measuring geometrical vascular structures. Current challenges and future opportunities especially in using artificial intelligence and deep learning architectures for patient optimized precision medicine and clinical efficacy are highlighted.

Robust multi-object tracking with semantic color correlation

2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)

Multi-object tracking is an important computer vision task with wide variety of real-life applica... more Multi-object tracking is an important computer vision task with wide variety of real-life applications from surveillance and monitoring to biomedical video analysis. Multi-object tracking is a challenging problem due to complications such as partial or full occlusions, factors affecting object appearance, object interaction dynamics, etc. and computational cost. In this paper, we propose a detection-based multi-object tracking system that uses a two-step data association scheme to ensure time efficiency while preserving tracking accuracy; a robust but discriminative object appearance model that compares object color attributes using a novel color correlation cost matrix; and a framework that handles occlusions through prediction. Our experiments on UA-DETRAC multi-object tracking benchmark dataset consisting of challenging real-world traffic videos show promising results against state-of-the-art trackers.

VisDrone-MOT2019: The Vision Meets Drone Multiple Object Tracking Challenge Results

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

The Vision Meets Drone Multiple Object Tracking (MOT) Challenge 2019 is the second annual activit... more The Vision Meets Drone Multiple Object Tracking (MOT) Challenge 2019 is the second annual activity focusing on evaluating multi-object tracking algorithms on drones, held in conjunction with the 17-th International Conference on Computer Vision (ICCV 2019). Results of 12 submitted MOT algorithms on the collected drone-based dataset are presented. Meanwhile, we also report the results of 6 state-of-the-art MOT algorithms, and provide a comprehensive analysis and discussion of the results. The results of all submissions are publicly available at the website: http://www.aiskyeye.com/. The challenge results show that MOT on drones is far from being solved. We believe the challenge can largely boost the research and development in MOT on drone platforms.

SHREC 2020: Classification in cryo-electron tomograms

Computers & Graphics

Cryo-electron tomography (cryo-ET) is an imaging technique that allows us to three-dimensionally ... more Cryo-electron tomography (cryo-ET) is an imaging technique that allows us to three-dimensionally visualize both the structural details of macro-molecular assemblies under near-native conditions and its cellular context. Electrons strongly interact with biological samples, limiting electron dose. The latter limits the signal-to-noise ratio and hence resolution of an individual tomogram to about 50 (5 nm). Biological molecules can be obtained by averaging volumes, each depicting copies of the molecule, allowing for resolutions beyond 4 (0.4 nm). To this end, the ability to localize and classify components is crucial, but challenging due to the low signal-to-noise ratio. Computational innovation is key to mine biological information from cryo-electron tomography. To promote such innovation, we provide a novel simulated dataset to benchmark different methods of localization and classification of biological macromolecules in cryo-electron tomograms. Our publicly available dataset contains ten tomographic reconstructions of simulated cell-like volumes. Each volume contains twelve different types of complexes, varying in size, function and structure. In this paper, we have evaluated seven different methods of finding and classifying proteins. Six research groups present results obtained with learning-based methods and trained on the simulated dataset, as well as a baseline template matching, a traditional method widely used in cryo-ET research. We find that method performance correlates with particle size, especially noticeable for template matching which performance degrades rapidly as the size decreases. We learn that neural networks can achieve significantly better localization and classification performance, in particular convolutional networks with focus on high-resolution details such as those based on U-Net architecture.

Facial expression recognition for monitoring neurological disorders based on convolutional neural network

Multimedia Tools and Applications

Facial expressions are a significant part of non-verbal communication. Recognizing facial express... more Facial expressions are a significant part of non-verbal communication. Recognizing facial expressions of people with neurological disorders is essential because these people may have lost a significant amount of their verbal communication ability. Such an assessment requires time consuming examination involving medical personnel, which can be quite challenging and expensive. Automated facial expression recognition systems that are low-cost and noninvasive can help experts detect neurological disorders. In this study, an automated facial expression recognition system is developed using a novel deep learning approach. The architecture consists of four-stage networks. The first, second and third networks segment the facial components which are essential for facial expression recognition. Owing to the three networks, an iconize facial image is obtained. The fourth network classifies facial expressions using raw facial images and iconize facial images. This four-stage method combines holistic facial information with local part-based features to achieve more robust facial expression recognition. Preliminary experimental results achieved 94.44% accuracy for facial expression recognition on RaFD database. The proposed system produced 5% improvement than the facial expression recognition system by using raw images. This study presents a quantitative, objective and non-invasive facial expression recognition system to help in the monitoring and diagnosis of neurological disorders influencing facial expressions.

Complex Non-sinus-associated Pachymeningeal Lymphatic Structures: Interrelationship With Blood Microvasculature

Frontiers in Physiology

The contribution of cranial dura mater vascular networks, as means for maintaining brain fluid mo... more The contribution of cranial dura mater vascular networks, as means for maintaining brain fluid movement and balance, and as the source of significant initiators and/or contributors to neurological disorders, has been overlooked. These networks consist of both blood and lymphatic vessels. The latter were discovered recently and described as sinus-associated structures thus changing the old paradigm that central nervous system lacks lymphatics. In this study, using markers specific to blood and lymphatic endothelia, we demonstrate the existence of the complex non-sinus-associated pachymeningeal lymphatic vasculature. We further show the interrelationship and possible connections between lymphatic vessels and the dural blood circulatory system. Our novel findings reveal the presence of lymphatic-like structures that exist on their own and/or in close proximity to microvessels. Of particular interest are subsets of vascular complexes with dual (lymphatic and blood) vessel identity representing a unique microenvironment within the cranial dura. The close association of the systemic blood circulation and meningeal lymphatics achieved in these complexes could facilitate fluid exchange between the two compartments and constitute an alternative route for CSF drainage.

Microvascular Pachymeningeal Lymphatic Structures: Association with Blood Vessels and Cranial Bones

The FASEB Journal

Phase transitions during fruiting body formation in Myxococcus

The formation of a collectively moving group benefits individuals within a popula-tion in a varie... more The formation of a collectively moving group benefits individuals within a popula-tion in a variety of ways such as ultra-sensitivity to perturbation, collective modes of feeding, and protection from environmental stress. While some collective groups use a single organizing principle, others can dynamically shift the behavior of the group by modifying the interaction rules at the individual level. The surface-dwelling bac-terium Myxococcus xanthus forms dynamic collective groups both to feed on prey and to aggregate during times of starvation. The latter behavior, termed fruiting-body for-mation, involves a complex, coordinated series of density changes that ultimately lead to three-dimensional aggregates comprising hundreds of thousands of cells and spores. This multi-step developmental process most likely involves several different single-celled behaviors as the population condenses from a loose, two-dimensional sheet to a three-dimensional mound. Here, we use high-resolution micr...

Fast Integral Histogram Computations on GPU for Real-Time Video Analytics

ArXiv, 2017

In many Multimedia content analytics frameworks feature likelihood maps represented as histograms... more In many Multimedia content analytics frameworks feature likelihood maps represented as histograms play a critical role in the overall algorithm. Integral histograms provide an efficient computational framework for extracting multi-scale histogram-based regional descriptors in constant time which are considered as the principle building blocks of many video content analytics frameworks. We evaluate four different mappings of the integral histogram computation onto Graphics Processing Units (GPUs) using different kernel optimization strategies. Our kernels perform cumulative sums on row and column histograms in a cross-weave or wavefront scan order, use different data organization and scheduling methods that is shown to critically affect utilization of GPU resources (cores and shared memory). Tiling the 3-D array into smaller regular data blocks significantly speeds up the efficiency of the computation compared to a strip-based organization. The tiled integral histogram using a diagon...

Segmentation of Breast Cancer Tissue Microarrays for Computer-Aided Diagnosis in Pathology

Computer-Aided Diagnosis (CAD) systems for pathologists can act as an intelligent digital assista... more Computer-Aided Diagnosis (CAD) systems for pathologists can act as an intelligent digital assistant supporting automated grading and morphometric-based discovery of tissue features that are important in cancer diagnosis and patient prognosis. Automated image segmentation is an essential component of computer-based grading in CAD. We describe a novel tissue segmentation algorithm using local feature-based active contours in a globally convex formulation. Preliminary results using the Stanford Tissue MicroArray database shows promising stromal/epithelial superpixel segmentation.

Facial Component Segmentation Using Convolutional Neural Network

Facial components are important for many face image analysis applications. Facial component segme... more Facial components are important for many face image analysis applications. Facial component segmentation is a challenging task due to variations in illumination conditions, pose, scale, skin color etc. Deep learning is a novel branch of machine learning, very efficient in solving complex problems. In this study, we developed a deep Convolutional Neural Network (CNN) to automatically segment facial components in face images. The network has been trained with face images in Radboud face database. Training labels have been created using Face++ SDK. The developed CNN produces a segmentation mask where mouth, eyes, and eyebrows components of the face are marked as foreground. We have focused on these components because they can include very important information for facial image analysis studies such as facial expression recognition. The segmentation success rate of the study is 98.01 according to average accuracy.

Level set-based histology image segmentation with region-based comparison

Automated histological grading of tissue biopsies for clinical cancer care is a challenging probl... more Automated histological grading of tissue biopsies for clinical cancer care is a challenging problem that requires sophisticated algorithms for image segmentation, tissue architecture characterization, global texture feature extraction, and high-dimensional clustering and classification algorithms. Currently there are no automatic image-based grading systems for establishing the pathology of cancer tissues. A primary step in computational histology is accurate image segmentation to detect significant regions such as nuclei, lumen and epithelial cytoplasm which together make up a gland structure. We describe a new approach for tissue segmentation using fuzzy spatial clustering and level set active contours. The proposed technique shows improvement in segmentation accuracy and outperform the classical clustering and level set methods, when compared to ground truth segmentation.

Persistent target tracking using likelihood fusion in wide-area and full motion video sequences

2012 15th International Conference on Information Fusion, 2012

Vehicle tracking using airborne wide-area motion imagery (WAMI) for monitoring urban environments... more Vehicle tracking using airborne wide-area motion imagery (WAMI) for monitoring urban environments is very challenging for current state-of-the-art tracking algorithms, compared to object tracking in full motion video (FMV). Characteristics that constrain performance in WAMI to relatively short tracks range from the limitations of the camera sensor array including low frame rate and georegistration inaccuracies, to small target support size, presence of numerous shadows and occlusions from buildings, continuously changing vantage point of the platform, presence of distractors and clutter among other confounding factors. We describe our Likelihood of Features Tracking (LoFT) system that is based on fusing multiple sources of information about the target and its environment akin to a track-before-detect approach. LoFT uses image-based feature likelihood maps derived from a template-based target model, object and motion saliency, track prediction and management, combined with a novel ad...

Semi-automatic System for Rapid Annotation of Moving Objects in Surveillance Videos using Deep Detection and Multi-object Tracking Techniques

2020 IEEE Applied Imagery Pattern Recognition Workshop (AIPR), 2020

With the recent advances in video sensor technologies, emergence of new applications associated w... more With the recent advances in video sensor technologies, emergence of new applications associated with these technologies, and demand for automated video analytics have increased the need for ground-truth annotations. Researchers attempt to explore different methodologies and algorithms on different challenging datasets. Ground-truth annotations are needed for quantitative evaluation and comparison of video analysis methods, and training machine learning approaches. Ground Truth Generator (GTG) is our semi-automatic solution for rapid moving object annotation. GTG allows annotation of high densities of moving objects in a video with considerably fewer human interactions, and greatly reduces annotation effort and time. For each individual moving object in a video GTG generates bounding boxes, object classes, and motion trajectories. GTG integrates a fully-automated detection, tracking, and fusion pipeline with an interactive visualization, annotation, and editing interface. Our propose...

Motion U-Net: Multi-cue Encoder-Decoder Network for Motion Segmentation

2020 25th International Conference on Pattern Recognition (ICPR), 2021

Detection of moving objects is a critical component of many computer vision tasks. Recently, deep... more Detection of moving objects is a critical component of many computer vision tasks. Recently, deep learning architectures have been developed for supervised learning based moving object change detection. Some top performing architectures, like FgSegNet are single frame spatial appearance cue-based detection and tend to overfit to the training videos. We propose a novel compact multi-cue autoencoder deep architecture, Motion U-Net (MU-Net) for robust moving object detection that generalizes much better than FgSegNet and requires nearly 30 times fewer weight parameters. Motion and change cues are estimated using a multi-modal background subtraction module combined with flux tensor motion estimation. MU-Net was trained and evaluated on the CDnet-2014 change detection challenge video sequences and had an overall F-measure of 0.9369. We used the unseen SBI-2015 video dataset to assess generalization capacity where MU-Net had an F-measure of 0.7625 while FgSegNet_v2 was 0.3519, less than h...

Robust target tracking using adaptive color feature and likelihood fusion

Designing a robust and accurate object tracker is important in many computer vision applications.... more Designing a robust and accurate object tracker is important in many computer vision applications. The problem becomes more complicated when additional factors like changing appearance, illumination, and scale are introduced in the sequence. Recently, trackers that are based on the correlation filter method like Sum of Template and Pixel-wise Learners (STAPLE)1 have shown state-of-the-art short-term tracking performance. STAPLE consists two major modules: learning correlation filter on HOG features and representing color information using RGB histogram. In this paper, we propose an improved STAPLE (iSTAPLE) tracker by adding the Color Names (CN)2 to the correlation part of the tracker. CN complements HOG feature because using only HOG can lead to tracking failures in some cases where occlusion or deformation is present. As the color information could be a confusing factor and unreliable in tracking due to the rapid illumination changes, Bhattacharyya distance is used to measure the c...

Geospatial content summarization of UAV aerial imagery using mosaicking

In this study, we devise a method for summarizing geospatial content of aerial imagery using mosa... more In this study, we devise a method for summarizing geospatial content of aerial imagery using mosaicking. Mosaicking has been a popular method for combining large sequence of image information and providing a wide field of view in computer vision. We propose a feature based registration method which brings the complexity and homogeneity nature of maize data under consideration and formulates a two step registration method which reduces error accumulation acquired through traditional frame-to-frame registration. Experimental result shows good quality mosaic generation without any metadata information available for long sequence of aerial video. This geospatial summarization of crop field data can be helpful for phenotypic analysis, monitoring of field, checking movement and growth of plants, change in field dynamics, etc.

Improving Nuclei Classification Performance in H&E Stained Tissue Images Using Fully Convolutional Regression Network and Convolutional Neural Network

Detection and classification of nuclei in histopathology images is an important step in the resea... more Detection and classification of nuclei in histopathology images is an important step in the research of understanding tumor microenvironment and evaluating cancer progression and prognosis. The task is challenging due to imaging factors such as varying cell morphologies, batch-to-batch variations in staining, and sample preparation. We present a two-stage deep learning pipeline that combines a Fully Convolutional Regression Network (FCRN) that performs nuclei localization with a Convolution Neural Network (CNN) that performs nuclei classification. Instead of using hand-crafted features, the system learns the visual features needed for detection and classification of nuclei making the process robust to the aforementioned variations. The performance of the proposed system has been quantitatively evaluated on images of hematoxylin and eosin (H&E) stained colon cancer tissues and compared to the previous studies using the same data set. The proposed deep learning system produces promisi...

Multi-object Tracking Cascade with Multi-Step Data Association and Occlusion Handling

2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)

Multi-object tracking is a fundamental computer vision task with a wide variety of real-life appl... more Multi-object tracking is a fundamental computer vision task with a wide variety of real-life applications ranging from surveillance and monitoring to biomedical video analysis. Multi-object tracking is a challenging task due to complications caused by object appearance changes, complex object dynamics, clutter in the environment, and partial or full occlusions. In this paper, we propose a time-efficient detection-based multi-object tracking system using a threestep cascaded data association scheme that combines a fast spatial distance only short-term data association, a robust tracklet linking step using discriminative object appearance models, and an explicit occlusion handling unit relying not only on tracked objects' motion patterns but also on environmental constraints such as presence of potential occluders in the scene. Our experiments on UA-DETRAC multiobject tracking benchmark dataset consisting of challenging real-world traffic videos show promising results against state-of-the-art trackers.

DeepDDK: A Deep Learning based Oral-Diadochokinesis Analysis Software

2019 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI)

Oromotor dysfunction caused by neurological disorders can result in significant speech and swallo... more Oromotor dysfunction caused by neurological disorders can result in significant speech and swallowing impairments. Current diagnostic methods to assess oromotor function are subjective and rely on perceptual judgments by clinicians. In particular, the widely used oral-diadochokinesis (oral-DDK) test, which requires rapid, alternate repetitions of speech-based syllables, is conducted and interpreted differently among clinicians. It is therefore prone to inaccuracy, which results in poor test reliability and poor clinical application. In this paper, we present a deep learning based software to extract quantitative data from the oral DDK signal, thereby transforming it into an objective diagnostic and treatment monitoring tool. The proposed software consists of two main modules: a fully automated syllable detection module and an interactive visualization and editing module that allows inspection and correction of automated syllable units. The DeepDDK software was evaluated on speech files corresponding to 9 different DDK syllables (e.g., “Pa”, “Ta”, “Ka”). The experimental results show robustness of both syllable detection and localization across different types of DDK speech tasks.

Image Analysis for Ophthalmology: Segmentation and Quantification of Retinal Vascular Systems

Ocular Fluid Dynamics

The retina is directly connected to the central nervous system and the vascular circulation, whic... more The retina is directly connected to the central nervous system and the vascular circulation, which uniquely enables three-dimensional retinal tissue structures and blood flow dynamics to be imaged and visualized from the exterior using non-invasive imaging modalities. Rapid advances in the types of diagnostic imaging modalities, combined with image processing, computer vision, artificial intelligence, and machine learning algorithms for quantitative image analytics are opening up a host of new possibilities for early diagnosis and treatment of a broad range of eye and systemic diseases with clinical impact. Incorporating patient-specific imaging to estimate geometric structures of vessel morphology and boundary conditions as input to the mathematical and computational fluid-dynamics modeling frameworks described in earlier chapters will enable new ways to predict treatment outcomes and model physiological effects at the systemic level. This chapter describes a set of widely used retinal imaging modalities, including fundoscopy, fluorescein angiography (FA), and optical coherence tomography (OCT), along with emerging modalities to measure retinal blood flow dynamics like optical coherence tomography angiography (OCTA) and laser speckle flowgraphy (LSFG). We use vessel segmentation and quantification as a prototypical ophthalmology image analysis pipeline that can be applied across imaging modalities, to describe processing techniques for measuring geometrical vascular structures. Current challenges and future opportunities especially in using artificial intelligence and deep learning architectures for patient optimized precision medicine and clinical efficacy are highlighted.

Robust multi-object tracking with semantic color correlation

2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)

Multi-object tracking is an important computer vision task with wide variety of real-life applica... more Multi-object tracking is an important computer vision task with wide variety of real-life applications from surveillance and monitoring to biomedical video analysis. Multi-object tracking is a challenging problem due to complications such as partial or full occlusions, factors affecting object appearance, object interaction dynamics, etc. and computational cost. In this paper, we propose a detection-based multi-object tracking system that uses a two-step data association scheme to ensure time efficiency while preserving tracking accuracy; a robust but discriminative object appearance model that compares object color attributes using a novel color correlation cost matrix; and a framework that handles occlusions through prediction. Our experiments on UA-DETRAC multi-object tracking benchmark dataset consisting of challenging real-world traffic videos show promising results against state-of-the-art trackers.

VisDrone-MOT2019: The Vision Meets Drone Multiple Object Tracking Challenge Results

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

The Vision Meets Drone Multiple Object Tracking (MOT) Challenge 2019 is the second annual activit... more The Vision Meets Drone Multiple Object Tracking (MOT) Challenge 2019 is the second annual activity focusing on evaluating multi-object tracking algorithms on drones, held in conjunction with the 17-th International Conference on Computer Vision (ICCV 2019). Results of 12 submitted MOT algorithms on the collected drone-based dataset are presented. Meanwhile, we also report the results of 6 state-of-the-art MOT algorithms, and provide a comprehensive analysis and discussion of the results. The results of all submissions are publicly available at the website: http://www.aiskyeye.com/. The challenge results show that MOT on drones is far from being solved. We believe the challenge can largely boost the research and development in MOT on drone platforms.

SHREC 2020: Classification in cryo-electron tomograms

Computers & Graphics

Cryo-electron tomography (cryo-ET) is an imaging technique that allows us to three-dimensionally ... more Cryo-electron tomography (cryo-ET) is an imaging technique that allows us to three-dimensionally visualize both the structural details of macro-molecular assemblies under near-native conditions and its cellular context. Electrons strongly interact with biological samples, limiting electron dose. The latter limits the signal-to-noise ratio and hence resolution of an individual tomogram to about 50 (5 nm). Biological molecules can be obtained by averaging volumes, each depicting copies of the molecule, allowing for resolutions beyond 4 (0.4 nm). To this end, the ability to localize and classify components is crucial, but challenging due to the low signal-to-noise ratio. Computational innovation is key to mine biological information from cryo-electron tomography. To promote such innovation, we provide a novel simulated dataset to benchmark different methods of localization and classification of biological macromolecules in cryo-electron tomograms. Our publicly available dataset contains ten tomographic reconstructions of simulated cell-like volumes. Each volume contains twelve different types of complexes, varying in size, function and structure. In this paper, we have evaluated seven different methods of finding and classifying proteins. Six research groups present results obtained with learning-based methods and trained on the simulated dataset, as well as a baseline template matching, a traditional method widely used in cryo-ET research. We find that method performance correlates with particle size, especially noticeable for template matching which performance degrades rapidly as the size decreases. We learn that neural networks can achieve significantly better localization and classification performance, in particular convolutional networks with focus on high-resolution details such as those based on U-Net architecture.

Facial expression recognition for monitoring neurological disorders based on convolutional neural network

Multimedia Tools and Applications

Facial expressions are a significant part of non-verbal communication. Recognizing facial express... more Facial expressions are a significant part of non-verbal communication. Recognizing facial expressions of people with neurological disorders is essential because these people may have lost a significant amount of their verbal communication ability. Such an assessment requires time consuming examination involving medical personnel, which can be quite challenging and expensive. Automated facial expression recognition systems that are low-cost and noninvasive can help experts detect neurological disorders. In this study, an automated facial expression recognition system is developed using a novel deep learning approach. The architecture consists of four-stage networks. The first, second and third networks segment the facial components which are essential for facial expression recognition. Owing to the three networks, an iconize facial image is obtained. The fourth network classifies facial expressions using raw facial images and iconize facial images. This four-stage method combines holistic facial information with local part-based features to achieve more robust facial expression recognition. Preliminary experimental results achieved 94.44% accuracy for facial expression recognition on RaFD database. The proposed system produced 5% improvement than the facial expression recognition system by using raw images. This study presents a quantitative, objective and non-invasive facial expression recognition system to help in the monitoring and diagnosis of neurological disorders influencing facial expressions.

Complex Non-sinus-associated Pachymeningeal Lymphatic Structures: Interrelationship With Blood Microvasculature

Frontiers in Physiology

The contribution of cranial dura mater vascular networks, as means for maintaining brain fluid mo... more The contribution of cranial dura mater vascular networks, as means for maintaining brain fluid movement and balance, and as the source of significant initiators and/or contributors to neurological disorders, has been overlooked. These networks consist of both blood and lymphatic vessels. The latter were discovered recently and described as sinus-associated structures thus changing the old paradigm that central nervous system lacks lymphatics. In this study, using markers specific to blood and lymphatic endothelia, we demonstrate the existence of the complex non-sinus-associated pachymeningeal lymphatic vasculature. We further show the interrelationship and possible connections between lymphatic vessels and the dural blood circulatory system. Our novel findings reveal the presence of lymphatic-like structures that exist on their own and/or in close proximity to microvessels. Of particular interest are subsets of vascular complexes with dual (lymphatic and blood) vessel identity representing a unique microenvironment within the cranial dura. The close association of the systemic blood circulation and meningeal lymphatics achieved in these complexes could facilitate fluid exchange between the two compartments and constitute an alternative route for CSF drainage.