For training supervised classifiers to recognize different patterns, large data collections with ... more For training supervised classifiers to recognize different patterns, large data collections with accurate labels are necessary. In this paper, we propose a generic, semi-automatic labeling technique for large handwritten character collections. In order to speed up the creation of a large scale ground truth, the method combines unsupervised clustering and minimal expert knowledge. To exploit the potential discriminant complementarities across features, each character is projected into five different feature spaces. After clustering the images in each feature space, the human expert labels the cluster centers. Each data point inherits the label of its cluster's center. A majority (or unanimity) vote decides the label of each character image. The amount of human involvement (labeling) is strictly controlled by the number of clusters -produced by the chosen clustering approach. To test the efficiency of the proposed approach, we have compared, and evaluated three state-of-the art clustering methods (k-means, self-organizing maps, and growing neural gas) on the MNIST digit data set, and a Lampung Indonesian character data set, respectively. Considering a k-nn classifier, we show that labeling manually only 1.3% (MNIST), and 3.2% (Lampung) of the training data, provides the same range of performance than a completely labeled data set would.
A domain where, even in the era of electronic document processing, handwriting is still widely us... more A domain where, even in the era of electronic document processing, handwriting is still widely used is note-taking on a whiteboard. Such documents are either captured by a pen-tracking device or -which is much more challengingby a camera. In both cases the layout analysis of realistic whiteboard notes is an open research problem.
Statistical modelling techniques for automatic reading systems substantially rely on the availabi... more Statistical modelling techniques for automatic reading systems substantially rely on the availability of compact and meaningful feature representations. State-ofthe-art feature extraction for offline handwriting recognition is usually based on heuristic approaches that describe either basic geometric properties or statistical distributions of raw pixel values. Working well on average, still fundamental insights into the nature of handwriting are desired. In this paper we present a novel approach for the automatic extraction of appearance-based representations of offline handwriting data. Given the framework of deep belief networks -Restricted Boltzmann Machines -a two-stage method for feature learning and optimization is developed. Given two standard corpora of both Arabic and Roman handwriting data it is demonstrated across script boundaries, that automatically learned features achieve recognition results comparable to state-of-theart handcrafted features. Given these promising results the potential of feature learning for future reading systems is discussed.
The recognition of handwritten characters is an almost solved problem thanks to efficient machine... more The recognition of handwritten characters is an almost solved problem thanks to efficient machine learning techniques. However, the evaluation and the choice of thresholds to meet a certain level of performance remains a challenge. In this paper, we compare different rejection techniques to determine if a character has been successfully detected or not. Whereas the evaluation of binary classifiers through ROC curves and cost curves has been largely exploited in the literature, multiclass problems can involve different issues like the computational cost or the choice of having an adaptive threshold for each class. The proposed methods for rejection criteria include the maximization of the distance to the optimal performance in the ROC space. We show that optimizing this criterion is a suitable approach on two databases: Lampung handwritten characters, and Arabic handwritten digits. The results support the conclusion that in database where the accuracy is high, pairwise ROC curves analysis in multi-class problems can lead to a finer evaluation of the performance and thresholds definition.
The recognition of fully multi-oriented handwritten characters is a challenging problem. Contrary... more The recognition of fully multi-oriented handwritten characters is a challenging problem. Contrary to univariate signals where the shift invariance property in the Fourier transform can be used, multivariate signals like images require special care to extract rotation invariant features. Several strategies to solve such classification tasks are possible. The proposed method considers input features obtained by the Radon transform or Polar transform. A convolutional neural network is then used for extracting higher level features. This classifier includes in addition the Fast Fourier Transform for extracting shift invariant features at the neural network level. The Radon transform and the convolutional layers process the image at the pixel level while the Fourier transform and the upper layers of the neural network process rotation invariant features. The classifier is evaluated on multi-oriented handwritten digits based on the MNIST database (Arabic digits) and on the ISI database (Bangla digits). The average recognition rate for multi-oriented characters is 93.10% for the Arabic digits and 77.01% for the Bangla digits. This neural architecture highlights the interest of the radial convolutional layer for the recognition of multi-oriented shapes.
2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health (CICARE), 2014
In this paper we present a fast and effective method for labeling images in a large image collect... more In this paper we present a fast and effective method for labeling images in a large image collection. Image modality detection has been of research interest for querying multimodal medical documents. To accurately predict the different image modalities using complex visual and textual features, we need advanced classification schemes with supervised learning mechanisms and accurate training labels. Our proposed method, on the other hand, uses a multiview-approach and requires minimal expert knowledge to semi-automatically label the images. The images are first projected in different feature spaces, and are then clustered in an unsupervised manner. Only the cluster representative images are labeled by an expert. Other images from the cluster "inherit" the labels from these cluster representatives. The final label assigned to each image is based on a voting mechanism, where each vote is derived from different feature space clustering. Through experiments we show that using only 0.3% of the labels was sufficient to annotate 300,000 medical images with 49.95% accuracy. Although, automatic labeling is not as precise as manual, it saves approximately 700 hours of manual expert labeling, and may be sufficient for next-stage classifier training. We find that for this collection accuracy improvements are feasible with better disparate feature selection or different filtering mechanisms.
2013 IEEE International Conference on Healthcare Informatics, 2013
The National Library of Medicine (NLM) has developed People Locator TM (PL), a Web-based system f... more The National Library of Medicine (NLM) has developed People Locator TM (PL), a Web-based system for family reunification in cases of a natural or man-made disaster. PL accepts photos and brief text meta-data (name, age, etc.) of missing or found persons. Searchers may query PL with text information, but text data is often incomplete or inconsistent. Adding an image-based search capability, i.e., matching faces in query photos to those already stored in the system, would significantly benefit the user experience. We report on our face matching R&D that aims to provide robust face localization and matching on digital photos of variable quality. In this article, we review relevant research and present our approach to robust nearduplicate image detection as well as face matching. We describe the integration of our face matching system with PL, report on its performance, and compare it to other publicly available face recognition systems. In contrast to these systems that have many good quality well-illuminated sample images for each person, our algorithms are hampered by the lack of training examples for individual faces, as those are unlikely in a disaster setting.
2013 12th International Conference on Document Analysis and Recognition, 2013
Lampung Script is a non-cursive script where a rich set of diacritics is used to modify the sylla... more Lampung Script is a non-cursive script where a rich set of diacritics is used to modify the syllable denoted by a character symbol. Consequently, the analysis of the relation between characters and diacritic marks associated with them plays an important role in the recognition process. As diacritics can appear in three different relative positions with respect to a character (top, bottom, and right) associating them correctly with a character is a challenging problem. In this paper we propose a novel approach for modeling the relations between characters and diacritics in handwritten Lampung documents. First, a document is segmented into characters and diacritic marks. Then every character defines a normalized coordinate system into which nearby diacritics can be mapped. The relation between a diacritic mark and its associated character can then be described by a statistical model. In a writer independent experimental evaluation we investigate models with different degrees of specialization with respect to their capability of predicting the correct character-todiacritic associations. We achieve significant error rate reductions with respect to a naive association model using a nearest-neighbor criterion.
In this paper a fast data driven learning-corpus building algorithm (FDDLCB) is proposed. The gen... more In this paper a fast data driven learning-corpus building algorithm (FDDLCB) is proposed. The generic technique allows to build dynamically a representative and compact learning corpus for a connectionist model. The constructed dataset contains just a reduced number of patterns but sufficiently descriptive to characterize the different classes which should be separated. The method is based on a double least mean squares (LMS) error minimization mechanism trying to find the optimal boundaries of the different pattern classes. In the classical learning process the LMS is serving to minimize the error during the learning and this process is improved with a second one, as the new samples selection is also based on the idea to minimize the recognition error. Reinforcing the class boundaries where the recognition fails let us achieve a rapid and good generalization without any loss of occuracy. A modified version of the algorithm will be also presented. The experiments were performed on MNIST 1 separated digit dataset. The encouraging result (98.51%) using just 1.85% of the available patterns from the original training dataset is comparable even with the state of the art techniques.
The recognition of mind maps written on a whiteboard is a challenging task due to the unconstrain... more The recognition of mind maps written on a whiteboard is a challenging task due to the unconstrained handwritten text and the different graphical elements -i.e. lines, circles and arrows -available for drawing a mind map. In this paper we propose a prototype system to recognize and visualize such mind maps written on a whiteboard. After the image acquisition by a camera, a binarization process is performed, and the different connected components are extracted. Without presuming any prior knowledge about the document, its style, layout, etc., the analysis starts with connected components, labeling them as text, lines, circles or arrows based on a neural network classifier trained on some statistical features extracted from the components. Once the text patches are identified, word detection is performed, modeling the text patches by their gravity centers and grouping them into possible words by a density based clustering. Finally, the grouped connected components are recognized by a Hidden Markov Model based recognizer. The paper also presents a software tool integrating all these processing stages, allowing a digital transcription of the mind map and the interaction between the user, the mind map, and the whiteboard.
Abstract: In this report we describe a special multi-layer perceptron wich was trained on the ima... more Abstract: In this report we describe a special multi-layer perceptron wich was trained on the image database in order to obtain the classification of the test database. We used the same database to train for each class an SVM and after that we used a normalization process in order to be able to compare the different results given by the SVMs on the test database, as they are not n-class classifiers. As the results obtained were not enough satisfactory for our industrial partner, we proposed a combination scheme, in order to interact the different ...
Résumé: In this report it is described the characterization of the image database containing char... more Résumé: In this report it is described the characterization of the image database containing characters in different sizes, different orientations and in the mean time written with different font types, coming from our industrial partner. In order to normalize the images we used the Goshtasby transform, which is a size and rotation invariant transform and works with shape matrices. In order to capture in the shapes more local information from the images, we proposed a modification of the transformation.
Résumé: In this report we describe two widely used combinational methods: stacked generalization ... more Résumé: In this report we describe two widely used combinational methods: stacked generalization and respectively the behavior knowledge space in order to improve the recognition scores obtained with the multi-layer perceptron and respectively the support vector machines. By the combination of these two classifiers we achieved good results, taking in consideration the drawbacks, in the database were low represented classes, and in the mean time the quality of the images was not sufficiently good. By introducing some ...
If an image does not correspond to one of these rules then the pattern is considered as being rej... more If an image does not correspond to one of these rules then the pattern is considered as being rejected. The purpose of the method proposed is to use Viterbi paths extracted from the NSHP-HMM model and to show how it can improve the relevance of the recognition.
Résumé: In this paper a fast data driven learning-corpus building algorithm (FDDLCB) is proposed.... more Résumé: In this paper a fast data driven learning-corpus building algorithm (FDDLCB) is proposed. The generic technique allows to build dynamically a representative and compact learning corpus for a connectionist model. The constructed dataset contains just a reduced number of patterns but sufficiently descriptive to characterize the different classes which should be separated. The method is based on a double least mean squares (LMS) error minimization mechanism trying to find the optimal boundaries of the different pattern ...
For training supervised classifiers to recognize different patterns, large data collections with ... more For training supervised classifiers to recognize different patterns, large data collections with accurate labels are necessary. In this paper, we propose a generic, semi-automatic labeling technique for large handwritten character collections. In order to speed up the creation of a large scale ground truth, the method combines unsupervised clustering and minimal expert knowledge. To exploit the potential discriminant complementarities across features, each character is projected into five different feature spaces. After clustering the images in each feature space, the human expert labels the cluster centers. Each data point inherits the label of its cluster's center. A majority (or unanimity) vote decides the label of each character image. The amount of human involvement (labeling) is strictly controlled by the number of clusters -produced by the chosen clustering approach. To test the efficiency of the proposed approach, we have compared, and evaluated three state-of-the art clustering methods (k-means, self-organizing maps, and growing neural gas) on the MNIST digit data set, and a Lampung Indonesian character data set, respectively. Considering a k-nn classifier, we show that labeling manually only 1.3% (MNIST), and 3.2% (Lampung) of the training data, provides the same range of performance than a completely labeled data set would.
A domain where, even in the era of electronic document processing, handwriting is still widely us... more A domain where, even in the era of electronic document processing, handwriting is still widely used is note-taking on a whiteboard. Such documents are either captured by a pen-tracking device or -which is much more challengingby a camera. In both cases the layout analysis of realistic whiteboard notes is an open research problem.
Statistical modelling techniques for automatic reading systems substantially rely on the availabi... more Statistical modelling techniques for automatic reading systems substantially rely on the availability of compact and meaningful feature representations. State-ofthe-art feature extraction for offline handwriting recognition is usually based on heuristic approaches that describe either basic geometric properties or statistical distributions of raw pixel values. Working well on average, still fundamental insights into the nature of handwriting are desired. In this paper we present a novel approach for the automatic extraction of appearance-based representations of offline handwriting data. Given the framework of deep belief networks -Restricted Boltzmann Machines -a two-stage method for feature learning and optimization is developed. Given two standard corpora of both Arabic and Roman handwriting data it is demonstrated across script boundaries, that automatically learned features achieve recognition results comparable to state-of-theart handcrafted features. Given these promising results the potential of feature learning for future reading systems is discussed.
The recognition of handwritten characters is an almost solved problem thanks to efficient machine... more The recognition of handwritten characters is an almost solved problem thanks to efficient machine learning techniques. However, the evaluation and the choice of thresholds to meet a certain level of performance remains a challenge. In this paper, we compare different rejection techniques to determine if a character has been successfully detected or not. Whereas the evaluation of binary classifiers through ROC curves and cost curves has been largely exploited in the literature, multiclass problems can involve different issues like the computational cost or the choice of having an adaptive threshold for each class. The proposed methods for rejection criteria include the maximization of the distance to the optimal performance in the ROC space. We show that optimizing this criterion is a suitable approach on two databases: Lampung handwritten characters, and Arabic handwritten digits. The results support the conclusion that in database where the accuracy is high, pairwise ROC curves analysis in multi-class problems can lead to a finer evaluation of the performance and thresholds definition.
The recognition of fully multi-oriented handwritten characters is a challenging problem. Contrary... more The recognition of fully multi-oriented handwritten characters is a challenging problem. Contrary to univariate signals where the shift invariance property in the Fourier transform can be used, multivariate signals like images require special care to extract rotation invariant features. Several strategies to solve such classification tasks are possible. The proposed method considers input features obtained by the Radon transform or Polar transform. A convolutional neural network is then used for extracting higher level features. This classifier includes in addition the Fast Fourier Transform for extracting shift invariant features at the neural network level. The Radon transform and the convolutional layers process the image at the pixel level while the Fourier transform and the upper layers of the neural network process rotation invariant features. The classifier is evaluated on multi-oriented handwritten digits based on the MNIST database (Arabic digits) and on the ISI database (Bangla digits). The average recognition rate for multi-oriented characters is 93.10% for the Arabic digits and 77.01% for the Bangla digits. This neural architecture highlights the interest of the radial convolutional layer for the recognition of multi-oriented shapes.
2014 IEEE Symposium on Computational Intelligence in Healthcare and e-health (CICARE), 2014
In this paper we present a fast and effective method for labeling images in a large image collect... more In this paper we present a fast and effective method for labeling images in a large image collection. Image modality detection has been of research interest for querying multimodal medical documents. To accurately predict the different image modalities using complex visual and textual features, we need advanced classification schemes with supervised learning mechanisms and accurate training labels. Our proposed method, on the other hand, uses a multiview-approach and requires minimal expert knowledge to semi-automatically label the images. The images are first projected in different feature spaces, and are then clustered in an unsupervised manner. Only the cluster representative images are labeled by an expert. Other images from the cluster "inherit" the labels from these cluster representatives. The final label assigned to each image is based on a voting mechanism, where each vote is derived from different feature space clustering. Through experiments we show that using only 0.3% of the labels was sufficient to annotate 300,000 medical images with 49.95% accuracy. Although, automatic labeling is not as precise as manual, it saves approximately 700 hours of manual expert labeling, and may be sufficient for next-stage classifier training. We find that for this collection accuracy improvements are feasible with better disparate feature selection or different filtering mechanisms.
2013 IEEE International Conference on Healthcare Informatics, 2013
The National Library of Medicine (NLM) has developed People Locator TM (PL), a Web-based system f... more The National Library of Medicine (NLM) has developed People Locator TM (PL), a Web-based system for family reunification in cases of a natural or man-made disaster. PL accepts photos and brief text meta-data (name, age, etc.) of missing or found persons. Searchers may query PL with text information, but text data is often incomplete or inconsistent. Adding an image-based search capability, i.e., matching faces in query photos to those already stored in the system, would significantly benefit the user experience. We report on our face matching R&D that aims to provide robust face localization and matching on digital photos of variable quality. In this article, we review relevant research and present our approach to robust nearduplicate image detection as well as face matching. We describe the integration of our face matching system with PL, report on its performance, and compare it to other publicly available face recognition systems. In contrast to these systems that have many good quality well-illuminated sample images for each person, our algorithms are hampered by the lack of training examples for individual faces, as those are unlikely in a disaster setting.
2013 12th International Conference on Document Analysis and Recognition, 2013
Lampung Script is a non-cursive script where a rich set of diacritics is used to modify the sylla... more Lampung Script is a non-cursive script where a rich set of diacritics is used to modify the syllable denoted by a character symbol. Consequently, the analysis of the relation between characters and diacritic marks associated with them plays an important role in the recognition process. As diacritics can appear in three different relative positions with respect to a character (top, bottom, and right) associating them correctly with a character is a challenging problem. In this paper we propose a novel approach for modeling the relations between characters and diacritics in handwritten Lampung documents. First, a document is segmented into characters and diacritic marks. Then every character defines a normalized coordinate system into which nearby diacritics can be mapped. The relation between a diacritic mark and its associated character can then be described by a statistical model. In a writer independent experimental evaluation we investigate models with different degrees of specialization with respect to their capability of predicting the correct character-todiacritic associations. We achieve significant error rate reductions with respect to a naive association model using a nearest-neighbor criterion.
In this paper a fast data driven learning-corpus building algorithm (FDDLCB) is proposed. The gen... more In this paper a fast data driven learning-corpus building algorithm (FDDLCB) is proposed. The generic technique allows to build dynamically a representative and compact learning corpus for a connectionist model. The constructed dataset contains just a reduced number of patterns but sufficiently descriptive to characterize the different classes which should be separated. The method is based on a double least mean squares (LMS) error minimization mechanism trying to find the optimal boundaries of the different pattern classes. In the classical learning process the LMS is serving to minimize the error during the learning and this process is improved with a second one, as the new samples selection is also based on the idea to minimize the recognition error. Reinforcing the class boundaries where the recognition fails let us achieve a rapid and good generalization without any loss of occuracy. A modified version of the algorithm will be also presented. The experiments were performed on MNIST 1 separated digit dataset. The encouraging result (98.51%) using just 1.85% of the available patterns from the original training dataset is comparable even with the state of the art techniques.
The recognition of mind maps written on a whiteboard is a challenging task due to the unconstrain... more The recognition of mind maps written on a whiteboard is a challenging task due to the unconstrained handwritten text and the different graphical elements -i.e. lines, circles and arrows -available for drawing a mind map. In this paper we propose a prototype system to recognize and visualize such mind maps written on a whiteboard. After the image acquisition by a camera, a binarization process is performed, and the different connected components are extracted. Without presuming any prior knowledge about the document, its style, layout, etc., the analysis starts with connected components, labeling them as text, lines, circles or arrows based on a neural network classifier trained on some statistical features extracted from the components. Once the text patches are identified, word detection is performed, modeling the text patches by their gravity centers and grouping them into possible words by a density based clustering. Finally, the grouped connected components are recognized by a Hidden Markov Model based recognizer. The paper also presents a software tool integrating all these processing stages, allowing a digital transcription of the mind map and the interaction between the user, the mind map, and the whiteboard.
Abstract: In this report we describe a special multi-layer perceptron wich was trained on the ima... more Abstract: In this report we describe a special multi-layer perceptron wich was trained on the image database in order to obtain the classification of the test database. We used the same database to train for each class an SVM and after that we used a normalization process in order to be able to compare the different results given by the SVMs on the test database, as they are not n-class classifiers. As the results obtained were not enough satisfactory for our industrial partner, we proposed a combination scheme, in order to interact the different ...
Résumé: In this report it is described the characterization of the image database containing char... more Résumé: In this report it is described the characterization of the image database containing characters in different sizes, different orientations and in the mean time written with different font types, coming from our industrial partner. In order to normalize the images we used the Goshtasby transform, which is a size and rotation invariant transform and works with shape matrices. In order to capture in the shapes more local information from the images, we proposed a modification of the transformation.
Résumé: In this report we describe two widely used combinational methods: stacked generalization ... more Résumé: In this report we describe two widely used combinational methods: stacked generalization and respectively the behavior knowledge space in order to improve the recognition scores obtained with the multi-layer perceptron and respectively the support vector machines. By the combination of these two classifiers we achieved good results, taking in consideration the drawbacks, in the database were low represented classes, and in the mean time the quality of the images was not sufficiently good. By introducing some ...
If an image does not correspond to one of these rules then the pattern is considered as being rej... more If an image does not correspond to one of these rules then the pattern is considered as being rejected. The purpose of the method proposed is to use Viterbi paths extracted from the NSHP-HMM model and to show how it can improve the relevance of the recognition.
Résumé: In this paper a fast data driven learning-corpus building algorithm (FDDLCB) is proposed.... more Résumé: In this paper a fast data driven learning-corpus building algorithm (FDDLCB) is proposed. The generic technique allows to build dynamically a representative and compact learning corpus for a connectionist model. The constructed dataset contains just a reduced number of patterns but sufficiently descriptive to characterize the different classes which should be separated. The method is based on a double least mean squares (LMS) error minimization mechanism trying to find the optimal boundaries of the different pattern ...
Uploads
Papers by Szilárd Vajda