Minimum description length

We propose an unprecedented approach to post-hoc interpretable machine learning. Facing a complex phenomenon, rather than fully capturing its mechanisms through a universal learner, albeit structured in modular building blocks, we train a robust neural network, no matter its complexity, to use as an oracle. Then we approximate its behavior via a linear combination of simple, explicit functions of its input. Simplicity is achieved by (i) marginal functions mapping individual inputs to the network output, (ii) the same consisting of univariate polynomials with a low degree,(iii) a small number of polynomials being involved in the linear combination, whose input is properly granulated. With this contrivance, we handle various real-world learning scenarios arising from expertise and experimental frameworks' composition. They range from cooperative training instances to transfer learning. Concise theoretical considerations and comparative numerical experiments further detail and support the proposed approach .

In this paper, we propose a novel information criteria-based approach to select the dimensionality of the word2vec Skip-gram (SG). From the perspective of the probability theory, SG is considered as an implicit probability distribution estimation under the assumption that there exists a true contextual distribution among words. Therefore, we apply information criteria with the aim of selecting the best dimensionality so that the corresponding model can be as close as possible to the true distribution. We examine the following information criteria for the dimensionality selection problem: the Akaike’s Information Criterion (AIC), Bayesian Information Criterion (BIC), and Sequential Normalized Maximum Likelihood (SNML) criterion. SNML is the total codelength required for the sequential encoding of a data sequence on the basis of the minimum description length. The proposed approach is applied to both the original SG model and the SG Negative Sampling model to clarify the idea of using...

Curve evolution implementations [3][17] [18] of the Mumford-Shah functional are of broad interest in image segmentation. These implementations, however, have initialization problems . A mathematical analysis of the initialization problem for the bi-modal Chan-Vese model [3] is provided in this paper. The initialization problem is a result of the nonconvexity of the Mumford-Shah functional and the top-down hierarchy of the model's use of global region information in the image. An efficient image segmentation method is proposed that alleviates the initialization problem, based on region growing, region competition and the Mumford Shah functional . This algorithm is able to automatically and efficiently segment objects in complicated images. Using a bottom-up hierarchy, the method avoids the initialization problem in the Chan-Vese model and works for images with multiple junctions and color images. It can be extended to textured images. Experimental results show that the proposed method is robust to the effects of noise.

Continuation processes in chemical and/or biotechnical plants always generate a large amount of time series data. However, since conventional process models are described as a set of control models, it is difficult to explain the complicated and active plant behaviours. Based on the background, this paper proposes a novel method to develop a process response model from continuous time-series data. The method consists of the following phases: (1) reciprocal correlation analysis, (2) process response model, (3) extraction of control rules, (4) extraction of a workflow and (5) detecting outliers. The main contribution of the research is to establish a method to mine a set of meaningful control rules from Learning Classifier System (LCS) using the Minimum Description Length (MDL) criteria and Tabu search method. The proposed method has been applied to an actual process of a biochemical plant and has shown the validity and the effectiveness.

Junctions are important features for image analysis and form a critical aspect of image understanding tasks such as object recognition. We present a unified approach to detecting (location of the center of the junction), classifying (by the number of wedges-lines, corners, three-junctions such as T or Y junctions, or four-junctions such as X-junctions), and reconstructing junctions (in terms of radius size, the angles of each wedge and the intensity in each of the wedges) in images. Our main contribution is a modeling of the junction which is complex enough to handle all these issues and yet simple enough to admit an effective dynamic programming solution. Broadly, we use a template deformation framework along with a gradient criterium to detect radial partitions of the template. We use the minimum description length principle to obtain the optimal number of partitions that best describes the junction. Kona [27] is an implementation of this model. We (quantitatively) demonstrate the stability and robustness of the detector by analyzing its behavior in the presence of noise, using synthetic/controlled apparatus. We also present a qualitative study of its behavior on real images.

No statistical model is right or wrong, true or false in a strict sense. We only evaluate and compare their contributions. Based on this theme, Jorma Rissanen has written a short but beautiful book titled "Information and Complexity in Statistical Modeling" (Springer, 2007), where modeling is done primarily by extracting the information from the data that can be learned with suggested classes of probability models. The note reviews this book and on the way rediscovers the chain information-knowledge-wisdom.

The contribution of this work is the creation and development of a novel procedure for signals estimation, procedure LDM-G. The procedure will be developed with details and will consist of algorithms implementation for signals estimation via wavelet transform, applying the minimum description length principle for simultaneous and automatic selection of wavelet base and threshold value. By means of simulations, the implemented procedure was compared with techniques of universal threshold and Stein's SURE, where the effectiveness of the elaborated procedure was verified by means of the estimation of mean square error.

We present a method for detecting repeated structures, which is applied on facade images for describing the regularity of their windows. Our approach finds and explicitly represents repetitive structures and thus gives initial representation of facades. No explicit notion of a window is used; thus, the method also appears to be able to identify other manmade structures, e.g., paths with regular tiles. A method for detection of dominant symmetries is adapted for detection of multiply repeated structures. A compact description of the repetitions is derived from the detected translations in the image by a heuristic search method and the criterion of the minimum description length.

We present a method for detecting repeated structures, which is applied on facade images for describing the regularity of their windows. Our approach finds and explicitly represents repetitive structures and thus gives initial representation of facades. No explicit notion of a window is used, thus the method also appears to be able to identify other man made structures, e.g. paths with regular tiles. A method for detection of dominant symmetries is adapted for detection of multiple repeated structures. A compact description of repetitions is derived from translations detected in an image by a heuristic search method and the model selection criterion of the minimum description length. : Erkennung und Beschreibung wiederholter Strukturen in entzerrten Fassadenbildern. In dieser Arbeit wird eine Methode zur Detektion wiederholter Strukturen vorgestellt. Das Verfahren wird auf Fassadenbildern angewendet, um daraus eine Beschreibung der Regelmäßigkeiten der Fenster abzuleiten. Diese Beschreibung definiert eine kompakte Repräsentation der Fassadenstruktur. Es wird kein Vorwissen über Fenster eingeführt, so dass die Methode direkt auf andere von Menschen erzeugte Strukturen übertragbar ist. Zunächst wird eine Methode zur Detektion dominanter Symmetrien in Bildern vorgestellt und daraus im Anschluss ein Verfahren zur Detektion multipler wiederholter Strukturen abgeleitet. Aus den beobachteten Translationen im Bild wird die kompakte Beschreibung der Fassadenstruktur abgeleitet, über ein Suchverfahren und das Modellauswahlkriterium der minimalen Beschreibungslänge.

We present a method for detecting repeated structures, which is applied on facade images for describing the regularity of their windows. Our approach finds and explicitly represents repetitive structures and thus gives initial representation of facades. No explicit notion of a window is used; thus, the method also appears to be able to identify other manmade structures, e.g., paths with regular tiles. A method for detection of dominant symmetries is adapted for detection of multiply repeated structures. A compact description of the repetitions is derived from the detected translations in the image by a heuristic search method and the criterion of the minimum description length.

Some model-selection criteria for choosing among a set of alternative models are reviewed. Particular modelselection problems considered here include the choice of a regression equation for prediction, choice of the number of bins for a histogram, and choice of the number of component p.d.f.s in a finite mixture model. Several general methods of scoring the choices in such problems are considered. Minimum description length and penalized likelihood criteria are discussed, in particular AIC (Akaike's Information Criterion), BIC (Bayesian Information Criterion) and KIC (Kashyap's Information Criterion). Interpretation of BIC and KIC in terms of posterior probabilities of alternative models is given. Averaging a prediction or classification over models is considered.

Because many databases contain or can be embellished with structural information, a method for identifying interesting and repetitive substructures is an essential component to discovering knowledge in such databases. This paper describes the SUBDUE system, which uses the minimum description length (MDL) principle to discover substructures that compress the database and represent structural concepts in the data. By replacing previously-discovered substructures in the data, multiple passes of SUBDUE produce a hierarchical description of the structural regularities in the data. Inclusion of background knowledgeguides SUBDUE toward appropriate substructures for a particular domain or discovery goal, and the use of an inexact graph match allows a controlled amount of deviations in the instance of a substructure concept. We describe the application of SUBDUE to a variety of domains. We also discuss approaches to combining SUBDUE with non-structural discovery systems.

The ability to identify interesting and repetitive substructures is an essential component to discovering knowledge in structural data. We describe a new version of our SUBDUE substructure discovery system based on the minimum description length principle. The SUBDUE system discovers substructures that compress the original data and represent structural concepts in the data. By replacing previously-discovered substructures in the data, multiple passes of SUBDUE produce a hierarchical description of the structural regularities in the data. SUBDUE uses a computationally-bounded inexact graph match that identifies similar, but not identical, instances of a substructure and finds an approximate measure of closeness of two substructures when under computational constraints. In addition to the minimum description length principle, other background knowledge can be used by SUBDUE to guide the search towards more appropriate substructures. Experiments in a variety of domains demonstrate SUB...

Because many databases contain or can be embellished with structural information, a method for identifying interesting and repetitive substructures is an essential component to discovering knowledge in such databases. This paper describes the SUBDUE system, which uses the minimum description length (MDL) principle to discover substructures that compress the database and represent structural concepts in the data. By replacing previously-discovered substructures in the data, multiple passes of SUBDUE produce a hierarchical description of the structural regularities in the data. Inclusion of background knowledgeguides SUBDUE toward appropriate substructures for a particular domain or discovery goal, and the use of an inexact graph match allows a controlled amount of deviations in the instance of a substructure concept. We describe the application of SUBDUE to a variety of domains. We also discuss approaches to combining SUBDUE with non-structural discovery systems.

Scene geometry can be inferred from point correspondences between two images. The inference process includes the selection of a model. Four models are considered: background (or null), collineation, affine fundamental matrix and fundamental matrix. It is shown how Minimum Description Length (MDL) can be used to compare the different models. The main result is that there is little reason for preferring the fundamental matrix model over the collineation model, even when the former the 'true' model.

This work plants a method of estimate of harmful in structure by no lineal signals separation, based in wavelet transform. The obtained results are superior in capacity of harmful estimate for reported in the literature. By system acceleration register and wavelet transform, the proposed method permits to estimate the instant and the amplitude of any structural failing

Accelerograms are the random time series representation of the strong ground motions called earthquake. Processing of accelerograms is done before using them for any seismic and engineering applications. This article focuses the impact of threshold and noise in wavelet processing using experiments on random generated time series signals. Experiments conducted on the global thresholds currently in use lead us to propose a novel fuzzy based threshold to remove the random noise. Therefore a fuzzy based threshold determination in the domain of wavelet coefficients has been proposed and effectively used in denoising the real observed accelerograms. Also, the experimental results on synthetic signals has outperformed the existing universal and sure thresholds in terms of SNR of 20.3 and RMSE of 0.0019, respectively.

Combining (i) a statistical interpretation of the minimum of a Weighted Least Squares cost function and (ii) the principle of parsimony, a model selection strategy is proposed. First, it is compared via simulation to model selection methods based on information criteria (AIC and MDL type). The first kind of simulations shows that the cost function approach outperforms in selecting the true model, especially when the number of data is very small compared with the number of parameters to be estimated. Next, the model metaselection proposed by de Luna and Skouras [X. De Luna, K. Skouras, Choosing a model selection strategy, Scand. J. Stat. 30(1) (2003) 113-128.] is employed as an objective method to choose the best model selection method. Applied to one of their examples, clearly the cost function strategy is selected as the best method. Finally, on a set of field data, the cost function approach is used for selecting the relevant parameters of a complex model.

An approach is presented to automatically segment and label a continuous observation sequence of hand gestures for a complete unsupervised model acquisition. The method is based on the assumption that gestures can be viewed as repetitive sequences of atomic components, similar to phonemes in speech, starting and ending in a rest position and governed by a high level structure controlling the temporal sequence. It is shown that the generating processes for the atomic components and derived gesture models can be described by a mixture of Gaussian in their respective component and gesture space. Mixture components modelling atomic components and gestures respectively are determined using a standard EM approach, while the determination of the number of mixture components and therefore the number of atomic components and gestures is based on an information criterion, the Minimum Description Length (MDL).

By a "covering" we mean a Gaussian mixture model fit to observed data. Approximations of the Bayes factor can be availed of to judge model fit to the data within a given Gaussian mixture model. Between families of Gaussian mixture models, we propose the Rényi quadratic entropy as an excellent and tractable model comparison framework. We exemplify this using the segmentation of an MRI image volume, based (1) on a direct Gaussian mixture model applied to the marginal distribution function, and (2) Gaussian model fit through k-means applied to the 4D multivalued image volume furnished by the wavelet transform. Visual preference for one model over another is not immediate. The Rényi quadratic entropy allows us to show clearly that one of these modelings is superior to the other.

We study the problem of estimating the overall mutual information in M independent parallel discrete-time memory-less Gaussian channels from N independent data sample pairs per channel (inputs and outputs). We focus on the case where the number of active channels L is sparse in comparison with the total number of channels (L M), for which the direct application of the maximum likelihood principle is problematic due to overfitting, especially for moderate to small N. For this regime, we show that the bias of the mutual information estimate is reduced by resorting to the minimum description length (MDL) principle. As a result, simple pre-processing based on a perchannel threshold on the empirical squared correlation coefficient is required with a fixed threshold that monotonically decreases with N as 1 − N −1/N , for N ≥ 4. The resulting improvement is shown in terms of the estimated information bias.

In this work, we propose a Compression Rate Distance, a new distance measure for time series data. The main idea behind this distance is based on the Minimum Description Length (MDL) principle. The higher compression rate between two time series is, the closer they should be. Besides, we also propose a relaxed version of the new distance, called the Extended Compression Rate Distance. The Extended Compression Rate Distance can satisfy some crucial characteristics on time series such as Early Abandoning, Lower Bounding, and Relaxed-Triangular Inequality which help the new distance easily adapt with traditional indexing structures and searching methods. We tested our distances on classification problem with numerous datasets and compared the results with most of the commonly used distances in time series such as Euclidean Distance, Dynamic Time Warping, and a recently proposed Complexity-Invariant Distance. Experimental results reveal that our novel distances outperform several previous important distance measures in a vast majority of the datasets.

Wavelet transforms enable us to represent signals with a high degree of sparsity. This is the principle behind a non-linear wavelet based signal estimation technique known as wavelet denoising. In this report we explore wavelet denoising of images using several ...

The design of an architecture for a family of products is a challenging task. Traditionally, only a single product case is considered when optimization techniques are used to define the optimum clustering of the components, or in other words the module definition for the product. This article proposes a framework to analyse multiple products of a given family and achieve an optimal module definition considering component sharing across some or all the products. The optimal sharing decision process is implemented with a dual objective function: a 'minimum description length' representation of the overall product architecture, and the 'impact metric' description of the selected components to share. An illustration example is developed for three digital cameras of the same brand and the results reveal the recommended set of components to share across the family as well as the optimal architecture for each one of the products.

We present an approach to automatically segment and label a continuous observation sequence of hand gestures for a complete unsupervised model acquisition. The method is based on the assumption that gestures can be viewed as repetitive sequences of atomic components, similar to phonemes in speech, governed by a high level structure controlling the temporal sequence. We show that the generating process for the atomic components can be described in gesture space by a mixture of Gaussian, with each mixture component tied to one atomic behavioul: Mixture components are determined using a standard EM approach while the determination of the number of components is based on an information criteria, the Minimum Description Length.

An approach is presented to automatically segment and label a continuous observation sequence of hand gestures for a complete unsupervised model acquisition. The method is based on the assumption that gestures can be viewed as repetitive sequences of atomic components, similar to phonemes in speech, starting and ending in a rest position and governed by a high level structure controlling the temporal sequence. It is shown that the generating processes for the atomic components and derived gesture models can be described by a mixture of Gaussian in their respective component and gesture space. Mixture components modelling atomic components and gestures respectively are determined using a standard EM approach, while the determination of the number of mixture components and therefore the number of atomic components and gestures is based on an information criterion, the Minimum Description Length (MDL).

An approach is presented to automatically segment and label a continuous observation sequence of hand gestures for a complete unsupervised model acquisition. The method is based on the assumption that gestures can be viewed as repetitive sequences of atomic components, similar to phonemes in speech, starting and ending in a rest position and governed by a high level structure controlling the temporal sequence. It is shown that the generating processes for the atomic components and derived gesture models can be described by a mixture of Gaussian in their respective component and gesture space. Mixture components modelling atomic components and gestures respectively are determined using a standard EM approach, while the determination of the number of mixture components and therefore the number of atomic components and gestures is based on an information criterion, the Minimum Description Length (MDL).

We consider the perturbed harmonic oscillator T D ψ = −ψ ′′ + x 2 ψ + q(x)ψ, ψ(0) = 0 in L 2 (R +), where q ∈ H + = {q ′ , xq ∈ L 2 (R +)} is a real-valued potential. We prove that the mapping q → spectral data = {eigenvalues of T D } ⊕ {norming constants} is one-to-one and onto. The complete characterization of the set of spectral data which corresponds to q ∈ H + is given. Moreover, we solve the similar inverse problem for the family of boundary conditions ψ ′ (0) = bψ(0), b ∈ R.

We consider a new class of information sources called wordvalued sources in order to investigate coding algorithms based upon string parsing. A word-valued source is defined as a pair of an independent and identically distributed (i.i.d.) source with a countable alphabet and a function that maps each symbol into a finite sequence over a finite alphabet. A word-valued source is a nonstationary process and has countable states. If the function of a word-valued source is prefix-free, the entropy rate is characterized with a simple expression and the AEP (Asymptotic Equipartition Property) holds.

Universal coding for the Slepian-Wolf data compression system is considered. We shall demonstrate based on a simple observation that the error exponent given by Csiszhr and Komer for the universal coding system can strictly be sharpened in general for a region of relatively higher rates. This kind of observation can be carried over also to the case of lower rates outside the Slepian-Wolf region, which establishes the strong converse along with the optimal exponent. Index Terms-Separate coding, correlated sources, universal coding, error exponents, strong converse theorem.

There exists a substantial problem in obtaining good generalisation performance in the application of arttficial neural network technology where training data is limited. Generalisation ability is analysed for a number of computational paradigms which attempt to alleviate this for the multilayer perceptron. The problem of line detection in a time~frequency sonar image or "lofargram' is adopted as a case study on which to assess these techniques. The effect on neural network generalisation performance is studied for (a) heuristically changing the number of hidden nodes, (b) weight decay, (c) soft weight-sharing, and (d) Ockham's networks. These techniques are introduced from the perspective of the Minimum Description Length principle. Results show that the use of weight decay and Ockham's networks are able to improve generalisation beyond that available by simply altering the number of hidden nodes. It is shown that line detection in lofargram images is possible at a success rate of 85% for data outside of the training set.

The problem of fitting a model composed of a number of superimposed signals to noisy data using the maximum likelihood criterion is considered. It is shown, using the Cra-m&-Rao bound for the estimation accuracy, that in many instances useful models for the composite signal can be restricted without loss of generality to component signals that directly interact only with one or two of their closest neighbors in parameter space. It is shown that for such models, the global extremum of the criterion can be found efficiently by dynamic programming. The computation requirements are linear in the number of signals, rather than exponential as in the case of exhaustive search. For example, for sinusoids the computation is O(mq6), where m is their number, and q is the resolution frequency, amplitude, and phase. The technique applies for arbitrary sampling of the signals. The dynamic programming method is easily adapted to determine the number of signals as well, as demonstrated using the minimum description length (MDL) principle. Computer simulation results of several examples are presented. Sze Fong Yau received the B.S. degree from the

We develop a code length principle which is invariant to the choice of parameterization on the model distributions. An invariant approximation formula for easy computation of the marginal distribution is provided for gaussian likelihood models. We provide invariant estimators of the model parameters and formulate conditions under which these estimators are essentially posteriori unbiased for gaussian models. An upper bound on the coarseness of discretization on the model parameters is deduced. We introduce a discrimination measure between probability distributions and use it to construct probability distributions on model classes. The total code length is shown to equal the NML code length of Rissanen to within an additive constant when choosing Jeffreys prior distribution on the model parameters together with a particular choice of prior distribution on the model classes. Our model selection principle is applied to a gaussian estimation problem for data in a wavelet representation and its performance is tested and compared to alternative wavelet-based estimation methods in numerical experiments.

Standard system identification methods often provide biased estimates with closed-loop data. With the prediction error method (PEM), the bias issue is solved by using a noise model that is flexible enough to capture the noise spectrum. However, a too flexible noise model (i.e., too many parameters) can cause additional numerical problems for PEM. In this paper, we perform a theoretical analysis of the weighted null-space fitting (WNSF) method when a parametric noise model is not estimated. With this method, the system is first captured using a non-parametric ARX model, which is then reduced to a parametric model of interest. In the reduction step, a noise model does not need to be estimated if it is not of interest. In open loop, this still provides asymptotically efficient estimates of the dynamic model. In closed loop, the estimates are consistent, and their covariance is optimal for a non-parametric noise model. In this paper, we prove these results, which require additional technical details compared with the case with a full parametric model structure. In particular, we use a geometric approach for variance analysis, deriving a new result that will be instrumental to our end. Finally, we use a simulation study to illustrate the benefits of the method when the noise model cannot be parametrized by a low-order model.

Over the years, ensemble methods have become a staple of machine learning. Similarly, generalized linear models (GLMs) have become very popular for a wide variety of statistical inference tasks. The former have been shown to enhance outof-sample predictive power and the latter possess easy interpretability. Recently, ensembles of GLMs have been proposed as a possibility. On the downside, this approach loses the interpretability that GLMs possess. We show that minimum description length (MDL)-motivated compression of the inferred ensembles can be used to recover interpretability without much, if any, downside to performance and illustrate on a number of standard classification data sets. 1 Introduction Ensemble classifiers have become very popular for classification and regression tasks. They offer the potential advantages of robustness via bootstrapping, feature prioritization, and good out-of-sample performance characteristics (Chitraa [2013]). However, they suffer from lack of interpretability, and oftentimes features are reported as "word bags"-e.g. by feature importance (Liaw [2002]). Generalized linear models, a venerable statistical toolchest, offer good predictive performance across a range of prediction and classification tasks, well-understood theory (advantages and modes of failure) and implementation considerations and, most importantly, excellent interpretability. Until recently, there has been little progress in bringing together ensemble learning and GLMs, but some recent work in this area (e.g. Song [2013]) has resulted in publicly-available implementations of GLM ensembles. Nevertheless, the resulting ensembles of GLMs remain difficult to interpret. Meantime, human understanding of models is pivotal in some fields-e.g. in translational medicine, where machine learning influences drug positioning, clinical trial design, treatment guidelines, and other outcomes that directly influence people's lives. Improvement in performance without interpretability can be useless in such context. To improve performance of maximum-likelihood models, Carvalho [2008] proposed to learn multiple centroids of parameter space. Built bottom-up, such ensembles would have only a limited number of models, keeping the ensemble interpretable. In this paper, we work from a model ensemble down. We demonstrate that minimum description length-motivated ensemble summarization can dramatically improve interpretability of model ensembles with little if any loss of predictive power, and outline some key directions in which these approaches may evolve in the future. 2 Methods 2.1 Theoretical considerations The problem of ML estimators being drawn to dominant solutions is well understood. Likewise, an ensemble consensus can be drawn to the (possibly infeasible) mode, despite potentially capturing the

Pattern mining based on data compression has been successfully applied in many data mining tasks. For itemset data, the Krimp algorithm based on the minimum description length (MDL) principle was shown to be very effective in solving the redundancy issue in descriptive pattern mining. However, for sequence data, the redundancy issue of the set of frequent sequential patterns is not fully addressed in the literature. In this article, we study MDL-based algorithms for mining nonredundant sets of sequential patterns from a sequence database. First, we propose an encoding scheme for compressing sequence data with sequential patterns. Second, we formulate the problem of mining the most compressing sequential patterns from a sequence database. We show that this problem is intractable and belongs to the class of inapproximable problems. Therefore, we propose two heuristic algorithms. The first of these uses a two-phase approach similar to Krimp for itemset data. To overcome performance issues in candidate generation, we also propose GoKrimp, an algorithm that directly mines compressing patterns by greedily extending a pattern until no additional compression benefit of adding the extension into the dictionary. Since checks for additional compression benefit of an extension are computationally expensive we propose a dependency test which only chooses related events for extending a given pattern. This technique improves the efficiency of the GoKrimp algorithm significantly while it still preserves the quality of the set of patterns. We conduct an empirical study on eight datasets to show the effectiveness of our approach in comparison to the state-of-the-art algorithms in terms of interpretability of the extracted patterns, run time, compression ratio, and classification accuracy using the discovered patterns as features for different classifiers.

We propose a streaming algorithm, based on the minimal description length (MDL) principle, for extracting non-redundant sequential patterns. For static databases, the MDL-based approach that selects patterns based on their capacity to compress data rather than their frequency, was shown to be remarkably effective for extracting meaningful patterns and solving the redundancy issue in frequent itemset and sequence mining. The existing MDL-based algorithms, however, either start from a seed set of frequent patterns, or require multiple passes through the data. As such, the existing approaches scale poorly and are unsuitable for large datasets. Therefore, our main contribution is the proposal of a new, streaming algorithm, called Zips, that does not require a seed set of patterns and requires only one scan over the data. For Zips, we extended the Lempel-Ziv (LZ) compression algorithm in three ways: first, whereas LZ assigns codes uniformly as it builds up its dictionary while scanning the input, Zips assigns codewords according to the usage of the dictionary words; more heaviliy used words get shorter code-lengths. Secondly, Zips exploits also non-consecutive occurences of dictionary words for compression. And, third, the well-known space-saving algorithm is used to evict unpromising words from the dictionary. Experiments on one synthetic and two real-world large-scale datasets show that our approach extracts meaningful compressing patterns with similar quality to the state-of-the-art multi-pass algorithms proposed for static databases of sequences. Moreover, our approach scales linearly with the size of data streams while all the existing algorithms do not.

A deep-learning-based approach to estimating the number of coherent sources in radar is presented. A proper estimate of the number of sources in a signal enables improved angle-of-arrival (AoA) estimation common in applications such as radar, sonar, and communication systems. Many AoA estimators utilised in these areas require accurate estimates of the number of sources for enhanced performance. Herein, a robust method that performs well under the existence of coherent sources is developed. The proposed method is based on deep learning and it is shown to perform better than state-of-the-art versions of the Akaike Information Criteria (AIC), Minimum Description Length (MDL), and Exponentially Embedded Families (EEF) estimators, which employ spatial smoothing of the covariance matrix to handle coherent signals. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

Nonlinear time series modeling with a multilayer perceptron network is presented. An important aspect of this modeling is the model selection, i.e., the problem of determining the size as well as the complexity of the model. To overcome this problem we apply the predictive minimum description length (PMDL) principle as a minimization criterion. In the neural network scheme it means minimizing the number of input and hidden units. Three time series modeling experiments are used to examine the usefulness of the PMDL model selection scheme. A comparison with the widely used cross-validation technique is also presented. In our experiments the PMDL scheme and the cross-validation scheme yield similar results in terms of model complexity. However, the PMDL method was found to be two times faster to compute. This is significant improvement since model selection in general is very time consuming.

This paper presents a model-selection strategy based on minimum description length (MDL) that keeps the kernel least-mean-square (KLMS) model tuned to the complexity of the input data. The proposed KLMS-MDL filter adapts its model order as well as its coefficients online, behaving as a self-organizing system and achieving a good compromise between system accuracy and computational complexity without a priori knowledge. Particularly, in a nonstationary scenario, the model order of the proposed algorithm changes continuously with the input data structure. Experiments show the proposed algorithm successfully builds compact kernel adaptive filters with better accuracy than KLMS with sparsity or fixed-budget algorithms.

Reference assisted assembly requires the use of a reference sequence, as a model, to assist in the assembly of the novel genome. The standard method for identifying the best reference sequence for the assembly of a novel genome aims at counting the number of reads that align to the reference sequence, and then choosing the reference sequence which has the highest number of reads aligning to it. This article explores the use of minimum description length (MDL) principle and its two variants, the two-part MDL and Sophisticated MDL, in identifying the optimal reference sequence for genome assembly. The article compares the MDL based proposed scheme with the standard method coming to the conclusion that "counting the number of reads of the novel genome present in the reference sequence" is not a sufficient condition. Therefore, the proposed MDL scheme includes within itself the standard method of "counting the number of reads that align to the reference sequence" and also moves forward towards looking at the model, the reference sequence, as well, in identifying the optimal reference sequence. The proposed MDL based scheme not only becomes the sufficient criterion for identifying the optimal reference sequence for genome assembly but also improves the reference sequence so that it becomes more suitable for the assembly of the novel genome.

Bioinformatics skills required for genome sequencing often represent a significant hurdle for many researchers working in computational biology. This dissertation highlights the significance of genome assembly as a research area, focuses on its need to remain accurate, provides details about the characteristics of the raw data, examines some key metrics, emphasizes some tools and outlines the whole pipeline for next-generation sequencing. Currently, a major effort is being put towards the assembly of the genomes of all living organisms. Given the importance of comparative genome assembly, herein dissertation, the principle of Minimum Description Length (MDL) and its two variants, the Two-Part MDL and Sophisticated MDL, are explored in identifying the optimal reference sequence for genome assembly. Thereafter, a Modular Approach to Reference Assisted Genome Assembly Pipeline, referred to as MARAGAP, is developed. MARAGAP uses the principle of Minimum Description Length (MDL) to determine the optimal reference sequence for the assembly. The optimal reference sequence is used as a template to infer inversions, insertions, deletions and Single Nucleotide Polymorphisms (SNPs) in the target genome. MARA-GAP uses an algorithmic approach to detect and correct inversions and deletions, a De-Bruijn graph based approach to infer insertions, an affine-match affine-gap local alignment tool to estimate the locations of insertions and a Bayesian estimation framework for detecting SNPs (called BECA). BECA effectively capitalizes on the 'alignment-layout-consensus' paradigm and Quality (Q-) values for detecting and correcting SNPs by evaluating a number of probabilistic measures. However, the entire process is conducted once. BECA's framework is further extended by using Gibbs Sampling for further iterations of

A novel genetic algorithm (GA) using minimal representation size cluster (MRSC) analysis is designed and implemented for solving multimodal function optimization problems. The problem of multimodal function optimization is framed within a hypothesize-and-test paradigm using minimal representation size (minimal complexity) for species formation and a GA. A multiple-population GA is developed to identify different species. The number of populations, thus the number of different species, is determined by the minimal representation size criterion. Therefore, the proposed algorithm reveals the unknown structure of the multimodal function when a priori knowledge about the function is unknown. The effectiveness of the algorithm is demonstrated on a number of multimodal test functions. The proposed scheme results in a highly parallel algorithm for finding multiple local minima. In this paper, a path-planning algorithm is also developed based on the MRSC-GA algorithm. The algorithm utilizes ...

XML is rapidly emerging as the new standard for data representation and exchange on the Web. An XML document can be accompanied by a Document Type Descriptor (DTD) which plays the role of a schema for an XML data collection. DTDs contain valuable information on the structure of documents and thus have a crucial role in the efficient storage of XML data, as well as the effective formulation and optimization of XML queries. In this paper, we propose XTRACT, a novel system for inferring a DTD schema for a database of XML documents. Since the DTD syntax incorporates the full expressive power of regular expressions , naive approaches typically fail to produce concise and intuitive DTDs. Instead, the XTRACT inference algorithms employ a sequence of sophisticated steps that involve: (1) finding patterns in the input sequences and replacing them with regular expressions to generate “general” candidate DTDs, (2) factoring candidate DTDs using adaptations of algorithms from the logic optimiza...