Plenty of structural patterns in real world have been represented as graph like molecules, chemic... more Plenty of structural patterns in real world have been represented as graph like molecules, chemical compounds, social network, road network etc. Mining this graph for extracting some useful information is of special interest and has many applications. The application includes drug discovery, compound synthesis, anomaly detection in network, social network analysis for finding groups etc. One of the most interesting problems in graph mining is graph containment problem. In graph containment problem ,given a query graph q ,it is asked to find all graph in given graph dataset containing this query (query graph as subgraph).This means finding all graph which is isomorphic to query graph. As in real world there is vast number of graph in graph dataset so this task of subgraph isomorphism test become tedious, complex, time and space consuming. So it is necessary to create an index of graphs present in dataset for cost efficient query processing. In this paper we proposed a time efficient ...
Graph Indexing using Distributed and Parallelization Approach
Graphs are ubiquitous like chemical compounds, protein structure, road networks, web, social netw... more Graphs are ubiquitous like chemical compounds, protein structure, road networks, web, social network analysis, etc. So graphs are very important means to model schemaless data with such complicated structures .This paper focuses on indexing structures present in the large graph datasets and proposes a time efficient indexing technique for efficient graph mining for molecular database. First step is to enumerating all frequent substructures in the graph database (derived from search tree) using proper pruning strategies at the time of candidate generation so removing the redundant search which will generate compact index for input database .An inverted index between these structures and graphs are then created. So indexing feature used is frequent subgraphs. But in the real world these datasets are very large in numbers so process of creating index may take large time. The time required has been minimized by using parallel and distributed approach.
Estimating the k-mer Coverage Frequencies in Genomic Datasets: A Comparative Assessment of the State-of-the-art
Current Genomics
Background: In bioinformatics, estimation of k-mer abundance histograms or just enumerating the n... more Background: In bioinformatics, estimation of k-mer abundance histograms or just enumerating the number of unique k-mers and the number of singletons are desirable in many genome sequence analysis applications. The applications include predicting genome sizes, data pre-processing for de Bruijn graph assembly methods (tune runtime parameters for analysis tools), repeat detection, sequencing coverage estimation, measuring sequencing error rates, etc. Different methods for cardinality estimation in sequencing data have been developed in recent years. Objective: In this article, we present a comparative assessment of the different k-mer frequency estimation programs (ntCard, KmerGenie, KmerStream and Khmer (abundance-dist-single.py and unique-kmers.py) to assess their relative merits and demerits. Methods: Principally, the miscounts/error-rates of these tools are analyzed by rigorous experimental analysis for a varied range of k. We also present experimental results on runtime, scalabili...
Manekar and Sathe survey the current state of the literature (and available tools) related to the... more Manekar and Sathe survey the current state of the literature (and available tools) related to the problem of k-mer counting. The manuscript begins with a rather nice background section that defines the problem and then touches upon many of the advancements made in tackling this problem with ever-more-efficient solutions. The current k-mer counting approaches are divided into categories based on the strategies they take for solving the problem-though the boundaries between different approaches isn't always entirely clear. Actually, I think that an overview figure corresponding to the "ontology of k-mer counting approaches", categorizing the different tools mentioned according to the underlying principles they use as well as show the relationship between these approaches (especially for those tools spanning multiple approaches or borrowing / using many different ideas). The manuscript then describes a fairly wide-ranging benchmark of many different popular k-mer counters, showing how their resource utilization (in terms of CPU, RAM and disk) scale with different data sets, k-mer sizes, etc. This turns out to be a useful benchmark, as general "broad scale" conclusions seem to be reasonably drawn. For example, Gerbil and KMC3 represent good defaults as they have consistently good performance metrics with well-controlled resource usage among a wide range of inputs and parameters. As a general, minor comment; though the manuscript is nicely laid out, the quality of the written English should be improved. There are a number of places with awkward phrasing, typos, etc.My major technical comments regard the methodology used to prepare the input datasets as well as the comparison based on metrics (e.g., wall clock time) that might not always be comparable given the different parameter settings required by different tools. For example, the authors state that input datasets are decompressed and "concatenated" into a single input file before running each of the tools. While this may seem a reasonable approach to normalize for potential differences (e.g., if not all of the tools support decompression directly), it may have a non-trivial effect on the results as they could deviate from how the tools might be used following best practices. Specifically, it is common to run most kmer counting tools directly on the compressed files without first decompressing the reads. This has the benefit, especially on "traditional" hard drives, of improving the I/O throughput and allowing the actual counting algorithms to consume data faster. This is because certain compression schemes like gzip have quite limited computational overhead, and so the cost of having to decompress the data in memory is overcome by the corresponding increase in data throughput that results from reading compressed data from disk. Since this is likely a common strategy for running k-mer counters, it should be included in the existing benchmarks (at least in addition to the current benchmarks).Another potential issue stems from normalizing the data by concatenating the reads into a single file. Some k-mer counting tools (e.g. KMC2, though I am not entirely certain of the changes made in KMC3) effectively perform parallelization in their first phase by reading from individual input files using separate threads. This means that restricting the input to a single file effectively limits them to 1 or 2 threads (e.g., one to parse and one to bin / partition). Given KMC3 %CPU utilization numbers this may, indeed, be occurring. It looks as though parallelism effectively increases when KMC3 enters the second phase (that doesn't depend on input), but it would likely be able to do even better on multi-file inputs if those inputs were not first concatenated into a single file. This may also be true of other tools.My final concern is not, unfortunately, one where I am able to offer a great solution. That is, given the wide variation in adopted settings, it's not completely clear to me that the CPU resource comparisons are fair. For example, when different programs are executed with different numbers of threads, their running times aren't directly comparable. Moreover, given the difficulty in achieving "perfect scaling", it's not necessarily much better to examine non-wall-clock-time metrics. I realize that it is not always possible to compare all programs with exactly equal parameters-as some programs, for example, restrict
Manekar and Sathe survey the current state of the literature (and available tools) related to the... more Manekar and Sathe survey the current state of the literature (and available tools) related to the problem of k-mer counting. The manuscript begins with a rather nice background section that defines the problem and then touches upon many of the advancements made in tackling this problem with ever-more-efficient solutions. The current k-mer counting approaches are divided into categories based on the strategies they take for solving the problem-though the boundaries between different approaches isn't always entirely clear. Actually, I think that an overview figure corresponding to the "ontology of k-mer counting approaches", categorizing the different tools mentioned according to the underlying principles they use as well as show the relationship between these approaches (especially for those tools spanning multiple approaches or borrowing / using many different ideas). The manuscript then describes a fairly wide-ranging benchmark of many different popular k-mer counters, showing how their resource utilization (in terms of CPU, RAM and disk) scale with different data sets, k-mer sizes, etc. This turns out to be a useful benchmark, as general "broad scale" conclusions seem to be reasonably drawn. For example, Gerbil and KMC3 represent good defaults as they have consistently good performance metrics with well-controlled resource usage among a wide range of inputs and parameters. As a general, minor comment; though the manuscript is nicely laid out, the quality of the written English should be improved. There are a number of places with awkward phrasing, typos, etc.My major technical comments regard the methodology used to prepare the input datasets as well as the comparison based on metrics (e.g., wall clock time) that might not always be comparable given the different parameter settings required by different tools. For example, the authors state that input datasets are decompressed and "concatenated" into a single input file before running each of the tools. While this may seem a reasonable approach to normalize for potential differences (e.g., if not all of the tools support decompression directly), it may have a non-trivial effect on the results as they could deviate from how the tools might be used following best practices. Specifically, it is common to run most kmer counting tools directly on the compressed files without first decompressing the reads. This has the benefit, especially on "traditional" hard drives, of improving the I/O throughput and allowing the actual counting algorithms to consume data faster. This is because certain compression schemes like gzip have quite limited computational overhead, and so the cost of having to decompress the data in memory is overcome by the corresponding increase in data throughput that results from reading compressed data from disk. Since this is likely a common strategy for running k-mer counters, it should be included in the existing benchmarks (at least in addition to the current benchmarks).Another potential issue stems from normalizing the data by concatenating the reads into a single file. Some k-mer counting tools (e.g. KMC2, though I am not entirely certain of the changes made in KMC3) effectively perform parallelization in their first phase by reading from individual input files using separate threads. This means that restricting the input to a single file effectively limits them to 1 or 2 threads (e.g., one to parse and one to bin / partition). Given KMC3 %CPU utilization numbers this may, indeed, be occurring. It looks as though parallelism effectively increases when KMC3 enters the second phase (that doesn't depend on input), but it would likely be able to do even better on multi-file inputs if those inputs were not first concatenated into a single file. This may also be true of other tools.My final concern is not, unfortunately, one where I am able to offer a great solution. That is, given the wide variation in adopted settings, it's not completely clear to me that the CPU resource comparisons are fair. For example, when different programs are executed with different numbers of threads, their running times aren't directly comparable. Moreover, given the difficulty in achieving "perfect scaling", it's not necessarily much better to examine non-wall-clock-time metrics. I realize that it is not always possible to compare all programs with exactly equal parameters-as some programs, for example, restrict
Plenty of structural patterns in real world have been represented as graph like molecules, chemic... more Plenty of structural patterns in real world have been represented as graph like molecules, chemical compounds, social network, road network etc. Mining this graph for extracting some useful information is of special interest and has many applications. The application includes drug discovery, compound synthesis, anomaly detection in network, social network analysis for finding groups etc. One of the most interesting problems in graph mining is graph containment problem. In graph containment problem ,given a query graph q ,it is asked to find all graph in given graph dataset containing this query (query graph as subgraph).This means finding all graph which is isomorphic to query graph. As in real world there is vast number of graph in graph dataset so this task of subgraph isomorphism test become tedious, complex, time and space consuming. So it is necessary to create an index of graphs present in dataset for cost efficient query processing. In this paper we proposed a time efficient ...
Graph Indexing using Distributed and Parallelization Approach
Graphs are ubiquitous like chemical compounds, protein structure, road networks, web, social netw... more Graphs are ubiquitous like chemical compounds, protein structure, road networks, web, social network analysis, etc. So graphs are very important means to model schemaless data with such complicated structures .This paper focuses on indexing structures present in the large graph datasets and proposes a time efficient indexing technique for efficient graph mining for molecular database. First step is to enumerating all frequent substructures in the graph database (derived from search tree) using proper pruning strategies at the time of candidate generation so removing the redundant search which will generate compact index for input database .An inverted index between these structures and graphs are then created. So indexing feature used is frequent subgraphs. But in the real world these datasets are very large in numbers so process of creating index may take large time. The time required has been minimized by using parallel and distributed approach.
Estimating the k-mer Coverage Frequencies in Genomic Datasets: A Comparative Assessment of the State-of-the-art
Current Genomics
Background: In bioinformatics, estimation of k-mer abundance histograms or just enumerating the n... more Background: In bioinformatics, estimation of k-mer abundance histograms or just enumerating the number of unique k-mers and the number of singletons are desirable in many genome sequence analysis applications. The applications include predicting genome sizes, data pre-processing for de Bruijn graph assembly methods (tune runtime parameters for analysis tools), repeat detection, sequencing coverage estimation, measuring sequencing error rates, etc. Different methods for cardinality estimation in sequencing data have been developed in recent years. Objective: In this article, we present a comparative assessment of the different k-mer frequency estimation programs (ntCard, KmerGenie, KmerStream and Khmer (abundance-dist-single.py and unique-kmers.py) to assess their relative merits and demerits. Methods: Principally, the miscounts/error-rates of these tools are analyzed by rigorous experimental analysis for a varied range of k. We also present experimental results on runtime, scalabili...
Manekar and Sathe survey the current state of the literature (and available tools) related to the... more Manekar and Sathe survey the current state of the literature (and available tools) related to the problem of k-mer counting. The manuscript begins with a rather nice background section that defines the problem and then touches upon many of the advancements made in tackling this problem with ever-more-efficient solutions. The current k-mer counting approaches are divided into categories based on the strategies they take for solving the problem-though the boundaries between different approaches isn't always entirely clear. Actually, I think that an overview figure corresponding to the "ontology of k-mer counting approaches", categorizing the different tools mentioned according to the underlying principles they use as well as show the relationship between these approaches (especially for those tools spanning multiple approaches or borrowing / using many different ideas). The manuscript then describes a fairly wide-ranging benchmark of many different popular k-mer counters, showing how their resource utilization (in terms of CPU, RAM and disk) scale with different data sets, k-mer sizes, etc. This turns out to be a useful benchmark, as general "broad scale" conclusions seem to be reasonably drawn. For example, Gerbil and KMC3 represent good defaults as they have consistently good performance metrics with well-controlled resource usage among a wide range of inputs and parameters. As a general, minor comment; though the manuscript is nicely laid out, the quality of the written English should be improved. There are a number of places with awkward phrasing, typos, etc.My major technical comments regard the methodology used to prepare the input datasets as well as the comparison based on metrics (e.g., wall clock time) that might not always be comparable given the different parameter settings required by different tools. For example, the authors state that input datasets are decompressed and "concatenated" into a single input file before running each of the tools. While this may seem a reasonable approach to normalize for potential differences (e.g., if not all of the tools support decompression directly), it may have a non-trivial effect on the results as they could deviate from how the tools might be used following best practices. Specifically, it is common to run most kmer counting tools directly on the compressed files without first decompressing the reads. This has the benefit, especially on "traditional" hard drives, of improving the I/O throughput and allowing the actual counting algorithms to consume data faster. This is because certain compression schemes like gzip have quite limited computational overhead, and so the cost of having to decompress the data in memory is overcome by the corresponding increase in data throughput that results from reading compressed data from disk. Since this is likely a common strategy for running k-mer counters, it should be included in the existing benchmarks (at least in addition to the current benchmarks).Another potential issue stems from normalizing the data by concatenating the reads into a single file. Some k-mer counting tools (e.g. KMC2, though I am not entirely certain of the changes made in KMC3) effectively perform parallelization in their first phase by reading from individual input files using separate threads. This means that restricting the input to a single file effectively limits them to 1 or 2 threads (e.g., one to parse and one to bin / partition). Given KMC3 %CPU utilization numbers this may, indeed, be occurring. It looks as though parallelism effectively increases when KMC3 enters the second phase (that doesn't depend on input), but it would likely be able to do even better on multi-file inputs if those inputs were not first concatenated into a single file. This may also be true of other tools.My final concern is not, unfortunately, one where I am able to offer a great solution. That is, given the wide variation in adopted settings, it's not completely clear to me that the CPU resource comparisons are fair. For example, when different programs are executed with different numbers of threads, their running times aren't directly comparable. Moreover, given the difficulty in achieving "perfect scaling", it's not necessarily much better to examine non-wall-clock-time metrics. I realize that it is not always possible to compare all programs with exactly equal parameters-as some programs, for example, restrict
Manekar and Sathe survey the current state of the literature (and available tools) related to the... more Manekar and Sathe survey the current state of the literature (and available tools) related to the problem of k-mer counting. The manuscript begins with a rather nice background section that defines the problem and then touches upon many of the advancements made in tackling this problem with ever-more-efficient solutions. The current k-mer counting approaches are divided into categories based on the strategies they take for solving the problem-though the boundaries between different approaches isn't always entirely clear. Actually, I think that an overview figure corresponding to the "ontology of k-mer counting approaches", categorizing the different tools mentioned according to the underlying principles they use as well as show the relationship between these approaches (especially for those tools spanning multiple approaches or borrowing / using many different ideas). The manuscript then describes a fairly wide-ranging benchmark of many different popular k-mer counters, showing how their resource utilization (in terms of CPU, RAM and disk) scale with different data sets, k-mer sizes, etc. This turns out to be a useful benchmark, as general "broad scale" conclusions seem to be reasonably drawn. For example, Gerbil and KMC3 represent good defaults as they have consistently good performance metrics with well-controlled resource usage among a wide range of inputs and parameters. As a general, minor comment; though the manuscript is nicely laid out, the quality of the written English should be improved. There are a number of places with awkward phrasing, typos, etc.My major technical comments regard the methodology used to prepare the input datasets as well as the comparison based on metrics (e.g., wall clock time) that might not always be comparable given the different parameter settings required by different tools. For example, the authors state that input datasets are decompressed and "concatenated" into a single input file before running each of the tools. While this may seem a reasonable approach to normalize for potential differences (e.g., if not all of the tools support decompression directly), it may have a non-trivial effect on the results as they could deviate from how the tools might be used following best practices. Specifically, it is common to run most kmer counting tools directly on the compressed files without first decompressing the reads. This has the benefit, especially on "traditional" hard drives, of improving the I/O throughput and allowing the actual counting algorithms to consume data faster. This is because certain compression schemes like gzip have quite limited computational overhead, and so the cost of having to decompress the data in memory is overcome by the corresponding increase in data throughput that results from reading compressed data from disk. Since this is likely a common strategy for running k-mer counters, it should be included in the existing benchmarks (at least in addition to the current benchmarks).Another potential issue stems from normalizing the data by concatenating the reads into a single file. Some k-mer counting tools (e.g. KMC2, though I am not entirely certain of the changes made in KMC3) effectively perform parallelization in their first phase by reading from individual input files using separate threads. This means that restricting the input to a single file effectively limits them to 1 or 2 threads (e.g., one to parse and one to bin / partition). Given KMC3 %CPU utilization numbers this may, indeed, be occurring. It looks as though parallelism effectively increases when KMC3 enters the second phase (that doesn't depend on input), but it would likely be able to do even better on multi-file inputs if those inputs were not first concatenated into a single file. This may also be true of other tools.My final concern is not, unfortunately, one where I am able to offer a great solution. That is, given the wide variation in adopted settings, it's not completely clear to me that the CPU resource comparisons are fair. For example, when different programs are executed with different numbers of threads, their running times aren't directly comparable. Moreover, given the difficulty in achieving "perfect scaling", it's not necessarily much better to examine non-wall-clock-time metrics. I realize that it is not always possible to compare all programs with exactly equal parameters-as some programs, for example, restrict
Uploads
Papers by swati manekar