Skip to main content

Michael J Mior

Rochester Institute of Technology, Computer Science, Faculty Member

Followers

48

Following

54

Co-author

1

Public Views

Supervisors: Ken Salem and Eyal de Lara
Phone: 585-475-5810
Address: Computer Science Department
102 Lomb Memorial Drive
Rochester, NY 14623-5608

less

Université Toulouse 1 Capitole

Universidad de los Andes (Colombia)

Ecole Nationale Superieur D'informatique

Hassan Aït-Kaci

University of Hagen

InterestsView All (6)

Uploads

Papers by Michael J Mior

Jsonoid: Monoid-Based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

Schema discovery is an important aspect to working with data in formats such as JSON. Unlike rela... more Schema discovery is an important aspect to working with data in formats such as JSON. Unlike relational databases, JSON data sets often do not have associated structural information. Consumers of such datasets are often left to browse through data in an attempt to observe commonalities in structure across documents to construct suitable code for data processing. However, this process is timeconsuming and error-prone. Existing distributed approaches to mining schemas present a significant usability advantage as they provide useful metadata for large data sources. However, depending on the data source, ad hoc queries for estimating other properties to help with crafting an efficient data pipeline can be expensive. We propose JSONoid, a distributed schema discovery process augmented with additional metadata in the form of monoid data structures that are easily maintainable in a distributed setting. JSONoid subsumes several existing approaches to distributed schema discovery with similar performance. Our approach also adds significant useful additional information about data values to discovered schemas with linear scalability.

JSONoid: Monoid-based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

arXiv (Cornell University), Jul 6, 2023

Schema discovery is an important aspect to working with data in formats such as JSON. Unlike rela... more Schema discovery is an important aspect to working with data in formats such as JSON. Unlike relational databases, JSON data sets often do not have associated structural information. Consumers of such datasets are often left to browse through data in an attempt to observe commonalities in structure across documents to construct suitable code for data processing. However, this process is timeconsuming and error-prone. Existing distributed approaches to mining schemas present a significant usability advantage as they provide useful metadata for large data sources. However, depending on the data source, ad hoc queries for estimating other properties to help with crafting an efficient data pipeline can be expensive. We propose JSONoid, a distributed schema discovery process augmented with additional metadata in the form of monoid data structures that are easily maintainable in a distributed setting. JSONoid subsumes several existing approaches to distributed schema discovery with similar performance. Our approach also adds significant useful additional information about data values to discovered schemas with linear scalability.

Comprehending Semantic Types in JSON Data with Graph Neural Networks

arXiv (Cornell University), Jul 24, 2023

Semantic types are a more powerful and detailed way of describing data than atomic types such as ... more Semantic types are a more powerful and detailed way of describing data than atomic types such as strings or integers. They establish connections between columns and concepts from the real world, providing more nuanced and fine-grained information that can be useful for tasks such as automated data cleaning, schema matching, and data discovery. Existing deep learning models trained on large text corpora have been successful at performing single-column semantic type prediction for relational data. However, in this work, we propose an extension of the semantic type prediction problem to JSON data, labeling the types based on JSON Paths. Similar to columns in relational data, JSON Path is a query language that enables the navigation of complex JSON data structures by specifying the location and content of the elements. We use a graph neural network to comprehend the structural information within collections of JSON documents. Our model outperforms a state-ofthe-art existing model in several cases. These results demonstrate the ability of our model to understand complex JSON data and its potential usage for JSON-related data processing tasks.

NoSQL Schema Design for Time-Dependent Workloads

arXiv (Cornell University), Mar 29, 2023

In this paper, we propose a schema optimization method for time-dependent workloads for NoSQL dat... more In this paper, we propose a schema optimization method for time-dependent workloads for NoSQL databases. In our proposed method, we migrate schema according to changing workloads, and the estimated cost of execution and migration are formulated and minimized as a single integer linear programming problem. Furthermore, we propose a method to reduce the number of optimization candidates by iterating over the time dimension abstraction and optimizing the workload while updating constraints.

Stateless services are easy to scale in the cloud since new replicas of these services can be cre... more Stateless services are easy to scale in the cloud since new replicas of these services can be created at any time and they operate completely independently of other instances. In contrast, scaling stateful services, such as a database system, can take minutes or even hours due to the need to present a consistent view of the system for users of the service. Currently, this problem is addressed by resource overprovisioning in anticipation of demand spikes. FlurryDB uses virtual machine cloning to improve resource utilization by drastically reducing the latency required to add a new replica. We also show that FlurryDB is capable of handling updates to resources in a fashion that preserves consistency across the cloning boundary.

Learning from Uncurated Regular Expressions for Semantic Type Classification

Proceedings of the 1st Workshop on Simplicity in Management of Data

Significant work has been done on learning regular expressions from a set of data values. Dependi... more Significant work has been done on learning regular expressions from a set of data values. Depending on the domain, this approach can be very successful. However, significant time is required to learn these expressions and the resulting expressions can either become very complex or inaccurate in the presence of dirty data. The alternative of manually writing regular expressions becomes unattractive when faced with a large number of values which must be matched. As an alternative, we propose learning from a large corpus of manually authored, but uncurated regular expressions mined from a public repository. The advantage of this approach is that we are able to extract salient features from a set of strings with limited overhead to feature engineering. Since the set of regular expressions covers a wide range of application domains, we expect them to widely applicable. To demonstrate the potential effectiveness of our approach, we train a model using the extracted corpus of regular expressions for the class of semantic type classification. While our approach generally yields results that are inferior to the state of the art, our training data is much smaller and simpler and a closer analysis of the performance results suggests this approach holds significant promise. We also demonstrate the possibility of using uncurated regular expressions for unsupervised learning.

Relational Playground: Teaching the Duality of Relational Algebra and SQL

CoDEvo: Column family database evolution using model transformations

Journal of Systems and Software

Relational Playground: Teaching the Duality of Relational Algebra and SQL

Proceedings of the 2nd International Workshop on Data Systems Education: Bridging education practice with education research

Fast Discovery of Nested Dependencies on JSON Data

arXiv (Cornell University), Nov 19, 2021

Functional and inclusion dependencies are the most widely used classes of data dependencies in da... more Functional and inclusion dependencies are the most widely used classes of data dependencies in data profiling due to their ability to identify relationships in data such as primary and foreign keys. These relationships are equally important when dealing with nested data formats such as JSON. However, the definition of functional and inclusion dependencies makes use of a flat, unnested relational model which misses many useful types of dependencies on data which involve nested data models. In this work, we identify types of dependencies which are not captured by traditional functional and inclusion dependencies but which nevertheless capture meaningful relationships among nested data. We also demonstrate how algorithms for mining these traditional dependencies can be adapted to also mine nested dependencies. The first strategy simply flattens the input data and feeds into unmodified existing algorithms. We present a second strategy which instead adapts the algorithm to efficiently process JSON data as input which in some cases leads to a reduction in runtime by multiple orders of magnitude on real-world datasets. We further show how these algorithms can be adapted to produce useful results in the presence of a percentage of incomplete or invalid data.

Learning from Uncurated Regular Expressions

arXiv (Cornell University), Jun 14, 2022

Significant work has been done on learning regular expressions from a set of data values. Dependi... more Significant work has been done on learning regular expressions from a set of data values. Depending on the domain, this approach can be very successful. However, significant time is required to learn these expressions and the resulting expressions can either become very complex or inaccurate in the presence of dirty data. The alternative of manually writing regular expressions becomes unattractive when faced with a large number of values which must be matched. As an alternative, we propose learning from a large corpus of manually authored, but uncurated regular expressions mined from a public repository. The advantage of this approach is that we are able to extract salient features from a set of strings with limited overhead to feature engineering. Since the set of regular expressions covers a wide range of application domains, we expect them to widely applicable. To demonstrate the potential effectiveness of our approach, we train a model using the extracted corpus of regular expressions for the class of semantic type classification. While our approach generally yields results that are inferior to the state of the art, our training data is much smaller and simpler and a closer analysis of the performance results suggests this approach holds significant promise. We also demonstrate the possibility of using uncurated regular expressions for unsupervised learning.

NoSE: NoSQL Schema Evaluator

This algorithm comprises the core of NoSE: a schema design tool for NoSQL databases. More informa... more

How Inclusive are We?

ACM SIGMOD Record, 2022

ACM SIGMOD, VLDB and other database organizations have committed to fostering an inclusive and di... more ACM SIGMOD, VLDB and other database organizations have committed to fostering an inclusive and diverse community, as do many other scientific organizations. Recently, different measures have been taken to advance these goals, especially for underrepresented groups. One possible measure is double-blind reviewing, which aims to hide gender, ethnicity, and other properties of the authors. We report the preliminary results of a gender diversity analysis of publications of the database community across several peer-reviewed venues, and also compare women's authorship percentages in both single-blind and double-blind venues along the years. We also obtained a cross comparison of the obtained results in data management with other relevant areas in Computer Science.

SYSTOR 2011 Presentation

Netstore: leveraging network optimizations to improve distributed transaction processing performance

Proceedings of the Second International Workshop on Active Middleware on Modern Hardware, 2017

A tremendous amount of data is processed daily in the cloud, which inevitably introduces large am... more A tremendous amount of data is processed daily in the cloud, which inevitably introduces large amounts of data transfer within a datacenter network. Numerous studies have reported that the network is often the performance bottleneck for cloud applications. While flow scheduling techniques have been proposed to mitigate this performance problem, distributed network-aware applications have not received much attention. The advent of software-defined networking (SDN) enables leveraging flow scheduling to make cloud applications network-aware, opening up the potential to improve the performance of these applications. In this paper, we propose co-designing network and cloud applications to optimize performance in the presence of data-center network congestion. In particular, we built a network-aware distributed database framework which uses network state information to improve performance. We present two techniques to reduce transaction completion time for cloud database applications. Fir...

Semantic Data Understanding with Character Level Learning

2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), 2020

Databases are growing in size and complexity. With the emergence of data lakes, databases have be... more Databases are growing in size and complexity. With the emergence of data lakes, databases have become open, fast evolving and highly heterogeneous. Understanding the complex relationships among different entity types in such scenarios is both challenging and necessary to data scientists. We propose an approach that utilizes a convolutional neural network to learn patterns associated with each entity type in the database at the character level. We demonstrate that the learned character-level patterns can capture sufficient semantic information for many useful applications including data lake schema exploration, and interactive data cleaning.

Physical Design for Non-relational Data Systems

Decades of research have gone into the optimization of physical designs, query execution, and rel... more Decades of research have gone into the optimization of physical designs, query execution, and related tools for relational databases. These techniques and tools make it possible for non-expert users to make effective use of relational database management systems. However, the drive for flexible data models and increased scalability has spawned a new generation of data management systems which largely eschew the relational model. These include systems such as NoSQL databases and distributed analytics frameworks such as Apache Spark which make use of a diverse set of data models. Optimization techniques and tools developed for relational data do not directly apply in this setting. This leaves developers making use of these systems with the need to become intimately familiar with system details to obtain good performance. We present techniques and tools for physical design for non-relational data systems. We explore two settings: NoSQL database systems and distributed analytics framewo...

Renormalization of NoSQL Database Schemas

Conceptual Modeling

NoSQL applications often use denormalized databases in order to meet performance goals, but this ... more NoSQL applications often use denormalized databases in order to meet performance goals, but this introduces complications. In particular, application evolution may demand changes in the underlying database schema, which may in turn require further application revisions. The NoSQL DBMS itself can do little to aid in this process, as it has no understanding of application-level denormalization. In this paper, we describe a procedure for reconstructing a normalized conceptual schema from a denormalized NoSQL database. Exposing the conceptual schema provides application developers with information that can be used to guide application and database evolution. The procedure's input includes functional and inclusion dependencies, which may be mined from the NoSQL database. We illustrate the effectiveness of this procedure using several application case studies.

ReSpark: Automatic Caching for Iterative Applications in Apache Spark

by K. Salem and Michael J Mior

2020 IEEE International Conference on Big Data (Big Data)

Apache Spark is a distributed computing framework used for big data processing. A common pattern ... more Apache Spark is a distributed computing framework used for big data processing. A common pattern in many Spark applications is to iteratively evolve a dataset until reaching some user-specified convergence condition. Unfortunately, some aspects of Spark's execution model make it difficult for developers who are not familiar with the implementation-level details of Spark to write efficient iterative programs. Since results are constructed iteratively and results from previous iterations may be used multiple times, effective use of caching is necessary to avoid recomputing intermediate results. Currently, developers of Spark applications must manually indicate which intermediate results should be cached. We present a method for using metadata already captured by Spark to automate caching decisions for many Spark programs. We show how this allows Spark applications to benefit from caching without the need for manual caching annotations.

Column2Vec: Structural Understanding via Distributed Representations of Database Schemas

We present Column2Vec, a distributed representation of database columns based on column metadata.... more We present Column2Vec, a distributed representation of database columns based on column metadata. Our distributed representation has several applications. Using known names for groups of columns (i.e., a table name), we train a model to generate an appropriate name for columns in an unnamed table. We demonstrate the viability of our approach using schema information collected from open source applications on GitHub.

Jsonoid: Monoid-Based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

Schema discovery is an important aspect to working with data in formats such as JSON. Unlike rela... more Schema discovery is an important aspect to working with data in formats such as JSON. Unlike relational databases, JSON data sets often do not have associated structural information. Consumers of such datasets are often left to browse through data in an attempt to observe commonalities in structure across documents to construct suitable code for data processing. However, this process is timeconsuming and error-prone. Existing distributed approaches to mining schemas present a significant usability advantage as they provide useful metadata for large data sources. However, depending on the data source, ad hoc queries for estimating other properties to help with crafting an efficient data pipeline can be expensive. We propose JSONoid, a distributed schema discovery process augmented with additional metadata in the form of monoid data structures that are easily maintainable in a distributed setting. JSONoid subsumes several existing approaches to distributed schema discovery with similar performance. Our approach also adds significant useful additional information about data values to discovered schemas with linear scalability.

JSONoid: Monoid-based Enrichment for Configurable and Scalable Data-Driven Schema Discovery

arXiv (Cornell University), Jul 6, 2023

Schema discovery is an important aspect to working with data in formats such as JSON. Unlike rela... more Schema discovery is an important aspect to working with data in formats such as JSON. Unlike relational databases, JSON data sets often do not have associated structural information. Consumers of such datasets are often left to browse through data in an attempt to observe commonalities in structure across documents to construct suitable code for data processing. However, this process is timeconsuming and error-prone. Existing distributed approaches to mining schemas present a significant usability advantage as they provide useful metadata for large data sources. However, depending on the data source, ad hoc queries for estimating other properties to help with crafting an efficient data pipeline can be expensive. We propose JSONoid, a distributed schema discovery process augmented with additional metadata in the form of monoid data structures that are easily maintainable in a distributed setting. JSONoid subsumes several existing approaches to distributed schema discovery with similar performance. Our approach also adds significant useful additional information about data values to discovered schemas with linear scalability.

Comprehending Semantic Types in JSON Data with Graph Neural Networks

arXiv (Cornell University), Jul 24, 2023

Semantic types are a more powerful and detailed way of describing data than atomic types such as ... more Semantic types are a more powerful and detailed way of describing data than atomic types such as strings or integers. They establish connections between columns and concepts from the real world, providing more nuanced and fine-grained information that can be useful for tasks such as automated data cleaning, schema matching, and data discovery. Existing deep learning models trained on large text corpora have been successful at performing single-column semantic type prediction for relational data. However, in this work, we propose an extension of the semantic type prediction problem to JSON data, labeling the types based on JSON Paths. Similar to columns in relational data, JSON Path is a query language that enables the navigation of complex JSON data structures by specifying the location and content of the elements. We use a graph neural network to comprehend the structural information within collections of JSON documents. Our model outperforms a state-ofthe-art existing model in several cases. These results demonstrate the ability of our model to understand complex JSON data and its potential usage for JSON-related data processing tasks.

NoSQL Schema Design for Time-Dependent Workloads

arXiv (Cornell University), Mar 29, 2023

In this paper, we propose a schema optimization method for time-dependent workloads for NoSQL dat... more In this paper, we propose a schema optimization method for time-dependent workloads for NoSQL databases. In our proposed method, we migrate schema according to changing workloads, and the estimated cost of execution and migration are formulated and minimized as a single integer linear programming problem. Furthermore, we propose a method to reduce the number of optimization candidates by iterating over the time dimension abstraction and optimizing the workload while updating constraints.

Stateless services are easy to scale in the cloud since new replicas of these services can be cre... more Stateless services are easy to scale in the cloud since new replicas of these services can be created at any time and they operate completely independently of other instances. In contrast, scaling stateful services, such as a database system, can take minutes or even hours due to the need to present a consistent view of the system for users of the service. Currently, this problem is addressed by resource overprovisioning in anticipation of demand spikes. FlurryDB uses virtual machine cloning to improve resource utilization by drastically reducing the latency required to add a new replica. We also show that FlurryDB is capable of handling updates to resources in a fashion that preserves consistency across the cloning boundary.

Learning from Uncurated Regular Expressions for Semantic Type Classification

Proceedings of the 1st Workshop on Simplicity in Management of Data

Significant work has been done on learning regular expressions from a set of data values. Dependi... more Significant work has been done on learning regular expressions from a set of data values. Depending on the domain, this approach can be very successful. However, significant time is required to learn these expressions and the resulting expressions can either become very complex or inaccurate in the presence of dirty data. The alternative of manually writing regular expressions becomes unattractive when faced with a large number of values which must be matched. As an alternative, we propose learning from a large corpus of manually authored, but uncurated regular expressions mined from a public repository. The advantage of this approach is that we are able to extract salient features from a set of strings with limited overhead to feature engineering. Since the set of regular expressions covers a wide range of application domains, we expect them to widely applicable. To demonstrate the potential effectiveness of our approach, we train a model using the extracted corpus of regular expressions for the class of semantic type classification. While our approach generally yields results that are inferior to the state of the art, our training data is much smaller and simpler and a closer analysis of the performance results suggests this approach holds significant promise. We also demonstrate the possibility of using uncurated regular expressions for unsupervised learning.

Relational Playground: Teaching the Duality of Relational Algebra and SQL

CoDEvo: Column family database evolution using model transformations

Journal of Systems and Software

Relational Playground: Teaching the Duality of Relational Algebra and SQL

Proceedings of the 2nd International Workshop on Data Systems Education: Bridging education practice with education research

Fast Discovery of Nested Dependencies on JSON Data

arXiv (Cornell University), Nov 19, 2021

Functional and inclusion dependencies are the most widely used classes of data dependencies in da... more Functional and inclusion dependencies are the most widely used classes of data dependencies in data profiling due to their ability to identify relationships in data such as primary and foreign keys. These relationships are equally important when dealing with nested data formats such as JSON. However, the definition of functional and inclusion dependencies makes use of a flat, unnested relational model which misses many useful types of dependencies on data which involve nested data models. In this work, we identify types of dependencies which are not captured by traditional functional and inclusion dependencies but which nevertheless capture meaningful relationships among nested data. We also demonstrate how algorithms for mining these traditional dependencies can be adapted to also mine nested dependencies. The first strategy simply flattens the input data and feeds into unmodified existing algorithms. We present a second strategy which instead adapts the algorithm to efficiently process JSON data as input which in some cases leads to a reduction in runtime by multiple orders of magnitude on real-world datasets. We further show how these algorithms can be adapted to produce useful results in the presence of a percentage of incomplete or invalid data.

Learning from Uncurated Regular Expressions

arXiv (Cornell University), Jun 14, 2022

Significant work has been done on learning regular expressions from a set of data values. Dependi... more Significant work has been done on learning regular expressions from a set of data values. Depending on the domain, this approach can be very successful. However, significant time is required to learn these expressions and the resulting expressions can either become very complex or inaccurate in the presence of dirty data. The alternative of manually writing regular expressions becomes unattractive when faced with a large number of values which must be matched. As an alternative, we propose learning from a large corpus of manually authored, but uncurated regular expressions mined from a public repository. The advantage of this approach is that we are able to extract salient features from a set of strings with limited overhead to feature engineering. Since the set of regular expressions covers a wide range of application domains, we expect them to widely applicable. To demonstrate the potential effectiveness of our approach, we train a model using the extracted corpus of regular expressions for the class of semantic type classification. While our approach generally yields results that are inferior to the state of the art, our training data is much smaller and simpler and a closer analysis of the performance results suggests this approach holds significant promise. We also demonstrate the possibility of using uncurated regular expressions for unsupervised learning.

NoSE: NoSQL Schema Evaluator

This algorithm comprises the core of NoSE: a schema design tool for NoSQL databases. More informa... more

How Inclusive are We?

ACM SIGMOD Record, 2022

ACM SIGMOD, VLDB and other database organizations have committed to fostering an inclusive and di... more ACM SIGMOD, VLDB and other database organizations have committed to fostering an inclusive and diverse community, as do many other scientific organizations. Recently, different measures have been taken to advance these goals, especially for underrepresented groups. One possible measure is double-blind reviewing, which aims to hide gender, ethnicity, and other properties of the authors. We report the preliminary results of a gender diversity analysis of publications of the database community across several peer-reviewed venues, and also compare women's authorship percentages in both single-blind and double-blind venues along the years. We also obtained a cross comparison of the obtained results in data management with other relevant areas in Computer Science.

SYSTOR 2011 Presentation

Netstore: leveraging network optimizations to improve distributed transaction processing performance

Proceedings of the Second International Workshop on Active Middleware on Modern Hardware, 2017

A tremendous amount of data is processed daily in the cloud, which inevitably introduces large am... more A tremendous amount of data is processed daily in the cloud, which inevitably introduces large amounts of data transfer within a datacenter network. Numerous studies have reported that the network is often the performance bottleneck for cloud applications. While flow scheduling techniques have been proposed to mitigate this performance problem, distributed network-aware applications have not received much attention. The advent of software-defined networking (SDN) enables leveraging flow scheduling to make cloud applications network-aware, opening up the potential to improve the performance of these applications. In this paper, we propose co-designing network and cloud applications to optimize performance in the presence of data-center network congestion. In particular, we built a network-aware distributed database framework which uses network state information to improve performance. We present two techniques to reduce transaction completion time for cloud database applications. Fir...

Semantic Data Understanding with Character Level Learning

2020 IEEE 21st International Conference on Information Reuse and Integration for Data Science (IRI), 2020

Databases are growing in size and complexity. With the emergence of data lakes, databases have be... more Databases are growing in size and complexity. With the emergence of data lakes, databases have become open, fast evolving and highly heterogeneous. Understanding the complex relationships among different entity types in such scenarios is both challenging and necessary to data scientists. We propose an approach that utilizes a convolutional neural network to learn patterns associated with each entity type in the database at the character level. We demonstrate that the learned character-level patterns can capture sufficient semantic information for many useful applications including data lake schema exploration, and interactive data cleaning.

Physical Design for Non-relational Data Systems

Decades of research have gone into the optimization of physical designs, query execution, and rel... more Decades of research have gone into the optimization of physical designs, query execution, and related tools for relational databases. These techniques and tools make it possible for non-expert users to make effective use of relational database management systems. However, the drive for flexible data models and increased scalability has spawned a new generation of data management systems which largely eschew the relational model. These include systems such as NoSQL databases and distributed analytics frameworks such as Apache Spark which make use of a diverse set of data models. Optimization techniques and tools developed for relational data do not directly apply in this setting. This leaves developers making use of these systems with the need to become intimately familiar with system details to obtain good performance. We present techniques and tools for physical design for non-relational data systems. We explore two settings: NoSQL database systems and distributed analytics framewo...

Renormalization of NoSQL Database Schemas

Conceptual Modeling

NoSQL applications often use denormalized databases in order to meet performance goals, but this ... more NoSQL applications often use denormalized databases in order to meet performance goals, but this introduces complications. In particular, application evolution may demand changes in the underlying database schema, which may in turn require further application revisions. The NoSQL DBMS itself can do little to aid in this process, as it has no understanding of application-level denormalization. In this paper, we describe a procedure for reconstructing a normalized conceptual schema from a denormalized NoSQL database. Exposing the conceptual schema provides application developers with information that can be used to guide application and database evolution. The procedure's input includes functional and inclusion dependencies, which may be mined from the NoSQL database. We illustrate the effectiveness of this procedure using several application case studies.

ReSpark: Automatic Caching for Iterative Applications in Apache Spark

by K. Salem and Michael J Mior

2020 IEEE International Conference on Big Data (Big Data)

Apache Spark is a distributed computing framework used for big data processing. A common pattern ... more Apache Spark is a distributed computing framework used for big data processing. A common pattern in many Spark applications is to iteratively evolve a dataset until reaching some user-specified convergence condition. Unfortunately, some aspects of Spark's execution model make it difficult for developers who are not familiar with the implementation-level details of Spark to write efficient iterative programs. Since results are constructed iteratively and results from previous iterations may be used multiple times, effective use of caching is necessary to avoid recomputing intermediate results. Currently, developers of Spark applications must manually indicate which intermediate results should be cached. We present a method for using metadata already captured by Spark to automate caching decisions for many Spark programs. We show how this allows Spark applications to benefit from caching without the need for manual caching annotations.

Column2Vec: Structural Understanding via Distributed Representations of Database Schemas

We present Column2Vec, a distributed representation of database columns based on column metadata.... more We present Column2Vec, a distributed representation of database columns based on column metadata. Our distributed representation has several applications. Using known names for groups of columns (i.e., a table name), we train a model to generate an appropriate name for columns in an unnamed table. We demonstrate the viability of our approach using schema information collected from open source applications on GitHub.

FlurryDB: A Dynamically Scalable Relational Database with Virtual Machine Cloning

Column2Vec: Structural Understanding via Distributed Representations of Database Schemas

We present Column2Vec, a distributed representation of database columns based on column metadata.... more We present Column2Vec, a distributed representation of database columns based on column metadata. Our distributed representation has several applications. Using known names for groups of columns (i.e., a table name), we train a model to generate an appropriate name for columns in an unnamed table. We demonstrate the viability of our approach using schema information collected from open source applications on GitHub.