Avail 100% Scholarship on Certified AI & ML BlackBelt Plus Program* Avail Offer ×
Home
AWS Glue for Handling Metadata
Chetan Dekate — Updated On September 2nd, 2022
AWS Cloud Computing Data Engineering Intermediate
This article was published as a part of the Data Science Blogathon.
Introduction
AWS Glue helps Data Engineers to prepare data for other data consumers through the
Extract, Transform & Load (ETL) Process. The managed service offers a simple and cost-
effective method of categorizing and managing big data in an enterprise. It provides
organizations with a data integration tool that formats information from different data
sources and organizes it in a central repository where it can be used to inform business
decisions.
Source: – https://aws.amazon.com
How AWS Glue Works
Glue uses ETL jobs to extract data from a combination of other cloud services offered by
Amazon Web Services (AWS) and incorporate it into data lakes and data warehouses. It uses
an application programming interface (API) to transform the extracted dataset for
integration and helps users monitor jobs.
Users can schedule ETL jobs or select events to trigger the job. Once started, Glue extracts
the data, transforms it based on code that Glue automatically generates, and loads it into
Amazon S3 or Amazon Redshift. The glue then writes the metadata from the job to the
embedded AWS glue data catalog.
The service can automatically find enterprise structured or unstructured data when it is
stored in data lakes in S3, data warehouses in Amazon Redshift, and other databases that are
part
We useof the Amazon
cookies Relational
on Analytics Database
Vidhya websites Service.
to deliver Glue also
our services, supports
analyze MySQL,
web traffic, Oracle,
and improve your experience on the site. By using Analytics Vidhya, you
agree to our Privacy Policy and Terms of Use. Accept
Microsoft SQL Server, and PostgreSQL databases running on Amazon Elastic Compute Cloud
(EC2) instances in the Amazon Virtual Private Cloud.
The service then profiles the data in a data catalog, a metadata repository for all data assets
that includes details such as table definition, location, and other attributes. The team can also
use Glue Data Catalog as an alternative to Apache Hive Metastore for Amazon Elastic
MapReduce applications.
Configure the transform properties in AWS Glue.
Source: – https://kontext.tech/diagram/
To load metadata into the data catalog, the service uses Glue crawlers that scan the raw data
store and extract schema and other attributes. An IT professional can customize browsers as
needed.
Properties of AWS Glue
The basic properties of the glue are as follows:
Automatic schema detection. Glue allows developers to automate crawlers to retrieve
schema-related information and store it in a data catalog that can then be used to
manage jobs.
Task scheduler. Paste jobs can be set up and invoked on a flexible schedule using event-
based or on-demand triggers. Multiple jobs can run in parallel, and users can specify
dependencies between jobs.
Developer endpoints. Developers can use them to debug Glue and create custom
readers, writers, and transforms that can then be imported into custom libraries.
Automatic code generation. The ETL process automatically generates the code; the
only necessary input is the location/path to store the data. The code is either in Scala or
Python.
Integrated data catalog. It is a unique metadata repository for data from different
sources in the AWS pipeline. An AWS account has one catalog.
Advantages and Disadvantages of Using Glue
The advantages of AWS glue are as follows:
Building
We use cookiesMulti-Stage Reasoning
on Analytics Vidhya Systems
websites to deliver with
our services, LangChain
analyze web traffic, and improve your experience on the site. By using Analytics Vidhya, you
Date: 1 Nov 2023 Time: 6:00 PM –agree
7:00to
PMourIST
Privacy Policy and Terms of Use. Accept
RSVP!
Fault tolerance. Failed jobs in Glue can be retrieved, and logs in Glue can be debugged.
Filtering. Bad data filters.
Support, support. It supports several non-native Java Database Connectivity (JDBC)
data sources.
Maintenance and deployment. Easy maintenance and deployment because AWS
completely manages the service.
Source: -https://aws.amazon.com
Disadvantages of AWS Glue include:
Limited compatibility. While AWS Glue works with several commonly used data
sources, it only works with services running on AWS. Organizations may need a third-
party ETL service if the resources are not AWS-based.
No incremental data sync. All data is first scheduled to S3, so Glue is not the best choice
for real-time ETL jobs.
Learning curve. Teams using Glue should have a good understanding of Apache Spark.
Relational database queries. Glue has limited support for traditional relational
database queries, only SQL queries.
AWS Glue Use Cases
1. Improve speed and reduce costs
Financial is a technology company that specializes in product-related financial services.
These products are useful and help financial services companies in Southeast Asia. Its
headquarters are in Jakarta, Indonesia. She wanted to elevate and transform everyday ETL
jobs. The company opted for many other solutions but could not find the ideal solution for
the needs. Ultimately, they decided to use AWS Glue, which paid off as the team could easily
load the data, run the processes, and transform it for the next process on Redshift. It also
We use cookies on Analytics Vidhya websites to deliver our services, analyze web traffic, and improve your experience on the site. By using Analytics Vidhya, you
allowed the company to speed up the process and save costs. This association could not only
agree to our Privacy Policy and Terms of Use. Accept
prove to be faster and more efficient, but it was also much cheaper than the proven
alternatives. For a startup like Financial, cost savings were one of the deciding factors.
2. Build a data intelligence and analytics platform
Burt Corporation is a start-up data company specializing in data products to transform the
new online media landscape. It has New York, Berlin, and Gothenburg (Sweden) offices. Many
major online media publishers use Burt’s data intelligence and analytics platform to
understand and optimize their online marketing strategies. The platform must have effective
data collection, processing, analysis, and decision-making capabilities to meet these
requirements. Therefore, the company decided to use AWS Glue, Amazon Redshift, and
Amazon Athena to meet these requirements. Burt transformed his data analytics and
intelligence platform by integrating Amazon Athena and AWS Glue. As a result, Burt was able
to deliver a solution that met the client’s requirements.
3. Optimized cybersecurity environment
One of the world’s most recognized brands needs no introduction. It specializes in the
technology sector and specifically in industrial production. Siemens is a German
multinational corporation headquartered in Munich, Germany. She wanted to counter cyber
threats with a smart system. This system was expected to prepare data, analyze it and make
predictions using machine learning. The company decided to turn to Amazon Web Services
for a solution. AWS experts recommended Amazon Sage Maker, AWS Glue, and AWS
Lambda. AWS Glue played a key role in this process. It is a tool that extracts data and helps
data scientists categorize data easily.
Source: -https://aws.amazon.com
ETL Engine
After the data is cataloged, it is searchable and ready for ETL jobs. AWS Glue includes an ETL
script recommendation system for generating Python and Spark code (PySpark) and an ETL
library for running jobs. A developer can write ETL code through the custom Glue library or
PySpark code using the AWS Glue console script editor.
A developer can also import custom code or PySpark libraries. In addition, developers could
upload code for existing ETL jobs to an S3 bucket and then create a new Glue job to process
the code.
Plan and Organize ETL Jobs
AWS Glue jobs can be run on a schedule. A developer can schedule ETL jobs at minimum five-
minute
We intervals.
use cookies AWSVidhya
on Analytics Glue cannot
websiteshandle
to deliverstreaming
our services,data.
analyze web traffic, and improve your experience on the site. By using Analytics Vidhya, you
agree to our Privacy Policy and Terms of Use. Accept
The service allows for scheduled, on-demand, and task completion triggers if the
development team prefers to organize tasks. A scheduled launcher runs jobs at specified
intervals, while an on-demand launcher runs when the user prompts. With the task
completion trigger, one or more tasks can be executed after the tasks are completed. These
jobs can run concurrently or sequentially and can also be run from an external service such as
AWS Lambda.
AWS Glue Price
AWS charges users a monthly fee to store and access metadata in the Glue Data Catalog.
With AWS Glue pricing, there is a per-second fee, either a minimum of 10 minutes or 1
minute (depending on the Glue version users have), for the ETL job and crawler execution.
AWS also includes a per-second fee for connecting to a development endpoint for interactive
development.
Conclusion
It uses an application programming interface (API) to transform the extracted dataset for
integration and helps users monitor jobs. The service then profiles the data in a data catalog, a metadata
repository for all data assets that includes details such as table definition, location, and other attributes. The team can
also use Glue Data Catalog as an alternative to Apache Hive Metastore for Amazon Elastic MapReduce applications.
AWS Glues has Five Basic properties:- Automatic schema detection, Task scheduler, Developer
endpoint, Automatic code generation, and Integrated data catalog.
The advantages of AWS glue are as follows:- Fault tolerance, Filtering, Support,
support, Maintenance, and deployment. Disadvantages of AWS Glue include:- Limited
compatibility, No incremental data sync, Learning curve, and Relational database
queries.
A scheduled launcher runs jobs at specified intervals, while an on-demand launcher
runs when the user prompts. With the task completion trigger, one or more tasks can
be executed after the tasks are completed. These jobs can run concurrently or
sequentially and can also be run from an external service such as AWS Lambda.
The media shown in this article is not owned by Analytics Vidhya and is used at the
Author’s discretion.
AWS AWS Glue blogathon
About the Author
Chetan Dekate
Our Top Authors
We use cookies on Analytics Vidhya websites to deliver our services, analyze web traffic, and improve your experience on the site. By using Analytics Vidhya, you
agree to our Privacy Policy and Terms of Use. Accept
