Hi, I'm Sonali Surange Dev.
Data
scientists often end up spending a lot of time doing mundane tasks like
cleansing, shaping and preparing data. Typically these tasks are roadblocks for
starting the more enjoyable part of analyzing the data sets or building and
training machine learning models. This is because data sets typically are not in a
format that can be readily used. They first need to be cleansed, refined before
they are useable by a data scientist. IBM Data Refinery addresses this issue and
simplifies the task of refining data and its workflows. It provides a self-service
data preparation environment where you can quickly analyze, cleanse and prepare
data sets. Data refinery is available with Watson Studio on public cloud, private
cloud and desktop. In the rest of the video we will walk through a scenario
and see Data Refinery in action. In this scenario we will use Data Refinery to
find the best deals using data about discounts offered
over time. We will then automate the analysis to run on a regular schedule. Before the Data
Scientist starts, she looks
at the data distribution and notices that the inSale column
is missing data. She visualizes the offer column and
notices that it contains valuable information about discounts. Many fields contain the percent
of
information, some contain references to previous
price indicating a new reduced price being available. She decides to derive sale from offer. She
uses a conditional decrease
operation to derive if the product is on sale. Next she uses a filter operation to eliminate deals
that are not on sale She then wants to pick up the bargains. She uses the
replace substring operation and provides a pattern that extracts the
discounts from the offer. After converting the discount values to a
decimal she can visually see the discounts that
were available. She needs to find the months that offered the best deals. She
visualizes the dateUpdated and notices that the date field has a variety of
formats, some with dashes some with slashes and some with months as text. She
hopes that Data Refinery can normalize the data and extract a month. She uses the
convert column operation to convert to date and selects ymd. Next she extracts
month and creates a derived column called discountMonth. The data now
represents all brands and products providing sales and the month the offer
was available. The data scientist is only interested in her preferred brands. Over
time she has built a list of preferred brands and has imported the data in her
project. Data Refinery provides relational transformations such as left, inner
right, full, semi and anti-join. To ensure that the data only contains her
preferred brand she uses a semi-join operation which narrows the brands to
match her preferences. She then selects the keys for the join
and the resulting fields. The visual results now confirms that the
brands match the preferences. To find the best possible deals
she needs to perform some aggregations. Several features determine a good deal. She is
interested in the best offer and duration when the discounts are active. Aggregating the sale
data will help understand the deals. She groups the
columns by brand and discountMonth and calculates the maximum discount. Finally
she sorts the result in descending order Data refinery is now displaying the
best deals by brand preferences and the duration which the offer is available. The last step is to
execute the analysis
on the full dataset. She starts the full analysis, which she can monitor for the
completion status. It's time to automate the analysis which
runs on a regular basis. The data in the database can grow over time. She uses a
personalized runtime to match the larger data volumes and sets a schedule for
automation. The hourly schedule reads from updated data from the database and
writes to the target table. Data Refinery has helped her uncover deals
in the raw data through a small set of operations and transformations with the
bulk of the work done for her. Thank you for watching
