KDD vs Data Mining

KDD (Knowledge Discovery in Databases) is a field of computer science, which includes the tools and theories to help
humans in extracting useful and previously unknown information (i.e., knowledge) from large collections of digitized data.
KDD consists of several steps, and Data Mining is one of them. Data Mining is the application of a specific algorithm to
extract patterns from data. Nonetheless, KDD and Data Mining are used interchangeably.

What is KDD?

KDD is a computer science field specializing in extracting previously unknown and interesting information from raw data.
KDD is the whole process of trying to make sense of data by developing appropriate methods or techniques. This process
deals with low-level mapping data into other forms that are more compact, abstract, and useful. This is achieved by creating
short reports, modeling the process of generating data, and developing predictive models that can predict future cases.

Due to the exponential growth of data, especially in areas such as business, KDD has become a very important process to
convert this large wealth of data into business intelligence, as manual extraction of patterns has become seemingly
impossible in the past few decades.

For example, it is currently used for various applications such as social network analysis, fraud detection, science,
investment, manufacturing, telecommunications, data cleaning, sports, information retrieval, and marketing. KDD is usually
used to answer questions like what are the main products that might help to obtain high-profit next year in V-Mart.

KDD Process Steps

Knowledge discovery in the database process includes the following steps, such as:

1. Goal identification: Develop and understand the application domain and the relevant prior knowledge and identify
the KDD process's goal from the customer perspective.

2. Creating a target data set: Selecting the data set or focusing on a set of variables or data samples on which the
discovery was made.

3. Data cleaning and preprocessing:Basic operations include removing noise if appropriate, collecting the necessary
information to model or account for noise, deciding on strategies for handling missing data fields, and accounting for
time sequence information and known changes.

4. Data reduction and projection: Finding useful features to represent the data depending on the purpose of the task.
The effective number of variables under consideration may be reduced through dimensionality reduction methods or
conversion, or invariant representations for the data can be found.
 5. Matching process objectives: KDD with step 1 a method of mining particular. For example, summarization,
classification, regression, clustering, and others.

6. Modeling and exploratory analysis and hypothesis selection: Choosing the algorithms or data mining and
selecting the method or methods to search for data patterns. This process includes deciding which model and
parameters may be appropriate (e.g., definite data models are different models on the real vector) and the matching of
data mining methods, particularly with the general approach of the KDD process (for example, the end-user might be
more interested in understanding the model in its predictive capabilities).

7. Data Mining: The search for patterns of interest in a particular representational form or a set of these
representations, including classification rules or trees, regression, and clustering. The user can significantly aid the
data mining method to carry out the preceding steps properly.

8. Presentation and evaluation: Interpreting mined patterns, possibly returning to some of the steps between steps 1
and 7 for additional iterations. This step may also involve the visualization of the extracted patterns and models or
visualization of the data given the models drawn.

9. Taking action on the discovered knowledge: Using the knowledge directly, incorporating the knowledge in another
system for further action, or simply documenting and reporting to stakeholders. This process also includes checking
and resolving potential conflicts with previously believed knowledge (or extracted).

What is Data Mining?

Data mining, also known as Knowledge Discovery in Databases, refers to the nontrivial extraction of implicit, previously
unknown, and potentially useful information from data stored in databases.

Data Mining is only a step within the overall KDD process. There are two major Data Mining goals defined by the
application's goal: verification of discovery. Verification verifies the user's hypothesis about data, while discovery
automatically finds interesting patterns.

There are four major data mining tasks: clustering, classification, regression, and association (summarization). Clustering is
identifying similar groups from unstructured data. Classification is learning rules that can be applied to new data. Regression
is finding functions with minimal error to model data. And the association looks for relationships between variables. Then,
the specific data mining algorithm needs to be selected. Different algorithms like linear regression, logistic regression,
decision trees, and Naive Bayes can be selected depending on the goal. Then patterns of interest in one or more symbolic
forms are searched. Finally, models are evaluated either using predictive accuracy or understandability.

Why do we need Data Mining?

The volume of information is increasing every day that we can handle from business transactions, scientific data, sensor data,
Pictures, videos, etc. So, we need a system that will be capable of extracting the essence of information available and that can
automatically generate reports, views, or summaries of data for better decision-making.
Why is Data Mining used in business?

Data mining is used in business to make better managerial decisions by:

● Automatic summarization of data.

● Discovering patterns in raw data.

● Extracting the essence of information stored.

Why KDD and Data Mining?

In an increasingly data-driven world, there would never be such a thing as too much data. However, data is only valuable
when you can parse, sort, and sift through it to extrapolate the actual value.

Most industries collect massive volumes of data, but without a filtering mechanism that graphs, charts, and trends data
models, pure data itself has little use.

However, the sheer volume of data and the speed with which it is collected makes sifting through it challenging. Thus, it has
become economically and scientifically necessary to scale up our analysis capability to handle the vast amount of data that
we now obtain.

Since computers have allowed humans to collect more data than we can process, we naturally turn to computational
techniques to help us extract meaningful patterns and structures from vast amounts of data.

Difference between KDD and Data Mining

Although the two terms KDD and Data Mining are heavily used interchangeably, they refer to two related yet slightly
different concepts.

KDD is the overall process of extracting knowledge from data, while Data Mining is a step inside the KDD process, which
deals with identifying patterns in data.

And Data Mining is only the application of a specific algorithm based on the overall goal of the KDD process.

KDD is an iterative process where evaluation measures can be enhanced, mining can be refined, and new data can be
integrated and transformed to get different and more appropriate results.

Why we need Data Mining?

Volume of information is increasing everyday that we can handle from business transactions, scientific data, sensor data,
Pictures, videos, etc. So, we need a system that will be capable of extracting essence of information available and that can
automatically generate report,

views or summary of data for better decision-making.

Why Data Mining is used in Business?

Data mining is used in business to make better managerial decisions by:

 ● Automatic summarization of data
● Extracting essence of information stored.
● Discovering patterns in raw data.

Data Mining also known as Knowledge Discovery in Databases, refers to the nontrivial extraction of implicit, previously
unknown and potentially useful information from data stored in databases.

Steps Involved in KDD Process:

KDD process

1. Data Cleaning: Data cleaning is defined as removal of noisy and irrelevant data from collection.
○ Cleaning in case of Missing values.
○ Cleaning noisy data, where noise is a random or variance error.
○ Cleaning with Data discrepancy detection and Data transformation tools.
2. Data Integration: Data integration is defined as heterogeneous data from multiple sources combined in a
common source(DataWarehouse).
○ Data integration using Data Migration tools.
○ Data integration using Data Synchronization tools.
○ Data integration using ETL(Extract-Load-Transformation) process.
3. Data Selection: Data selection is defined as the process where data relevant to the analysis is decided and
retrieved from the data collection.
○ Data selection using Neural network.
 ○ Data selection using Decision Trees.
○ Data selection using Naive bayes.
○ Data selection using Clustering, Regression, etc.
4. Data Transformation: Data Transformation is defined as the process of transforming data into appropriate form
required by mining procedure.
Data Transformation is a two step process:
○ Data Mapping: Assigning elements from source base to destination to capture transformations.
○ Code generation: Creation of the actual transformation program.
5. Data Mining: Data mining is defined as clever techniques that are applied to extract patterns potentially useful.
○ Transforms task relevant data into patterns.
○ Decides purpose of model using classification or characterization.
6. Pattern Evaluation: Pattern Evaluation is defined as identifying strictly increasing patterns representing
knowledge based on given measures.
○ Find interestingness score of each pattern.
○ Uses summarization and Visualization to make data understandable by user.
7. Knowledge representation: Knowledge representation is defined as technique which utilizes visualization tools
to represent data mining results.
○ Generate reports.
○ Generate tables.
○ Generate discriminant rules, classification rules, characterization rules, etc.

Note:

● KDD is an iterative process where evaluation measures can be enhanced, mining can be refined, new data can be
integrated and transformed in order to get different and more appropriate results.
● Preprocessing of databases consists of Data cleaning and Data Integration.

Data Mining Techniques

Data mining includes the utilization of refined data analysis tools to find previously unknown, valid patterns and
relationships in huge data sets. These tools can incorporate statistical models, machine learning techniques, and mathematical
algorithms, such as neural networks or decision trees. Thus, data mining incorporates analysis and prediction.
Depending on various methods and technologies from the intersection of machine learning, database management, and
statistics, professionals in data mining have devoted their careers to better understanding how to process and make
conclusions from the huge amount of data, but what are the methods they use to make it happen?

In recent data mining projects, various major data mining techniques have been developed and used, including association,
classification, clustering, prediction, sequential patterns, and regression.

1. Classification:

This technique is used to obtain important and relevant information about data and metadata. This data mining technique
helps to classify data in different classes.

Data mining techniques can be classified by different criteria, as follows:

1. Classification of Data mining frameworks as per the type of data sources mined:
This classification is as per the type of data handled. For example, multimedia, spatial data, text data, time-series
data, World Wide Web, and so on..

2. Classification of data mining frameworks as per the database involved:

This classification based on the data model involved. For example. Object-oriented database, transactional database,
relational database, and so on..

3. Classification of data mining frameworks as per the kind of knowledge discovered:

This classification depends on the types of knowledge discovered or data mining functionalities. For example,
 discrimination, classification, clustering, characterization, etc. some frameworks tend to be extensive frameworks
offering a few data mining functionalities together..

4. Classification of data mining frameworks according to data mining techniques used:

This classification is as per the data analysis approach utilized, such as neural networks, machine learning, genetic
algorithms, visualization, statistics, data warehouse-oriented or database-oriented, etc.
The classification can also take into account, the level of user interaction involved in the data mining procedure, such
as query-driven systems, autonomous systems, or interactive exploratory systems.

2. Clustering:

Clustering is a division of information into groups of connected objects. Describing the data by a few clusters mainly loses
certain confine details, but accomplishes improvement. It models data by its clusters. Data modeling puts clustering from a
historical point of view rooted in statistics, mathematics, and numerical analysis. From a machine learning point of view,
clusters relate to hidden patterns, the search for clusters is unsupervised learning, and the subsequent framework represents a
data concept. From a practical point of view, clustering plays an extraordinary job in data mining applications. For example,
scientific data exploration, text mining, information retrieval, spatial database applications, CRM, Web analysis,
computational biology, medical diagnostics, and much more.

In other words, we can say that Clustering analysis is a data mining technique to identify similar data. This technique helps to
recognize the differences and similarities between the data. Clustering is very similar to the classification, but it involves
grouping chunks of data together based on their similarities.

3. Regression:

Regression analysis is the data mining process is used to identify and analyze the relationship between variables because of
the presence of the other factor. It is used to define the probability of the specific variable. Regression, primarily a form of
planning and modeling. For example, we might use it to project certain costs, depending on other factors such as availability,
consumer demand, and competition. Primarily it gives the exact relationship between two or more variables in the given data
set.

4. Association Rules:

This data mining technique helps to discover a link between two or more items. It finds a hidden pattern in the data set.

Association rules are if-then statements that support to show the probability of interactions between data items within large
data sets in different types of databases. Association rule mining has several applications and is commonly used to help sales
correlations in data or medical data sets.

The way the algorithm works is that you have various data, For example, a list of grocery items that you have been buying
for the last six months. It calculates a percentage of items being purchased together.

These are three major measurements technique:

● Lift:
This measurement technique measures the accuracy of the confidence over how often item B is purchased.
(Confidence) / (item B)/ (Entire dataset)
 ● Support:
This measurement technique measures how often multiple items are purchased and compared it to the overall dataset.
(Item A + Item B) / (Entire dataset)

● Confidence:
This measurement technique measures how often item B is purchased when item A is purchased as well.
(Item A + Item B)/ (Item A)

5. Outer detection:

This type of data mining technique relates to the observation of data items in the data set, which do not match an expected
pattern or expected behavior. This technique may be used in various domains like intrusion, detection, fraud detection, etc. It
is also known as Outlier Analysis or Outilier mining. The outlier is a data point that diverges too much from the rest of the
dataset. The majority of the real-world datasets have an outlier. Outlier detection plays a significant role in the data mining
field. Outlier detection is valuable in numerous fields like network interruption identification, credit or debit card fraud
detection, detecting outlying in wireless sensor network data, etc.

6. Sequential Patterns:

The sequential pattern is a data mining technique specialized for evaluating sequential data to discover sequential patterns.
It comprises of finding interesting subsequences in a set of sequences, where the stake of a sequence can be measured in
terms of different criteria like length, occurrence frequency, etc.

In other words, this technique of data mining helps to discover or recognize similar patterns in transaction data over some
time.

7. Prediction:

Prediction used a combination of other data mining techniques such as trends, clustering, classification, etc. It analyzes past
events or instances in the right sequence to predict a future event.

Data Mining tools

Data Mining is the set of techniques that utilize specific algorithms, statical analysis, artificial intelligence, and database
systems to analyze data from different dimensions and perspectives.

Data Mining tools have the objective of discovering patterns/trends/groupings among large sets of data and transforming data
into more refined information.

1. Orange Data Mining:

Orange is a perfect machine learning and data mining software suite. It supports the visualization and is a software-based on
components written in Python computing language and developed at the bioinformatics laboratory at the faculty of computer
and information science, Ljubljana University, Slovenia.
As it is a software-based on components, the components of Orange are called "widgets." These widgets range from
preprocessing and data visualization to the assessment of algorithms and predictive modeling.

Widgets deliver significant functionalities such as:

● Displaying data table and allowing to select features

● Data reading

● Training predictors and comparison of learning algorithms

● Data element visualization, etc.

Besides, Orange provides a more interactive and enjoyable atmosphere to dull analytical tools. It is quite exciting to operate.

2. SAS Data Mining:

SAS stands for Statistical Analysis System. It is a product of the SAS Institute created for analytics and data management.
SAS can mine data, change it, manage information from various sources, and analyze statistics. It offers a graphical UI for
non-technical users.

SAS data miner allows users to analyze big data and provide accurate insight for timely decision-making purposes. SAS has
distributed memory processing architecture that is highly scalable. It is suitable for data mining, optimization, and text
mining purposes.

3. DataMelt Data Mining:

DataMelt is a computation and visualization environment which offers an interactive structure for data analysis and
visualization. It is primarily designed for students, engineers, and scientists. It is also known as DMelt.

DMelt is a multi-platform utility written in JAVA. It can run on any operating system which is compatible with JVM (Java
Virtual Machine). It consists of Science and mathematics libraries.

● Scientific libraries:
Scientific libraries are used for drawing the 2D/3D plots.

● Mathematical libraries:
Mathematical libraries are used for random number generation, algorithms, curve fitting, etc.

Melt can be used for the analysis of the large volume of data, data mining, and statistical analysis. It is extensively used in
natural sciences, financial markets, and engineering.

4. Rattle:

Ratte is a data mining tool based on GUI. It uses the R stats programming language. Rattle exposes the statical power of R by
offering significant data mining features. While rattle has a comprehensive and well-developed user interface, It has an
integrated log code tab that produces duplicate code for any GUI operation.

The data set produced by Rattle can be viewed and edited. Rattle gives the other facility to review the code, use it for many
purposes, and extend the code without any restriction.
5. Rapid Miner:

Rapid Miner is one of the most popular predictive analysis systems created by the company with the same name as the Rapid
Miner. It is written in JAVA programming language. It offers an integrated environment for text mining, deep learning,
machine learning, and predictive analysis.

The instrument can be used for a wide range of applications, including company applications, commercial applications,
research, education, training, application development, machine learning.

Rapid Miner provides the server on-site as well as in public or private cloud infrastructure. It has a client/server model as its
base. A rapid miner comes with template-based frameworks that enable fast delivery with few errors(which are commonly
expected in the manual coding writing process.

Business intelligence vs. data mining

Business intelligence and data mining differ in a few core ways. Namely, in purpose, volume, and results.

The purpose of business intelligence is to convert data into useful information for executives. Business intelligence tracks
key performance indicators and presents data in a way that encourages data-driven decisions. By contrast, data mining is
geared towards exploring data and finding solutions to particular business issues. Data mining has the computational
intelligence and algorithms to detect patterns that are interpreted and presented to management via business intelligence.

In that same vein, data mining is most optimal for processing datasets concentrated on a particular department, customer
segment, or competitor(s). By analyzing these smaller datasets, data mining can reveal hidden answers to specific business
questions. Unlike the specificity of data mining, business intelligence processes dimensional or relational databases in order
to deduce how an enterprise is performing on the whole.

Since data mining is more oriented towards getting data into a usable format and resolving unique business problems, the
results of data mining are unique datasets. Conversely business intelligence results are presented in charts, graphs,
dashboards, and reports. Displaying BI results is vital to influence data-driven decisions.

Lastly, data mining and business intelligence differ in their focus. Studying patterns through data mining helps companies
develop new KPIs for business intelligence. Business intelligence is therefore focused on showing progress towards data
mining-defined KPIs. Broad metrics like total revenue, total customer support tickets, and ARR over time paint a holistic
picture of company performance and give stakeholders the confidence to make significant decisions.

Feature Data Mining Business Intelligence

Purpose Exploring and formatting data to find answers Interpreting and presenting data to stakeholders
to business problems to inform data-driven decisions
 Volume Processes small, specific datasets for focused Processes relational databases to track
analysis enterprise-level metrics

Results Unique datasets in a usable data format Dashboards, graphs, charts, reports

Focus Identifying new KPIs Demonstrating KPI progress

Difference between Business Intelligence and Data Mining

1. Business Intelligence :

The term Business Intelligence (BI) alludes to advances, applications, and hones for the collection, integration, examination,
and introduction of business data. The reason for Commerce Insights is to bolster superior trade choice making. Basically,
Trade Insights frameworks are data-driven Decision Support Systems (DSS). Business Intelligence is now and then utilized
traded with briefing books, reports and inquiry instruments, and official data frameworks.

Business Intelligence frameworks give authentic, current, and prescient sees of commercial operations, most frequently
utilizing information that has been assembled into an information stockroom or an information shop and sometimes working
from operational information.

2. Data Mining :

It is the extraction of covered up prescient data from expansive databases could be a powerful modern innovation with great
potential to assist companies to focus on the foremost vital data in their data warehouses. It contains a gigantic scope is little
as well as enormous organizations. Data mining is essentially is utilized in the inverse course to that of Information
warehousing. By analyzing client’s information of a company, data mining apparatuses can construct a prescient demonstrate
that can tell you which clients are at chance or misfortune.

Difference between Business Intelligence and Data Mining :

Business Intelligence Data Mining

 Designed to investigate information
Changing over raw information into
and discover the arrangement for an
valuable data for business.
issue in business.

Data-driven makes a difference in choice Finds answers to an issue or a issue

making for a business. in trade.

Expansive Datasets processed on

dimensional / social databases Small handled on little parcel of data.
datasets

Uses calculations to distinguish

Volumetric in nature and display the exact
precise designs for an issue and
result utilizing visualizations.
distinguishes the daze spots.

Identifies the arrangement for an

Dashboards and Reports spoken to by charts
issue to be spoken to as one of the
and charts with KPI’s.
KPI’s in Dashboards or reports.

Depends on small-scale of past information, Focused on a specific issue in trade

there’s no intelligence involved; on small-scale information utilizing
administration needs to take the choice calculations to discover the
based on the information. arrangement.

Appears cost esteem, benefit, add up to Identifies arrangement for an issue

fetched, etc. as KPI’s. making modern KPI’s for BI
 Data Mining will unravel a specific
Business Intelligence makes a difference in
issue and contribute to
Decision-making .
decision-making.

Business Intelligence consists of creation, Data Mining consists of cleaning,

aggregation, analysis and visualization of combining, transforming and
data. interpretation of data.

Feature Data MiningDData mining BI

Purpose Extract data to solve business problems Visualizing & presenting data to
stakeholders

Volume Work on smaller data sets for focused Work on relational databases for
insights organizational-level insights

Results Unique data sets in a usable format Dashboards, pie charts, graphs, histograms,
etc.

Focus Highlight key performance indicators Indicate progress on KPIs

Tools Data mining techniques use tools like Business Intelligence techniques use tools
DataMelt, Orange Data Mining, R, Python, like Sisense, SAP for BI, Dundas BI, and
and Rattle GUI Tableau

How is Data Mining Used in Business Intelligence?

The way we use data mining for business analytics and intelligence varies from one business to another. But there is a

structure to this business process management that remains pretty much iron clad. Here’s a look at it.
Business Understanding

If you are undertaking data mining for business analytics and want it to be successful then begin by identifying the purpose of

data mining. Subsequent steps in the plan could tackle how to use the newfound data bits. Ideating your data mining

algorithm would be a far-fetched task lest you underline the purpose of data mining concisely.

Data Understanding

After getting to know the purpose of data mining it is time to get a touch and feel for your data. There could be just as many

ways to store and monetize data as there are businesses. How you create, curate, categorize, and commercialize your data is

upto your enterprise IT strategy and practices.

Data Preparation

Considered one of the most important stages in the course of nurturing data mining for business intelligence, company data

needs expert handling. Data engineers convert data into a readable format that non-IT professionals can interpret in addition

to cleansing and modeling it as per specific attributes.

Data Modeling

Statistical algorithms are deployed to decipher hidden patterns in data. A lot of trial and error goes into finding relevant

trends that can enhance revenue metrics.

Data Evaluation
The steps involved in data modeling should be evaluated microscopically for inconsistencies. Remember, all roads (must)

lead to streamlining operations and augmenting profits.

Implementation

The final step is to act on the findings in an observable way. Field trials of the recommendations should be piloted at a

smaller scale and then expanded onto branch outlets upon validation.

Now you know how the build-up of milestones distills into ground reality. Let us explore some of the technicalities of data

mining for business intelligence.