Data engineering
Data engineering build a system which collects
Data from various source like RDBMS, EXCEL,
CSV,PDF ETC.. AND STORE IN A WAREHOUSE
FOR PREDICTIVE ANALYSIS , DATA
SCIENCE ,MACHINE LEARNING & AI
DATA CAN BE IN THE FORM OF STRUCTURE ,
SEMISTRUCTURE, UNSTURCTURE.
EARLIER TIME DATA WAS GENERATED VERY
LESS BUT NOW HUGE VOLUMES OF DATA IS
GENERATING , HOW TO USE THIS DATA AND
MAKE KEY DESCISSION FOR THEIR BUSINESS
DEVELOPMENT.
DATA ENGINEERS ROLE IS TO MAKE DATA
PIPELINE FROM VARIOUS SOURCE TO STORE
DATA IN A DATA WAREHOUSE TO MAKE A
PROPER DECISSION MAKING BY DATA
SCIENTIST AND BUSINESS ANALYST.
SKILLS FOR DATA ENGINEERS
1. SQL (FULL KNOWLEDGE)
[Link] (BASIC KNOWLEDGE DATA TYPE,
LOOPS STATEMENT, LIST, TUPLE,
DCITIONARY,CLASSES FUNCTION ETC.. )
[Link] COMPUTING FRAMEWORK
LIKE HADOOP BASIC (SLOW PERFORMANCE )AND
SPARK ARCHTECTURE(FAST PERFORMANCE) AND
ITS USES.
[Link](PYTHON + SPARK)
[Link] TECHNOLOGY (AWS, AZURE, GCP, OCI)
AZURE SERVICES:-
i) AZURE DATA FACTORY :-IT IS DATA
ORCHESTRATION TOOL WHICH IS USED FOR
ETL OR ELT PROCESS. IT IS TOTALLY NO
CODING TOOL ONLY DRAG AN DROP
OPTIONS
ii) AZURE DATA BRICKS:- DATABRICKS IS A
CLOUD VERSION OF SPARK(DATA BRICKS
AND MICROSOFT JOINTLY STARTED)
HERE WE CAN WRITE A PYTHON ,
R,SCALA ,JAVA CODE
iii)AZURE SYNAPSE ANALYTICS FOR DATA
WAREHOUSE
DATABASE
(OLTP SYSTEM)
EXCEL
ADF(AZURE DATA ADB (AZURE DATAWARE
FACTORY COLLECT DATABRICKS HOUSE(SQL DB) BI
DATA FROM DIFFERENT PROCESS
SOURCE
BIGDATA )
CSV
DATA
INTEGRATION(ETL / ELT
)
IOT/ INTERNET ADLS
GEN2(STORAGE)
DATA ENGINEER DATA ANALYST
ANY QUESTON?
