DBT MODELS - TRANSFROMATION LOGIC
DBT SEEDS - CSV FILES
DBT SOURCES -
DBT MACROS
DBT VARIABLES
DBT HOOKS
DBT SNAPSHOTS
DBT INCREMENTAL
DBT EPHEMERAL
DBT TEST
DBT PACKAGES
============================
DBT UTILS
DBT EXPECTATIONS
table :
100 columns
select * from tb;
select col1,col2,col3,col4,col5..
from tb;
{{ codegen.generate_base_model(
source_name='raw_jaffle_shop',
table_name='customers',
materialized='view'
) }}
{{ codegen.generate_source(schema_name= 'raw_sc', database_name=
'dev_db',include_schema=True,
name='prav') }}
===========================
DEVELOPMENT
TESTING - DBT TEST
DBT DOCS GENEARATE
DATA LINAAGE
ELT DATA PIPEKINES ;
DEPOLYMNETS:
MOVING THE DB OBJCTE CFOME ONE EVBIRNINT TO ANITHE NEVIR
TABLE
DEV => TEST =>STG =>PORD
1.
github account creation
2. create 1 reposiirry
3. copy ssh
[email protected]:dbtpraveen3/dbt-sf-scripts.git
put the ssh value in the git clone
copy the ssh value and put in github
ci cd ??
DEV =>STG
DEV =>PROD
ORACLE TO SNOWFLAKR
ORACLE TO S3 ?? TASK
UI DIAGRAM
dev =>prod
=====================DEPOLYMENT
1. CREATE ENVIRONMENT
3 to 4 developers
code reposiory or version control
github
any devlopmemt:
we need to take the code from
laetst code
;
DEVELOPEMNT
ETSTING
PUT THE CODE IN THE REPOSIROY
MONTH QUKAREY O;
DEVOPS OR SUPPORT GUY THEY TRY TO DEPLOT YHE COIE IN DEIFFENT ENFIRNAONNT;
3RD PART ORCHETSTATION OR SCHEDULER TOOLS??
AIRFLOW/PERFECT
CONTROL-M
EXTERNAL DEPENDECCIDES ??
ANY FAILURE HAPPENDING
SLACK OR MAIL
WILL GET NOTIFICATION ?
====================
DBT PROJECT FLOW :
DIFFERNT SOURCE SYSTEM :
ORACEL
SQL SERVER
PISTGRE =>ELT TOOL(FIVETRAN)=> SNOWFLAKE()
SAP HANA
BACTH PROCESS ??
SNOWFLAKE(RAW_SC/RAW_sC1/RAW_SC2 TRANSIENT TABLE)
===================>DBT
SOURCES.YML ??? codegen genatesource
===>
we want bsic data maasaging
rmvoeing the spaces
based on the mapping docuente give proer name
column length
dbt models ??
spefcial charter ?? dbt macros
global variables
__var gunction
raw=>dbt =>stg(modles) =>sanitize(ref fucntoion)
i want to duplcaites
data quality ??DBT TEST(SINGUKAR GENRICE CUSTOM GENEIRC)
INCREMENATL
incrmenatl materilziation
SNApSHOTS
=================
final layer
i want to join multipole models
ref fucntion ??
=================
reporting views
{{config(materialized='view'}};
data sharing :
data unlaod
dbt macris
data sharing :
secure
{{config(materialized='view',secure=true}};
hooks
packages
SATURDAYA :
WORKING
8AM
I DID SOME CHANGE IN THE DEV ??
OCNCE WE COMMITED THE CIDE NEEDS TO AUTOAMTELY RFLECTED IN THE STG
