Posts

Showing posts from May, 2024

Databricks Medallion Architecture: Data Modeling Guide, Best practices, Standards, Examples

-
  Hello Data Pros, and welcome back to another exciting blog in our Databricks learning series! In our last blog, we explored various platform architectures, specifically focusing on the modern Data Lakehouse architecture and its implementation using the Delta Lake framework. But today, we're moving to the next big question: With this powerful platform architecture in place, how do we organize and model our data effectively? And that’s where the Medallion Architecture comes in! So, what exactly is Medallion Architecture? It’s a data design pattern, developed to logically structure and organize your data within a Lakehouse! Its main purpose is to progressively improve the quality and usability of your data, as it moves through different stages, such as Bronze, Silver, and Gold. Think of it as a transformation journey, where raw data is refined step by step into a polished and analysis-ready state! Some people call it a multi-hop architecture because the data flows through several tr...
Post a Comment
Read more

DBT Sources, Seeds & Analyses | Data Build Tool Demo with Real World Examples

-
############  seeds\salestargets.csv  ############  StoreID,SalesTarget 1876,260000 1396,290000 1146,305000 1107,345000 1384,350000 1155,300000 1913,300000 1882,345000 1289,300000 1005,375000 ############  models\storeperformance.sql  ###########  SELECT     OS . StoreID ,     SUM ( OFACT . Revenue ) AS ActualSales ,     SUM ( ST . SalesTarget ) AS TargetSales FROM     {{ ref ('orders_stg') }} OS JOIN     {{ ref ('orders_fact') }} OFACT ON OS . OrderID = OFACT . OrderID JOIN     {{ ref (' salestargets ') }} ST ON ST . StoreID = OS . StoreID GROUP BY 1 ############   analyses\StoreRevenue.sql  ############  SELECT     OS . StoreID ,     SUM ( OFACT . Revenue ) AS TotalRevenue FROM     {{ ref ('orders_stg') }} OS JOIN     {{ ref ('orders_fact') }} OFACT ON OS . OrderID = OFACT . OrderID GROUP BY 1 ############   mod...
Post a Comment
Read more

SNOWFLAKE TABLES HANDS-ON LAB

-
------------------------------------------------------------------------------------------------------- --SNOWFLAKE TABLES HANDS-ON LAB  (ALL COMMANDS ARE AVAILABLE IN THE LINK PROVIDED INT HE DESCRIPTION)-- ------------------------------------------------------------------------------------------------------- -- SET THE ROLE & WAREHOUSE USE ROLE ACCOUNTADMIN ; USE WAREHOUSE COMPUTE_WH ; -- CREATE & USE DATABASE AND SCHEMAS CREATE DATABASE IF NOT EXISTS SLEEK_OMS ; CREATE SCHEMA IF NOT EXISTS SLEEK_OMS . L1_LANDING ; USE DATABASE SLEEK_OMS ; USE SCHEMA L1_LANDING ; -- CREATE PERMANENT TABLE CREATE TABLE EMPLOYEE_PER (   ID INT ,   NAME STRING ,   AGE INT ,   SALARY DECIMAL (10, 2) ); -- CREATE TEMPORARY TABLE CREATE TEMPORARY TABLE EMPLOYEE_TMP (   ID INT ,   NAME STRING ,   AGE INT ,   SALARY DECIMAL (10, 2) ); -- CREATE TRANSIENT TABLE CREATE TRANSIENT TABLE EMPLOYEE_TRN (   ID INT ,   NA...
Post a Comment
Read more