Databricks Medallion Architecture: Data Modeling Guide, Best practices, Standards, Examples

  Hello Data Pros, and welcome back to another exciting blog in our Databricks learning series! In our last blog, we explored various platform architectures, specifically focusing on the modern Data Lakehouse architecture and its implementation using the Delta Lake framework. But today, we're moving to the next big question: With this powerful platform architecture in place, how do we organize and model our data effectively? And that’s where the Medallion Architecture comes in! So, what exactly is Medallion Architecture? It’s a data design pattern, developed to logically structure and organize your data within a Lakehouse! Its main purpose is to progressively improve the quality and usability of your data, as it moves through different stages, such as Bronze, Silver, and Gold. Think of it as a transformation journey, where raw data is refined step by step into a polished and analysis-ready state! Some people call it a multi-hop architecture because the data flows through several tr...

Snowflake Storage Layer Key Concepts | Micropartition | Columnar Storage | Partition pruning

 

Hello Data Pros, and welcome back to another exciting episode of our Snowflake learning series!

In our previous videos, we explored Snowflake's storage and processing layers, covering key concepts along the way!

Now, let's shift our focus to the Cloud Services Layer - the mastermind behind everything that happens in Snowflake!


--All commands and sample csv file in the video description--
create database SLEEKDATA;

create schema OMS;
create stage MY_STAGE;

create table MY_TABLE (
ID int,
Name varchar(100),
Age int,
Region varchar(10)
);

-- Run put command in SNOWSQL CLI
-- put file://D:/mydata.csv @MY_STAGE;

copy into MY_TABLE from @MY_STAGE/mydata.csv.gz;

select * from MY_TABLE


mydata.csv

1,Alice,25,US
2,Bob,30,EU
3,Charlie,28,AP
4,David,35,US
5,Emma,40,EU
6,Frank,32,AP
7,Grace,27,US
8,Henry,45,EU
9,Ivan,50,AP
10,Jasmine,23,US


Comments

Popular posts from this blog

How to Install Airflow on Windows

How to Install DBT and Set Up a Project, Create Your First dbt Model

Airflow DAGs, Operators, Tasks & Providers