Posts

Showing posts from July, 2024

Databricks Medallion Architecture: Data Modeling Guide, Best practices, Standards, Examples

-
  Hello Data Pros, and welcome back to another exciting blog in our Databricks learning series! In our last blog, we explored various platform architectures, specifically focusing on the modern Data Lakehouse architecture and its implementation using the Delta Lake framework. But today, we're moving to the next big question: With this powerful platform architecture in place, how do we organize and model our data effectively? And that’s where the Medallion Architecture comes in! So, what exactly is Medallion Architecture? It’s a data design pattern, developed to logically structure and organize your data within a Lakehouse! Its main purpose is to progressively improve the quality and usability of your data, as it moves through different stages, such as Bronze, Silver, and Gold. Think of it as a transformation journey, where raw data is refined step by step into a polished and analysis-ready state! Some people call it a multi-hop architecture because the data flows through several tr...
Post a Comment
Read more

Snowflake Tasks | Types | Examples | snowflake tasks and streams

-
Hello Data Pros, and welcome back to another exciting episode of our Snowflake learning series! In our previous video, we’ve covered Snowflake Streams; and demonstrated their role in change data capture. Today, we will explore Snowflake Tasks, which are essential for automating a variety of data processes within Snowflake. ------------------- Follow the link in the description for the SQLs used in this lab ------------------- --------------------------------------------------------------------------- ---------------- Setting Up Foundational Objects for the Lab--------------- --------------------------------------------------------------------------- --Create Database and Schema CREATE OR REPLACE DATABASE order_db ; CREATE OR REPLACE SCHEMA order_db . order_schema ; -- Create Tables USE SCHEMA order_db . order_schema ; CREATE or replace TABLE order_raw (     order_id INT ,     order_date DATE ,     cust_fname VARCHAR( 50 ),     cust_lna...
Post a Comment
Read more

Snowflake Zero copy Cloning | How to Clone Database Schema Table View Stage | Examples | Tutorial

-
  Hello Data Pros, and welcome back to another exciting episode of our Snowflake learning series! In our last video, we explained what time travel is! and demonstrated how it works with appropriate examples! Today, we'll be exploring cloning; most widely known as zero-copy cloning within the Snowflake ecosystem! Let’s jump right in! ----------------------------------------------------------------------------------------------------------- --                                         SNOWFLAKE CLONING LAB                                         -- --               All commands/statements can be found via the link in the Video Description              -- -------------------------------------------------------------------...
Post a Comment
Read more

Snowflake Time Travel & Fail-safe | What is Continuous data protection | How to | with Examples

-
  Hello Data Pros, and welcome back to another exciting episode of our Snowflake learning series! In our last video, we explored various types of Snowflake views and their practical applications, demonstrated with easy-to-follow examples! Today, we're diving deeper into Snowflake's data protection features, specifically Time Travel and Fail-safe! ----------------------------------------------------------------------------------------------------------- --                                        SNOWFLAKE TIME TRAVEL LAB                                      -- --               All commands/statements can be found via the link in the Video Description              -- ----------------------------------------------------------...
Post a Comment
Read more