Posts

Showing posts from June, 2024

Databricks Medallion Architecture: Data Modeling Guide, Best practices, Standards, Examples

-
  Hello Data Pros, and welcome back to another exciting blog in our Databricks learning series! In our last blog, we explored various platform architectures, specifically focusing on the modern Data Lakehouse architecture and its implementation using the Delta Lake framework. But today, we're moving to the next big question: With this powerful platform architecture in place, how do we organize and model our data effectively? And that’s where the Medallion Architecture comes in! So, what exactly is Medallion Architecture? It’s a data design pattern, developed to logically structure and organize your data within a Lakehouse! Its main purpose is to progressively improve the quality and usability of your data, as it moves through different stages, such as Bronze, Silver, and Gold. Think of it as a transformation journey, where raw data is refined step by step into a polished and analysis-ready state! Some people call it a multi-hop architecture because the data flows through several tr...
Post a Comment
Read more

Macros in dbt | Data Build Tool | Jinja and macros tutorial

-
  oms_dbt_proj\macros\oms_common.sql {% macro to_celsius ( fahrenheit_column , decimal_places = 1 ) %}   ROUND ( ({{ fahrenheit_column }} - 32 ) * 5 / 9 , {{ decimal_places }}) {% endmacro %} {% macro generate_profit_model ( table_name ) %} SELECT   sales_date ,   SUM ( quantity_sold * unit_sell_price ) as total_revenue ,   SUM ( quantity_sold * unit_purchase_cost ) as total_cost ,   SUM ( quantity_sold * unit_sell_price ) - SUM ( quantity_sold * unit_purchase_cost ) as total_profit FROM {{ source ( 'training' , table_name ) }} GROUP BY sales_date {% endmacro %} oms_dbt_proj\models\profit_uk.sql {{ generate_profit_model ( 'sales_uk' ) }}
Post a Comment
Read more

DBT doc blocks | DBT Docs | dbt documentation best practices

-
  oms_config.yml models :   - name : customers_stg     description : Staged customer data from order management system (OMS), with minor row-level transformations.       columns :       - name : Email         description : Customer's Primay Email address for promotions and offers.               tests :           - string_not_empty   - name : employees_stg     description : Staged employees data from order management system (oms), with minor row-level transformations.       columns :       - name : JobTitle         description : Employee's Job Title based on his current Roles and Responsibilities.               tests :           - string_not_empty   - name : orders_stg     description : Staged orders data from order ma...
Post a Comment
Read more