SleekData

Hello Data Pros,  In our last blog, we covered the fundamental concepts of Apache Airflow, including  dags , tasks and operators! In addition, we demonstrated the importance of the airflow configuration file, and the provider packages!   Today, we’ll learn about the power of Airflow Variables and Connections and how to use them effectively within your  Dags .   Let's start with Apache Airflow Variables! Variables are like little storage containers for values, that you can reuse across your  dags . Instead of hardcoding values, you can store them as variables, and reference them with its name whenever needed.   Technically each variable is a key and a value pair. You can think of the key as the variable name and the value as the data it holds.   There are two types of Airflow variables! One, Regular variables, Where the value can be any string. Two, JSON variables, where the value is a JSON string.   Let’s consider I have a  dag , that in...

  Hello Data Pros, In our previous blog, we explored what Airflow is! covered essential concepts such as Dags, Tasks and Operators! We also dissected its architecture and core components!   In this video, we'll demonstrate how to set up Airflow on your local machine and create your first Airflow Dag!   Let's begin right away!   I’m using windows, but the same approach works well on macOS as well.     As of this video, Airflow is not officially supported on Windows, so we'll be installing Docker and running Airflow on top of it.   Docker is a software containerization platform designed for developing, shipping, and running applications. It packages the entire application along with its dependencies and configurations within a standardized unit known as a container. These containerized applications are known for their consistency, repeatability, and portability across different operating systems.     Please download and install 'Docker Des...