SleekData

  Hello Data Pros, welcome back to another episode of our Apache Airflow series! Today,  we're  taking a deep dive into the world of Task flow API!   The Task flow API was introduced to Airflow, starting from version 2.0, aiming to simplify the  dag  authoring experience by significantly reducing the number of code lines needed by developers! this also makes the code more modular, understandable and easily maintainable!   The  Taskflow  API was built based on the concept of Python decorators!  So  to understand  Taskflow  API, you first need to know the basics of decorators.   Think of a decorator as a special function that takes another function as its input and returns a modified version of that original function.   It's  like wrapping a gift: the decorator adds extra features to the original function.   Decorator is applied using the @ symbol, above a function definition.   In our case...

  Hello Data Pros,  In our last blog, we uncovered the need for airflow X-coms! and demonstrated how to leverage them effectively in your dags! Today, we're shifting our focus to Airflow hooks!  We’re going to cover what hooks are! How they differ from Airflow operators! Lastly, when and how to use hooks, in your dags! Let's dive right in!   Technically, Hooks are pre-built Python classes. They simplify our interactions with external systems and services. For instance, the popular S3Hook, which is part of the AWS provider package, offers various methods to interact with S3 Storage.   For example, the create bucket method, Creates an Amazon S3 bucket! Load string method – can load a string value as a file in S3! Delete objects method - can be used to delete an S3 file.   Now, let's dive into the source code of this Hook! As you can see, it's low-level Python code. And if AWS has not provided this hook, you might find yourself having to write all this complex...

Hello Data Pros,  In our last blog, we covered the fundamental concepts of Apache Airflow, including  dags , tasks and operators! In addition, we demonstrated the importance of the airflow configuration file, and the provider packages!   Today, we’ll learn about the power of Airflow Variables and Connections and how to use them effectively within your  Dags .   Let's start with Apache Airflow Variables! Variables are like little storage containers for values, that you can reuse across your  dags . Instead of hardcoding values, you can store them as variables, and reference them with its name whenever needed.   Technically each variable is a key and a value pair. You can think of the key as the variable name and the value as the data it holds.   There are two types of Airflow variables! One, Regular variables, Where the value can be any string. Two, JSON variables, where the value is a JSON string.   Let’s consider I have a  dag , that in...