SleekData

  Hello Data Pros,  In our last blog, we uncovered the need for airflow X-coms! and demonstrated how to leverage them effectively in your dags! Today, we're shifting our focus to Airflow hooks!  We’re going to cover what hooks are! How they differ from Airflow operators! Lastly, when and how to use hooks, in your dags! Let's dive right in!   Technically, Hooks are pre-built Python classes. They simplify our interactions with external systems and services. For instance, the popular S3Hook, which is part of the AWS provider package, offers various methods to interact with S3 Storage.   For example, the create bucket method, Creates an Amazon S3 bucket! Load string method – can load a string value as a file in S3! Delete objects method - can be used to delete an S3 file.   Now, let's dive into the source code of this Hook! As you can see, it's low-level Python code. And if AWS has not provided this hook, you might find yourself having to write all this complex...

  Hello Data Pros, In our previous blog, we explored what Airflow is! covered essential concepts such as Dags, Tasks and Operators! We also dissected its architecture and core components!   In this video, we'll demonstrate how to set up Airflow on your local machine and create your first Airflow Dag!   Let's begin right away!   I’m using windows, but the same approach works well on macOS as well.     As of this video, Airflow is not officially supported on Windows, so we'll be installing Docker and running Airflow on top of it.   Docker is a software containerization platform designed for developing, shipping, and running applications. It packages the entire application along with its dependencies and configurations within a standardized unit known as a container. These containerized applications are known for their consistency, repeatability, and portability across different operating systems.     Please download and install 'Docker Des...