Airflow Tutorial - Xcom | How to Pass data between tasks
Hello Data Pros, In our last blog, we covered deferrable operators and triggers! Now, it’s time to explore Airflow's X-com feature! Let's dive right in! By design, Airflow tasks are isolated! which means they cannot exchange data with each other at run time! However, we frequently come across situations that require sharing data between tasks. For instance, you might need to extract a value from a table, and based on that value, perform something in the next task! or you may need to create a file with a dynamic name, such as one with a timestamp, and process the same file in the next task. This is where X-com comes into play. X-com, abbreviated as 'cross-communication,' provides a mechanism that allows tasks to exchange data with each other. Let’s consider this example Dag. In the first task, we create a file! And in the second task, we upload the same file to S3. With the current setup, this process works well! because we have the 'replace' parameter s...