awesome-apache-airflow

Video of

Screencast showing how to use Breeze environment by

Talks from

Slides from

A CLI tool that includes everything required to create, manage and deploy airflow projects faster and smoother.

A plugin for Apache Airflow that allows you to edit DAGs in browser.

An extensible framework to do transformations to an Airflow DAG and convert it into another DAG which is flow-isomorphic with the original DAG, to be able to run it on different environments (e.g. on different clouds, or even different container frameworks - Apache Spark on YARN vs Kubernetes). Comes with out-of-the-box support for EMR-to-HDInsight-DAG transforms.

Plugin for open-source version-control system for data science and Machine Learning pipelines -

Plugin to refresh AWS ECR login token at regular intervals. This is helpful where DockerOperator needs to pull images hosted on ECR.

Overview of Airflow, concept, basic use with use case.

(🇯🇵Japanese)

(🇧🇷Portuguese)

(🇨🇳Chinese)

(🇯🇵Japanese)

Overview of Airflow, basic concepts and how to write and trigger a DAG.

Blog post about new ways of writing DAGs in Airflow 2.0.

Blog post about providers packages in Airflow 2.0.

A Manning book (Early Access September 2019) on Airflow.

A blogpost that outlines how you can set up remote S3 logging when using KubernetesExecutor, without creating complex infrastructure.

Maxime's blog on medium that gives insight into the philosophy behind Apache Airflow.

Blog posts about data engineering with Apache Airflow, explains why and has examples in code.

Lightweight installer of federated Airflow-Airflow (RabbitMQ) reference architectrure on Compute node(s).

Chef cookbook for deploying Airflow.

A detailed tutorial to get a scalable, low maintenance airflow kubernetes executor environment deployed on

A guide on all relevant resources, scripts and projects that relate to running Airflow on Kubernetes.

Airflow Docker container that comes preconfigured for Spark and Hadoop. It can be docker pulled at

A free Azure Resource Manager (ARM) template by Bitnami providing a one-click solution for Airflow deployment on Azure for production use-cases.

Efficently manage DAGs release process by using Git Submodules

Managing python package dependencies across 100+ dags can become painful. It's hard to keep track of which packages are used by which dag, and hard to clean up during DAG removal/upgrade. Learn how KubernetesPodOperator and DockerOperator can fix this.

Deep dive into troubleshooting a troublesome Airflow DAG with good tips on how to diagnosis problems.

A boilerplate repository for developing locally with Airflow, with linting & tests for valid DAGs and plugins. Just clone and run

This article discusses the basic concepts that stand behind Airflow and discusses the problems it solves.

A two-part series by