In a previous post, we discussed Setting up an Apache Airflow Cluster. In this post we’ll talk about the shortcomings of a typical Apache Airflow Cluster and what can be done to provide a Highly Available Airflow Cluster. A Typical Apache Airflow Cluster In a typical multi-node Airflow cluster you can separate out all the major processes onto separate machines. Here are the main processes: Web Server A daemon which accepts HTTP requests and allows you to interact with Airflow via a Python Flask Web Application. It provides the ability to pause, unpause DAGs, manually trigger DAGs, view running DAGs, restart failed DAGs and much more. Scheduler A daemon which periodically polls to determine if any registered DAG and/or Task Instances needs to triggered based off its schedule. Executors/Workers A daemon that handles starting up and managing 1 to many CeleryD processes to execute the desired tasks of a particular DAG. High Availability in a…