In one of our previous blog posts, we described the process you should take when Installing and Configuring Apache Airflow. In this post, we will describe how to setup an Apache Airflow Cluster to run across multiple nodes. This will provide you with more computing power and higher availability for your Apache Airflow instance. Airflow Daemons A running instance of Airflow has a number of Daemons that work together to provide the full functionality of Airflow. The daemons include the Web Server, Scheduler, Worker, Kerberos Ticket Renewer, Flower and others. Bellow are the primary ones you will need to have running for a production quality Apache Airflow Cluster. Web Server A daemon which accepts HTTP requests and allows you to interact with Airflow via a Python Flask Web Application. It provides the ability to pause, unpause DAGs, manually trigger DAGs, view running DAGs, restart failed DAGs and much more. The Web Server Daemon starts up…
Tag: Apache Airflow
Installing and Configuring Apache Airflow
Apache Airflow is a platform to programmatically author, schedule and monitor workflows – it supports integration with 3rd party platforms so that you, our developer and user community, can adapt it to your needs and stack. Additional Documentation: Documentation: https://airflow.incubator.apache.org/ Install Documentation: https://airflow.incubator.apache.org/installation.html GitHub Repo: https://github.com/apache/incubator-airflow Preparing the Environment Install all needed system dependencies Ubuntu SSH onto target machine (s) where you want to install Airflow Login as Root Install Required Libraries #Run upgrade apt-get update #Unzip apt-get install unzip #Build Essentials – GCC Compiler apt-get install build-essential #Python Development apt-get install python-dev #SASL apt-get install libsasl2-dev #Pandas apt-get install python-pandas Check Python Version Run the command: python -V If the version comes back as “Python 2.7.X” you can skip the rest of this step Install Python 2.7.X cd /opt sudo wget –no-check-certificate https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz tar xf Python-2.7.6.tar.xz cd Python-2.7.6 ./configure –prefix=/usr/local make && make altinstall ls -ltr /usr/local/bin/python*…