In a previous post we explained how to Install and Configure Apache Airflow (a platform to programmatically author, schedule and monitor workflows). The technology is actively being worked on and more and more features and bug fixes are being added to the project in the form of new releases. At some point, you will want to upgrade to take advantage of these new feature. In this post we’ll go over the process that you should for upgrading apache airflow versions. Note: You will need to separately make sure that your dags will be able to work on the new version of Airflow. Upgrade Airflow Note: These steps can also work to downgrade versions of Airflow Note: Execute all of this on all the instances in your Airflow Cluster (if you have more then one machine) Gather information about your current environment and your target setup: Get the Airflow Home directory. Placeholder for…
Tag: distributed workflow management
Installing and Configuring Apache Airflow
Apache Airflow is a platform to programmatically author, schedule and monitor workflows – it supports integration with 3rd party platforms so that you, our developer and user community, can adapt it to your needs and stack. Additional Documentation: Documentation: https://airflow.incubator.apache.org/ Install Documentation: https://airflow.incubator.apache.org/installation.html GitHub Repo: https://github.com/apache/incubator-airflow Preparing the Environment Install all needed system dependencies Ubuntu SSH onto target machine (s) where you want to install Airflow Login as Root Install Required Libraries #Run upgrade apt-get update #Unzip apt-get install unzip #Build Essentials – GCC Compiler apt-get install build-essential #Python Development apt-get install python-dev #SASL apt-get install libsasl2-dev #Pandas apt-get install python-pandas Check Python Version Run the command: python -V If the version comes back as “Python 2.7.X” you can skip the rest of this step Install Python 2.7.X cd /opt sudo wget –no-check-certificate https://www.python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz tar xf Python-2.7.6.tar.xz cd Python-2.7.6 ./configure –prefix=/usr/local make && make altinstall ls -ltr /usr/local/bin/python*…