Airflow

Upgrading Apache Airflow Versions

In a previous post we explained how to Install and Configure Apache Airflow (a platform to programmatically author, schedule and monitor workflows). The technology is actively being worked on and more and more features and bug fixes are being added to the project in the form of new releases. At some point, you will want to upgrade to take advantage of these new feature. In this post we’ll go over the process that you should for upgrading apache airflow versions. Note: You will need to separately make sure that your dags will be able to work on the new version of Airflow. Upgrade Airflow Note: These steps can also work to downgrade versions of Airflow Note: Execute all of this on all the instances in your Airflow Cluster (if you have more then one machine) Gather information about your current environment and your target setup: Get the Airflow Home directory. Placeholder for…

Big Data

Installing Apache Zeppelin on a Hadoop Cluster

Apache Zeppelin(https://zeppelin.incubator.apache.org/)  is a web-based notebook that enables interactive data analytics. You can make data-driven, interactive and collaborative documents with SQL, Scala and more. This document describes the steps you can take to install Apache Zeppelin on a CentOS 7 Machine. Steps Note: Run all the commands as Root Configure the Environment Install Maven (If not already done) cd /tmp/ wget https://archive.apache.org/dist/maven/maven-3/3.1.1/binaries/apache-maven-3.1.1-bin.tar.gz tar xzf apache-maven-3.1.1-bin.tar.gz -C /usr/local cd /usr/local ln -s apache-maven-3.1.1 maven Configure Maven (If not already done) #Run the following export M2_HOME=/usr/local/maven export M2=${M2_HOME}/bin export PATH=${M2}:${PATH} Note: If you were to login as a different user or logout these settings will be whipped out so you won’t be able to run any mvn commands. To prevent this, you can append these export statements to the end of your ~/.bashrc file: #append the export statements vi ~/.bashrc #apply the export statements source ~/.bashrc Install NodeJS Note: Steps referenced from https://nodejs.org/en/download/package-manager/ curl –silent…