In order to layer on some easy data-at-rest security, I want to encrypt the boot volumes of my Amazon EC2 instances. I also want to use the centos.org CentOS images but those are not encrypted. How can I end up with an encrypted copy of those AMIs in the fewest steps? In the past, I have used shell scripts and the AWS CLI to perform the boot volume encryption dance. The steps are basically: Deploy an instance running the source AMI. Create an image from that instance. Copy the image and encrypt the copy. Delete the unencrypted image. Terminate the instance. Add tags to new AMI. The script has a need for a lot of VPC/subnet/security group preparation (which I guess could have been added to the script), and if there were errors during the execution then cleanup was very manual (more possible script work). The script is very flexible and meets…
Month: November 2016
Installing Livy on a Hadoop Cluster
Purpose Livy is an open source component to Apache Spark that allows you to submit REST calls to your Apache Spark Cluster. You can view the source code here: https://github.com/cloudera/livy In this post I will be going over the steps you would need to follow to get Livy installed on a Hadoop Cluster. The steps were derived from the above source code link, however, this post provides more information on how to test it in a more simple manner. Install Steps Determine which node in your cluster will act as the Livy server Note: the server will need to have Hadoop and Spark libraries and configurations deployed on them. Login to the machine as Root Download the Livy source code cd /opt wget https://github.com/cloudera/livy/archive/v0.2.0.zip unzip v0.2.0.zip cd livy-0.2.0 Get the version of spark that is currently installed on your cluster Run the following command spark-submit –version Example: 1.6.0 Use this value in downstream commands as…