Big Data

Installing RabbitMQ

RabbitMQ is a queueing service that implements the Advanced Message Queuing Protocol (AMQP).  It is a fast and dependable open-source message server that supports a wide range of use cases including reliable integration, content-based routing and global data delivery, and high volume monitoring and data ingestion.

Additional Documentation:

General Install: https://www.rabbitmq.com/download.html

Setting up RabbitMQ Clustering: https://www.rabbitmq.com/clustering.html

Install RabbitMQ

Install RabbitMQ on Ubuntu

  1. Login as root
  2. Install RabbitMQ Server
    apt-get install rabbitmq-server
  3. Verify status
    rabbitmqctl status
  4. Install RabbitMQ Web Interface
    rabbitmq-plugins enable rabbitmq_management

Install RabbitMQ on CentOS

  1. Login as root
  2. Install RabbitMQ Server
    yum install epel-release
    yum install rabbitmq-server
  3. Verify status
    rabbitmqctl status
  4. Install RabbitMQ Web Interface
    rabbitmq-plugins enable rabbitmq_management

Setting up and Running RabbitMQ Cluster as a Cluster

If you want to setup multiple machines to work as a RabbitMQ Cluster you can follow these instructions. Otherwise you can follow “Running RabbitMQ as Single Node” instructions to get it running on a single machine.

Note: The machines you want to use in the cluster need to be able to communicate with each other.

  1. Follow the above “Install RabbitMQ” steps for on each node you want to add to the RabbitMQ Cluster
  2. Ensure the RabbitMQ daemons are not running
    1. See the “Running RabbitMQ as Single Node” section bellow
  3. Choose one of the nodes as MASTER
  4. On the non-MASTER nodes backup the .erlang.cookie file
    mv /var/lib/rabbitmq/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie.backup
  5. Copy the file “/var/lib/rabbitmq/.erlang.cookie” from the MASTER node to the other nodes and store it at the same location.
    • Be careful during this step. If you copy the contents of the .erlang.cookie file and use the vi or nano editor to update the non-MASTER nodes .erlang.cookie file you may add a next line character to the file. You’re better off using an FTP service to copy the .erlang.cookie file down from the MASTER node and copy it onto the non-MASTER machines.
  6. Set permissions of the .erlang.cookie file
    chown rabbitmq:rabbitmq /var/lib/rabbitmq/.erlang.cookie
    chmod 600 /var/lib/rabbitmq/.erlang.cookie
  7. Startup the MASTER RabbitMQ Daemon in detached mode (as root)
    rabbitmq-server -detached
  8. On each of the of the non-MASTER nodes, add them to the cluster and start them up one at a time (as root)
    #Stop App
    rabbitmqctl stop_app
     
    #Add the current machine to the cluster
    rabbitmqctl join_cluster rabbit@{MASTER_HOSTNAME}
     
    #Startup
    rabbitmqctl start_app
     
    #Check Status
    rabbitmqctl cluster_status
    • The “Check Status” command should return something like the following:
      1. Cluster status of node rabbit@{NODE_HOSTNAME} ...
        [{nodes,[{disc,[rabbit@{MASTER_HOSTNAME},rabbit@{NODE_HOSTNAME}]}]},
         {running_nodes,[rabbit@{MASTER_HOSTNAME},rabbit@{NODE_HOSTNAME}]}]
  9. Setup HA/Replication between Nodes
    1. Access the Management URL of one of the nodes (See “Managing the RabbitMQ Instance(s)” section bellow)
    2. Click on the Admin tab on top
    3. Click on the Policies tab on the right
    4. Add an HA policy
      1. Name: ha-all
      2. Pattern:
        1. leave blank
      3. Definitions:
        1. ha-mode: all
        2. ha-sync-mode: automatic
      4. Priority: 0
    5. Verify its setup correctly by navigating to the Queues section and clicking on one of the queues. You should see an entry in the Node and Slave section.
  10. Setup a load balancer to balance requests between the the Nodes
    • Port Forwarding
      1. Port 5672 (TCP) → Port 5672 (TCP)
      2. Port 15672 (HTTP) → Port 15672 (HTTP)
    • Health Check
      1. Protocol: HTTP
      2. Ping Port: 15672
      3. Ping Path: /
  11. Point all processes to that LB

Controlling RabbitMQ

Start RabbitMQ

service rabbitmq-server start

Stop RabbitMQ

service rabbitmq-server stop

Restart RabbitMQ

service rabbitmq-server restart

Getting RabbitMQ Status

service rabbitmq-server status

Setting RabbitMQ to Run on Machine Startup

chkconfig rabbitmq-server on

Managing the RabbitMQ Instance(s)

When the RabbitMQ daemons are started you can then visit the management Web UI in a Web Browser to monitor the RabbitMQ instance(s):

http://{ANY_RABBITMQ_NODE_HOSTNAME}:15672/

Default credentials: guest/guest

Note: Change the password

2 comments

  1. Pingback: Installing and Configuring Apache Airflow – Home

  2. Pingback: Setting up an Apache Airflow Cluster - Home

Leave a Reply

Your email address will not be published. Required fields are marked *