Ansible is an essential DevOps tool for managing backups and rolling upgrades Cassandra in AWS/EC2. An excellent aspect of Ansible is that it uses ssh, so you do not have to install an agent to use Ansible.

The first article in this series was about setting up a Cassandra cluster with Vagrant (also appeared on DZone with some additional content DZone Setting up a Cassandra Cluster with Vagrant. The second article in this series was about setting up SSL for a Cassandra cluster using Vagrant (which also appeared with more content as DZone Setting up a Cassandra Cluster with SSL). You don't need those articles to follow along, but they might provide a lot of contexts. You can find the source for the first and second article at our Cloudurable Cassandra Image for Docker, AWS, and Vagrant. In later articles, we will use Ansible to create more complicated playbooks like doing a rolling Cassandra upgrade.

We continue to evolve the cassandra-image GitHub project. In an effort for the code to match the listings in the article, we created a new branch where the code was when this article was written (more or less): Article 3 Ansible Cassandra Vagrant.

To use Ansible for DevOps, we will need to setup ssh keys as Ansible uses ssh instead of running an agent on each server like Chef and Puppet.

The tool ssh-keygen manages authentication keys for ssh(secure shell). The utility ssh-keygen generates RSA or DSA keys for SSH (secure shell) protocol version 1 and 2. You can specify the key type with the -t option.

CLUSTER_NAME=test
...
ssh-keygen -t rsa -C "your_email@example.com" -N "" -C "setup for cloud" \
 -f "$PWD/resources/server/certs/${CLUSTER_NAME}_rsa"

chmod 400 "$PWD/resources/server/certs/"*
cp "$PWD/resources/server/certs/"* ~/.ssh
...

Let's break that down.

We use ssh-keygen to create a private key that we will use to log into our boxes.

ssh-keygen -t rsa -C "your_email@example.com" -N "" -C "setup for cloud" \
 -f "$PWD/resources/server/certs/${CLUSTER_NAME}_rsa"

Then we restrict the access to the file of the key otherwise, ansible, ssh and scp (secure copy) will not let us use it.

chmod 400 "$PWD/resources/server/certs/"*

The above chmod 400 changes the cert files so only the owner can read the file. This file chage mod makes sense. The certification file should be private to the user.

cp "$PWD/resources/server/certs/"* ~/.ssh

The above just puts the files where our provisioners (Packer and Vagrant) can pick them up and deploy them with the image.

Locally we are using Vagrant to launch a cluster to do some tests on our laptop. We also use Packer and aws command line tools to create EC2 AMIs (and Docker images), but we don't cover aws in this article.

Eventually, we would like to use a bastion server that is a public subnet to send commands to our Cassandra nodes that are in a private subnet in EC2. For local testing we set up a bastion server, which is well explained in this guide to Vagrant and Ansible.

We used [Learning Ansible with Vagrant (Part 2/4)](Learning Ansible with Vagrant (Part 2/4)) as a guide for some of the setup performed in this article. It is a reliable source of Ansible and Vagrant knowledge for DevOps. Their mgmt node corresponds to what we call a bastion server. A notable difference is we are using CentOS 7 not Ubuntu, and we made some slight syntax updates to some of the Ansible commands that we are using (we use a later version of Ansible).

We added a bastion server to our Vagrant config as follows:

  # Define Bastion Node
  config.vm.define "bastion" do |node|
            node.vm.network "private_network", ip: "192.168.50.20"
            node.vm.provider "virtualbox" do |vb|
                   vb.memory = "256"
                   vb.cpus = 1
            end


            node.vm.provision "shell", inline: <<-SHELL
                yum install -y epel-release
                yum update -y
                yum install -y  ansible

                mkdir /home/vagrant/resources
                cp -r /vagrant/resources/* /home/vagrant/resources/

                mkdir -p ~/resources
                cp -r /vagrant/resources/* ~/resources/

                mkdir  -p  /home/vagrant/.ssh/
                cp /vagrant/resources/server/certs/*  /home/vagrant/.ssh/

                sudo  /vagrant/scripts/002-hosts.sh

                ssh-keyscan node0 node1 node2  >> /home/vagrant/.ssh/known_hosts


                mkdir ~/playbooks
                cp -r /vagrant/playbooks/* ~/playbooks/
                sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts
                chown -R vagrant:vagrant /home/vagrant
            SHELL

The bastion server which could be on a public subnet in AWS in a VPC uses the ssh-keyscan to add nodes that we setup in the host file into /home/vagrant/.ssh/known_hosts.

ssh-keyscan node0 node1 node2  >> /home/vagrant/.ssh/known_hosts

This utility is to get around the problem of needing to verify nodes, and getting this error message: The authenticity of host ... can't be established. ... Are you sure you want to continue connecting (yes/no)? no when we are trying to run ansible command line tools.

Since we are using provision files to create different types of images (Docker, EC2 AMI, Vagrant/VirtualBox), then we use a provisioning script specific to vagrant.

In this vagrant provision script, we call another provision script to setup a hosts file.

mkdir  -p  /home/vagrant/.ssh/
cp /vagrant/resources/server/certs/*  /home/vagrant/.ssh/
...

scripts/002-hosts.sh
echo RUNNING TUNE OS

The provision script 002-hosts.sh configures /etc/ssh/sshd_config/sshd_config to allow public key auth. Then it restarts the daemon for ssh communication sshd.

Let's look at the 002-hosts.sh provision script. You can see some remnants from the last article where we setup csqlsh, and then it gets to business setting up sshd (secure server shell daemon).

#!/bin/bash
set -e



## Copy csqlshrc file that controls csqlsh connections to ~/.cassandra/cqlshrc.
mkdir ~/.cassandra
cp ~/resources/home/.cassandra/cqlshrc ~/.cassandra/cqlshrc

## Allow pub key login to ssh.
sed -ie 's/#PubkeyAuthentication no/PubkeyAuthentication yes/g' /etc/ssh/sshd_config

## System control restart sshd daemon to take sshd_config into effect.
systemctl restart sshd

# Create host file so it is easier to ssh from box to box
cat >> /etc/hosts <<EOL

192.168.50.20  bastion

192.168.50.4  node0
192.168.50.5  node1
192.168.50.6  node2
192.168.50.7  node3
192.168.50.8  node4
192.168.50.9  node5
EOL

This setup is fairly specific to our Vagrant setup at this point. To simplify access to the servers that hold the different Cassandra nodes, the 002-hosts.sh creates an \etc\hosts\ file on the bastion server.

With our certification keys added to sshd config and our hosts configured (and our inventory.ini file shipped), we can start using ansible from our bastion server.

This reminds me, we did not talk about the inventory.ini file.

Ansible has a ansible.cfg file, and an inventory.ini file. When you run ansible, it checks for ansible.cfg in the current working directory, then your home directory, and then for a master config file (/etc/ansible). We created an inventory.ini file which lives under ~\github\cassandra-image\resources\home, which gets mapped to \vagrant\resources\home on the virtual machines (node0, bastion, node1, and node2). A provision script moves the inventory.ini file to its proper location (sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts).

The inventory.ini contains servers that you want to manage with Ansible. A couple of things are going on here, we have a bastion group, this is for our bastion server, next we have the nodes group, and it is made up of node0, node1, and node2.

Let's see what the inventory.ini file actually looks like.

[bastion]
bastion


[nodes]
node0
node1
node2

Once we provision our cluster, we can log into bastion and start executing ansible commands.

To make this happen, we had to tell the other servers about our certification keys. We did this with an ansible playbook as follows:

Vagrant.configure("2") do |config|


  config.vm.box = "centos/7"


  # Define Cassandra Nodes
  (0..numCassandraNodes-1).each do |i|

        port_number = i + 4
        ip_address = "192.168.50.#{port_number}"
        seed_addresses = "192.168.50.4,192.168.50.5,192.168.50.6"
        config.vm.define "node#{i}" do |node|
            node.vm.network "private_network", ip: ip_address
            node.vm.provider "virtualbox" do |vb|
                   vb.memory = "2048"
                   vb.cpus = 4
            end
            ...

            node.vm.provision "ansible" do |ansible|
                  ansible.playbook = "playbooks/ssh-addkey.yml"
            end
        end
  end

Notice the line node.vm.provision "ansible" do |ansible| and ansible.playbook = "playbooks/ssh-addkey.yml".

If you are new to Vagrant and the above just is not making sense, please watch Vagrant Crash Course. It is by the same folks (guy) who created the Ansible series.

Ansible playbooks are like configuration playbooks. You can perform tons of operations that are important for DevOps (like yum installing software, specific tasks to Cassandra, etc.).

Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process. --Ansible Playbook documentation.

Here is the ansible playbook

---
- hosts: all
  become: true
  gather_facts: no
  remote_user: vagrant

  tasks:

  - name: install ssh key
    authorized_key: user=vagrant
                    key="{{ lookup('file', '../resources/server/certs/test_rsa.pub') }}"
                    state=present

The trick here is that Vagrant supports running Ansible playbooks as well.

The Vagrant Ansible provisioner allows you to provision the guest using Ansible playbooks by executing ansible-playbook from the Vagrant host. --(Vagrant Ansible documentation)[https://www.vagrantup.com/docs/provisioning/ansible.html]

If you have not done so already navigate to the project root dir (which is ~/github/cassandra-image on my dev box), download the binaries. The source code is at https://github.com/cloudurable/cassandra-image.

## cd ~; mkdir github; cd github; git clone https://github.com/cloudurable/cassandra-image
$ cd ~/github/cassandra-image
$ pwd
~/github/cassandra-image
## Setup keys
$ bin/setupkeys-cassandra-security.sh
## Download binaries
$ bin/prepare_binaries.sh
## Bring Vagrant cluster up
$ vagrant up

Even if you read the first article note that bin/prepare_binaries.sh is something we added after the first two articles. It downloads the binaries needed for the provisioning, does a checksum of the files and then installs them as part of the provisioning process.

Let's login into bastion and run ansible commands against the cassandra nodes.

$ vagrant ssh bastion

So we don't have to keep logging in, and passing our cert key, let's start up an ssh-agent and add our cert key ssh-add ~/.ssh/test_rsa to the agent.

The ssh-agent is a utility to hold private keys used for public key authentication (RSA, DSA, ECDSA, Ed25519) so you don't have to keep passing the keys around. The ssh-agent is usually started in the beginning of a login session. Other programs (scp, ssh, ansible) are started as clients to the ssh-agent utility.

First setup ssh-agent and add keys to it.

$ ssh-agent bash
$ ssh-add ~/.ssh/test_rsa

Now that the agent is running and our keys are added, we can use ansible without passing it the certification key.

Let's verify connectivity, by pinging some of these machines. Let's ping the node0. Then let's ping all of the nodes.

Let's use the ansible ping command to ping the node0 server.

$ ansible node0 -m ping

Output

node0 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Now let's ping all of the nodes.

$ ansible nodes  -m ping

Output

node0 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
node2 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
node1 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Looks like bastion can run ansible against all of the servers.

The script ~/github/cassandra-image/bin/setupkeys-cassandra-security.sh copies the test cluster key for ssh (secure shell) over to ~/.ssh/ (cp "$PWD/resources/server/certs/"* ~/.ssh). It was Run from the project root folder which is ~/github/cassandra-image on my box.

Move to the where you checked out the project.

cd ~/github/cassandra-image

In this folder is an ansible.cfg file and an inventory.ini file for local dev. Before you use these first modify your /etc/hosts file to configure entries for bastion, node0, node1, node2 servers.

$ cat /etc/hosts

### Used for ansible/ vagrant
192.168.50.20  bastion
192.168.50.4  node0
192.168.50.5  node1
192.168.50.6  node2
192.168.50.7  node3
192.168.50.8  node4
192.168.50.9  node5

We can use ssh-keyscan just like we did before to add these hosts to our known_hosts file.

$ ssh-keyscan node0 node1 node2  >> ~/.ssh/known_hosts

Then just like before we can start up an ssh-agent and add our keys.

$ ssh-agent bash
$ ssh-add ~/.ssh/test_rsa

Notice that the ansible.cfg and inventory.ini files are a bit different than on our bastion server because we have to add the user name.

$ cd ~/github/cassandra-image

$ cat ansible.cfg 
[defaults]
hostfile = inventory.ini

cat inventory.ini 
[nodes]
node0 ansible_user=vagrant
node1 ansible_user=vagrant
node2 ansible_user=vagrant

Ansible will use these.

From the project directory, you should be able to ping node0 and all of the nodes just like before.

Ping node0 with ansible.

$ ansible node0 -m ping

Output

node0 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Ping all of the Cassandra nodes with ansible.

$ ansible nodes  -m ping

Output

node0 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
node2 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
node1 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

You may recall from the first article that we would log into the servers (vagrant ssh node0) and then check that they could see the other nodes with nodetool describecluster. We could run this command with all three servers (from bastion or on our dev laptop) with ansible.

Let's use ansible to run describecluster against all of the nodes.

ansible nodes -a "/opt/cassandra/bin/nodetool describecluster"

This command allows us to check the status of every node quickly.

Let's say that we wanted to update a schema or do a rolling restart of our Cassandra nodes, which could be a very common task. Perhaps before the update, we want to decommission the node and back things up. To do this sort of automation, we could create an Ansible playbook.

Ansible Playbooks are more powerful than executing ad-hoc task execution and is especially powerful for managing a cluster of Cassandra servers.

Playbooks allow for configuration management and multi-machine deployment to manage complex tasks like a rolling upgrade or schema updates or perhaps a weekly backup.

Playbooks are declarative configurations. Ansible Playbooks also orchestrate steps into a simpler task. This automation gets rid of a lot of manually ordered process and allows for an immutable infrastructure.

Creating a complicated playbook is beyond the scope of this article. But let's create a simple playbook and execute it. This playbook will run the nodetool describecluster on each node.

---
- hosts: nodes
  gather_facts: no
  remote_user: vagrant

  tasks:

  - name: Run NodeTool Describe Cluster command
    command: /opt/cassandra/bin/nodetool describecluster

To run this, we use ansible-playbook as follow.

ansible-playbook playbooks/describe-cluster.yml --verbose

# -*- mode: ruby -*-
# vi: set ft=ruby :

numCassandraNodes = 3

Vagrant.configure("2") do |config|


  config.vm.box = "centos/7"


  # Define Cassandra Nodes
  (0..numCassandraNodes-1).each do |i|

        port_number = i + 4
        ip_address = "192.168.50.#{port_number}"
        seed_addresses = "192.168.50.4,192.168.50.5,192.168.50.6"
        config.vm.define "node#{i}" do |node|
            node.vm.network "private_network", ip: ip_address
            node.vm.provider "virtualbox" do |vb|
                   vb.memory = "2048"
                   vb.cpus = 4
            end


            node.vm.provision "shell", inline: <<-SHELL

                sudo /vagrant/scripts/000-vagrant-provision.sh



                sudo /opt/cloudurable/bin/cassandra-cloud -cluster-name test \
                -client-address     #{ip_address} \
                -cluster-address    #{ip_address} \
                -cluster-seeds      #{seed_addresses}

            SHELL

            node.vm.provision "ansible" do |ansible|
                  ansible.playbook = "playbooks/ssh-addkey.yml"
            end
        end
  end


  # Define Bastion Node
  config.vm.define "bastion" do |node|
            node.vm.network "private_network", ip: "192.168.50.20"
            node.vm.provider "virtualbox" do |vb|
                   vb.memory = "256"
                   vb.cpus = 1
            end


            node.vm.provision "shell", inline: <<-SHELL
                yum install -y epel-release
                yum update -y
                yum install -y  ansible

                mkdir /home/vagrant/resources
                cp -r /vagrant/resources/* /home/vagrant/resources/

                mkdir -p ~/resources
                cp -r /vagrant/resources/* ~/resources/

                mkdir  -p  /home/vagrant/.ssh/
                cp /vagrant/resources/server/certs/*  /home/vagrant/.ssh/

                sudo  /vagrant/scripts/002-hosts.sh

                ssh-keyscan node0 node1 node2  >> /home/vagrant/.ssh/known_hosts


                mkdir ~/playbooks
                cp -r /vagrant/playbooks/* ~/playbooks/
                sudo cp /vagrant/resources/home/inventory.ini /etc/ansible/hosts
                chown -R vagrant:vagrant /home/vagrant
            SHELL


  end



  #
  # View the documentation for the provider you are using for more
  # information on available options.

  # Define a Vagrant Push strategy for pushing to Atlas. Other push strategies
  # such as FTP and Heroku are also available. See the documentation at
  # https://docs.vagrantup.com/v2/push/atlas.html for more information.
  config.push.define "atlas" do |push|
     push.app = "cloudurable/cassandra"
  end


end

We created an ssh key and then set up our instances with this key so we could use ssh, scp, and ansible. We set up a bastion server with Vagrant. We used ansible playbook (ssh-addkey.yml) from Vagrant to install our test cluster key on each server. We ran ansible ping against a single server. We ran ansible ping against many servers (nodes). We set up our local dev machine with ansible.cfg and inventory.ini so we could use ansible commands direct to node0 and nodes. We ran nodetool describecluster against all of the nodes from our dev machine. Lastly, we created a very simple playbook that can run nodetool describecluster. Ansible is a very powerful tool that can help you manage a cluster of Cassandra instances. In later articles, we will use Ansible to create more complex playbooks like backing up Cassandra nodes to S3.

• Cool lists of things you can do with Cassandra and Ansible
• Learning Ansible with Vagrant Training Video
• Source code for this article
• The first article in this series was about setting up a Cassandra cluster with Vagrant
• First article on DZone with some additional content DZone DevOps Setting up a Cassandra Cluster with Vagrant
• The second article in this series: setting up SSL for a Cassandra cluster using Vagrant
• Second article on DZone with additional content: DZone DevOps Setting up a Cassandra Cluster with SSL
• Cloudurable Cassandra Image for Docker, AWS and Vagrant