Configuring Hadoop Cluster using Ansible playbook

To configure the Hadoop Cluster using ansible is requried to configure the Ansible in the Controller Node

pip install ansible

The Anisible is written in python.By using pip can able to install the Ansible software by default the python is install in the Linux. The above command is work only when python in install in the system.

To check the version of the ansible can use the ansible — version command to verify the ansible is installed or not.

Configuring the inventory file listing the all Managed node information.

Configuring the Ansible configuration file by Mentioning the inverntory file location in the configuration file.The Anisble Configuring is located in the location /etc/ansible/ansible.cfg

Checking the connectivity between the Controlled Node and Managed Nodes by ping the Nodes

Playbook For Namenode Configuration:

- hosts: namenode
  tasks:
  - name: "Copying the hadoop Software File"
    copy:
     src: "hadoop-1.2.1-1.x86_64.rpm"
     dest: "/root/"
     
  - name: "Copying the Jdk Software File"
    copy:
     src: "jdk-8u171-linux-x64.rpm"
     dest: "/root/"
     
  - name: "Installing Jdk"
    shell: "rpm -ivh jdk-8u171-linux-x64.rpm"
    register: Java
    ignore_errors: yes
    
  - name: "Installing Hadoop"
    shell: "rpm -ivh hadoop-1.2.1-1.x86_64.rpm  --force"
    register: Hadoop
    ignore_errors: yes
 
  - name: "Copying the core-site.xml file"
    copy:
      src: "core-site.xml"
      dest: "/etc/hadoop/core-site.xml"
      
  - name: "Copying the hdfs-site.xml file"
    copy:
      src: "namenode-hdfs-site.xml"
      dest: "/etc/hadoop/hdfs-site.xml"
     
  - name: "Creating a directory"
    file:
      state: directory
      path: "nn"
      
  - name: "Formatting Namenode"
    shell: "echo Y |  hadoop namenode -format"
    register: format

  - name: "Starting the namenode"
    shell: "hadoop-daemon.sh start namenode"
    ignore_errors: yes
    register: hadoop_started
    
  - name: "checking status of namenode"
    shell: "jps"
    register: jps

In this playbook copied the files of Hadoop Software and Jdk software and installed the softwares configured the hdfs-site.xml and core-site.xml files by copied the files from the Controlled Node created a directory and formatted the namenode started the service of the namenode.

Playbook Datanode Configuration:

- hosts: datanode
  tasks:
  - name: "Copying the hadoop Software File"
    copy:
     src: "hadoop-1.2.1-1.x86_64.rpm"
     dest: "/root/"
     
  - name: "Copying the Jdk Software File"
    copy:
     src: "jdk-8u171-linux-x64.rpm"
     dest: "/root/"
     
  - name: "Installing Jdk"
    shell: "rpm -ivh jdk-8u171-linux-x64.rpm"
    register: Java
    ignore_errors: yes
    
  - name: "Installing Hadoop"
    shell: "rpm -ivh hadoop-1.2.1-1.x86_64.rpm  --force"
    register: Hadoop
    ignore_errors: yes
 
  - name: "Copying the core-site.xml file"
    copy:
      src: "core-site.xml"
      dest: "/etc/hadoop/core-site.xml"
      
  - name: "Copying the hdfs-site.xml file"
    copy:
      src: "datanode-hdfs-site.xml"
      dest: "/etc/hadoop/hdfs-site.xml"
     
  - name: "Creating a directory"
    file:
      state: directory
      path: "dn"

  - name: "Starting the datanode"
    shell: "hadoop-daemon.sh start datanode"
    ignore_errors: yes
    register: hadoop_started
    
  - name: "checking status of datanode"
    shell: "jps"
    register: jps

In this playbook copied the files of Hadoop Software and Jdk software and installed the softwares configured the hdfs-site.xml and core-site.xml files by copied the files from the Controlled Node created a directory and started the service of the datanode.

By running the playbook of the namenode in the controlled node as ansible-playbook namenode.yml

The playbook executed succcessfully without any errors namenode is configured with this playbook

Again running the playbook of the datanode with ansible-playbook datanode.yml

The playbook executed succcessfully without any errors datanode is configured with this playbook

…………………………….Thanks for reading…………………………………

Task completed under the mentorship of Mr.Vimal Daga sir..

Vimal Daga - Sr. Machine Learning / Deep Learning / Data Scientist / NLP Consultant and Researcher…

ARTH - The School of Technologies