~ Tutorial 6 ~

Feed


  • Single-node ZooKeeper Setup
  • Single-node Kafka Setup
  • Single-node Storm Setup
  • Install Maven to Compile Storm Program

Single-node Zookeeper Setup

Prerequisites:

1). Sun Java 7:

Hadoop 2.7.1 requires a working Java 1.7+ (aka Java 7) installation. We will install Java 7 in this tutorial.

ubuntu@master:~$ sudo apt-get update
ubuntu@master:~$ sudo apt-get install openjdk-7-jre
ubuntu@master:~$ sudo apt-get install openjdk-7-jdk

Notes:Other methods to install java on Linux.link

Modify JAVA_HOME:

ubuntu@master:~$ vim ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
ubuntu@master:~$ source  ~/.bashrc

After installation, make a quick check whether JDK is correctly set up:

ubuntu@master:~$ java -version
java version "1.7.0_111"
OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1)
OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)

2). Create an User Account for Hadoop:

We do not want to run Hadoop as the root. So we will create a new user/group for hadoop related jobs. 

ubuntu@master:~$ sudo useradd -m hadoop -s /bin/bash
ubuntu@master:~$ sudo passwd hadoop
ubuntu@master:~$ sudo adduser hadoop sudo

3). Install SSH:

SSH (“Secure SHell”) is a protocol for securely accessing one machine from another. Hadoop uses SSH for accessing another slaves nodes to start and manage all HDFS and MapReduce daemons. SSH server should has been already installed. Otherwise, install SSH server by:

ubuntu@master:~$ sudo apt-get install openssh-server
ubuntu@master:~$ cd ~/.ssh/
ubuntu@master:~$ ssh-keygen -t rsa
ubuntu@master:~$ cat ./id_rsa.pub >> ./authorized_keys

After installation, make a quick check whether ssh is correctly set up:

ubuntu@master:~$ ssh localhost

Install ZooKeeper

1). Download and extract the ZooKeeper package

ubuntu@master:~$ wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz
ubuntu@master:~$ sudo tar -zxf zookeeper-3.4.6.tar.gz -C /usr/local
ubuntu@master:~$ cd /usr/local
ubuntu@master:~$ sudo mv zookeeper-3.4.6/ zookeeper/
ubuntu@master ~$ sudo chown -R hadoop ./zookeeper

2). Check and Start ZooKeeper Service

We will modify the configure file of the ZooKeeper and start the ZooKeeper Service.

ubuntu@master ~$ sudo su hadoop
hadoop@master:~$ cd /usr/local/zookeeper
hadoop@master:~$ mkdir tmp
hadoop@master:~$ vim ./conf/zoo.cfg 
                     dataDir=/usr/local/zookeeper/tmp
hadoop@master:~$ ./bin.zkServer.sh start

Single-node Kafka Setup

Prerequisites: - Single-node ZooKeeper Setup

Install Kafka

1). Download and extract the Kafka package

ubuntu@master:~$ wget https://archive.apache.org/dist/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
ubuntu@master:~$ sudo tar -zxf kafka_2.11-0.10.1.0.tgz -C /usr/local
ubuntu@master:~$ cd /usr/local
ubuntu@master:~$ sudo mv kafka_2.11-0.10.1.0/ kafka/
ubuntu@master ~$ sudo chown -R hadoop ./kafka

2). Check and Start Kafka Brokers We will start the Kafka Brokers and run kafka-console example.

ubuntu@master ~$ sudo su hadoop
hadoop@master:~$ cd /usr/local/kafka
hadoop@master:~$ bin/kafka-server-start.sh config/server.properties
hadoop@master:~$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partition 1 –topic dblad
hadoop@master:~$ bin/kafka-topics.sh --list --zookeeper localhost:2181 
hadoop@master:~$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic dblab
hadoop@master:~$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic dblab --from-beginning

Single-node Storm Setup

Prerequisites: - Single-node ZooKeeper Setup

Install Storm

1). Download and extract the Storm package

ubuntu@master:~$ https://archive.apache.org/dist/storm/apache-storm-1.0.1/apache-storm-1.0.1.tar.gz
ubuntu@master:~$ sudo tar -zxf apache-storm-1.0.1.tar.gz -C /usr/local
ubuntu@master:~$ cd /usr/local
ubuntu@master:~$ sudo mv apache-storm-1.0.1 storm
ubuntu@master ~$ sudo chown -R hadoop ./storm

2). Check and Start Storm Service We will start the storm nimbus and run StatefulTopology example.

ubuntu@master ~$ sudo su hadoop
hadoop@master:~$ cd /usr/local/storm
hadoop@master:~$ ./bin/storm nimbus
hadoop@master:~$ ./bin/storm supervisor
hadoop@master:~$ /usr/local/storm/bin/storm jar /usr/local/storm/examples/storm-starter/storm-starter-topologies-1.0.1.jar storm.starter.StatefulTopology

Install Maven to Compile Storm Program

Prerequisites: - Single-node Storm Setup

Install Maven

1). Download and extract the Maven package

ubuntu@master:~$ sudo tar -zxf apache-maven-3.3.9-bin.tar.gz -C /usr/local/
ubuntu@master:~$ cd /usr/local
ubuntu@master:~$ sudo mv ./apache-maven-3.3.9 maven
ubuntu@master:~$ sudo chown -R hadoop ./maven
ubuntu@master ~$ vim ~/.bashrc
                     export PATH=/usr/local/maven/bin:$PATH
ubuntu@master ~$ source ~/.bashrc

2). Check and Compile Storm Java program We will use mvn to compile Storm Java program

ubuntu@master ~$ sudo su hadoop
hadoop@master:~$ mvn -v
hadoop@master:~$ cd /usr/local/storm/examples/storm-starter
hadoop@master:~$ mvn compile exec:java -Dstorm.topology=storm.starter.StatefulTopology
hadoop@master:~$ mvn package
hadoop@master:~$ /usr/local/storm/bin/storm jar ./target/storm-starter-1.0.1.jar storm.starter.StatefulTopology