Hadoop installation on Windows 10

Haddop - Introduction

Here we will focus how to install Hadoop on Windows 10 environment rather than its detail framework, but will cover brief definition.

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage.

Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.

I went through Hadoop 2.8.0 version, though you can use any stable version.

Download Hadoop 2.8.0

Download Java JDK 1.8.0

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Check either Java 1.8.0 is already installed on your system or not, using command prompt –

Java –version

If Java is not installed on your system, then first installs java under "D:\JAVA" or your preferred drive –

STEP - 1: Extract the Hadoop file

Extract file Hadoop 2.8.0.tar.gz or Hadoop-2.8.0.zip and place under "D:\Hadoop", you can use any preferred location –

[1] You will get again a tar file post extraction –

[2] Go inside of Hadoop-2.8.0.tar folder and extract again –

[3] Copy the leaf folder “hadoop-2.8.0” and move to the root folder "D:\Hadoop" and removed all other files and folders –

STEP - 2: Configure Environment variables

Set the path HADOOP_HOME and JAVA_HOME Environment variables (User Variables) on windows 10 –

This PC - > Right Click - > Properties - > Advanced System Settings - > Advanced - > Environment Variables

STEP - 3: Configure System variables

Next onward need to set some particulars System variables, including Hadoop bin directory and JAVA bin directory path –

Variable: Path

Value:

D:\Hadoop\hadoop-2.8.0\bin
D:\Hadoop\hadoop-2.8.0\sbin
D:\Hadoop\hadoop-2.8.0\share\hadoop\common\*
D:\Hadoop\hadoop-2.8.0\share\hadoop\hdfs
D:\Hadoop\hadoop-2.8.0\share\hadoop\hdfs\lib\*
D:\Hadoop\hadoop-2.8.0\share\hadoop\hdfs\*
D:\Hadoop\hadoop-2.8.0\share\hadoop\yarn\lib\*
D:\Hadoop\hadoop-2.8.0\share\hadoop\yarn\*
D:\Hadoop\hadoop-2.8.0\share\hadoop\mapreduce\lib\*
D:\Hadoop\hadoop-2.8.0\share\hadoop\mapreduce\*
D:\Hadoop\hadoop-2.8.0\share\hadoop\common\lib\*
D:\Java\jdk1.8.0_171\bin

STEP - 4: Create required folders

Create some dedicated folders -
Create folder "data" under "D:\Hadoop\hadoop-2.8.0".
Create folder "datanode" under “D:\Hadoop\hadoop-2.8.0\data".
Create folder "namenode" under “D:\Hadoop\hadoop-2.8.0\data”
Create a folder to store temporary data during execution of a project, such as “D:\Hadoop\hadoop-2.8.0\temp.”
Create a log folder, such as “D:\Hadoop\hadoop-2.8.0\userlog”

For example -

STEP - 5: Configure required XML files

Now need to configure four key files with minimal required details –

core-site.xml
hdfs-site.xml
mapred.xml
yarn.xml

[1] Edit file D:/Hadoop/hadoop-2.8.0/etc/hadoop/core-site.xml, paste below xml paragraph and save this file.

<name>fs.defaultFS</name>

<value>hdfs://localhost:9000</value>

</property>

</configuration>

[2] Rename "mapred-site.xml.template" to "mapred-site.xml" and edit this file D:/Hadoop/hadoop-2.8.0/etc/hadoop/mapred-site.xml, paste below xml paragraph and save this file.

<name>mapreduce.framework.name</name>

</property>

</configuration>

[3] Edit file D:/Hadoop/hadoop-2.8.0/etc/hadoop/hdfs-site.xml, paste below xml paragraph and save this file.

<name>dfs.replication</name>

</property>

<name>dfs.namenode.name.dir</name>

<value>/D:/Hadoop/hadoop-2.8.0/data/namenode</value>

</property>

<name>dfs.datanode.data.dir</name>

<value>/D:/Hadoop/hadoop-2.8.0/data/datanode</value>

</property>

</configuration>

[4] Edit file D:/Hadoop/hadoop-2.8.0/etc/hadoop/yarn-site.xml, paste below xml paragraph and save this file.

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<name>yarn.nodemanager.log-dirs</name>

<value>/D:/Hadoop/hadoop-2.8.0/userlog</value><final>true</final>

</property>

<property><name>yarn.nodemanager.local-dirs</name>

<value>/D:/Hadoop/hadoop-2.8.0/temp/nm-localdir</value>

</property>

</configuration>

[5] Edit file D:/Hadoop/hadoop-2.8.0/etc/hadoop/hadoop-env.cmd by closing the command line"JAVA_HOME=%JAVA_HOME%"

STEP - 6: Manage Hadoop configuration

Time to manage Hadoop configuration, download file Hadoop Configuration.zip –

https://mindtreeonline-my.sharepoint.com/:u:/g/personal/m1045767_mindtree_com1/EaT3z5mOMyFAkdba7ywoPqoBCpZ920qGbajvMTXQvYGYkA?e=iOiP6y
Delete file bin on D:\Hadoop\hadoop-2.8.0\bin, replaced by file bin on file just download (from Hadoop Configuration.zip).
Do check the Hadoop version details; go to command prompt and type –D:\> hadoop version