Pig installation on Windows 10


Apache Pig


Pig Introduction


We have already walked through the Hadoop, HBae and Hive outline and installation over the Windows environment in previous posts. In the context of brief description, we can see the Hadoop can perform only batch processing, and data will be accessed only in a sequential manner, that means required to search the entire data-set even for the single task.

In a different scenario, to ease the process to access any point of data in a single unit of time (random access), the HBase introduced. HBase is an open source non-relational (NoSQL) distributed column-oriented database that runs on top of HDFS and real-time read/write access to those large data-sets that cannot be handled by the Hadoop. 

In this sequence, we talked about the Hive, an ETL tool for Hadoop ecosystem, enables developers to write Hive Query Language (HQL) statements very similar to SQL statements. It is a data warehouse software project built on top of Hadoop, that facilitate reading, writing, and managing large datasets residing in distributed storage using SQL.

Now time to talk about Pig, it was initially developed by Yahoo! for its data scientists who were using Hadoop. Pig is a platform for analyzing large sets of data that consists of a high-level language for expressing data analysis programs. It is a data flow language (PigLatin) to write Hadoop operations without using MapReduce Java code. 

Importance of Pig


Usage of Pig makes its reputation; few key usages are as follows - 

[1] Ease of programming
It is quiet tough for non-programming background aspirant to write and execute complex Java programs for map reduce. Pig makes this process easier, the queries are converted to map reduce internally.

[2] Optimization opportunities
The way in which tasks are encoded permits the system to optimize their execution automatically, allowing the user to focus on semantics rather than efficiency.

[3] Extensible
User defined function are written in which the user can write their own logic to execute over the data set.

Pig Installation


Pig is a layer of abstraction on top of Hadoop to simplify its use by giving a SQL –like interface to process data on Hadoop. Before moving ahead, it is essential to install Hadoop first, I am considering Hadoop is already installed, if not, then go to my previous post how to install Hadoop on Windows environment.

I went through Pig 0.17.0 version, though you can use any stable version.

Download Pig 0.17.0
  • https://pig.apache.org/

STEP - 1 : Extract the Pig file


Extract file pig-0.17.0.tar.gz and place under "D:\Pig", you can use any preferred location –
[1] You will get again a tar file post extraction – 


Pig initial file

[2] Go inside of pig-0.17.0.tar folder and extract again – 

Extract the file

[3] Copy the leaf folder “pig-0.17.0” and move to the root folder "D:\Pig" and removed all other files and folders – 

Local Pig Folder

Extracted Pig Files


STEP - 2: Configure Environment variable


Set the path for the following Environment variable (User Variables) on windows 10 – 
  • PIG_HOME - D:\Pig\pig-0.17.0
This PC - > Right Click - > Properties - > Advanced System Settings - > Advanced - > Environment Variables 

Environment variables

STEP - 3: Configure System variable



Next onward need to set System variable, including Hive bin directory path – 

Variable: Path 
Value: 
  • D:\Pig\pig-0.17.0\bin
System Variable

STEP - 4: Working with Pig command file


Now need to do a cross check with Pig command file for Hadoop executable details – 
Pig.cmd


[1] Edit file D:/Pig/pig-0.17.0/bin/pig.cmd, make below changes and save this file.
set HADOOP_BIN_PATH=%HADOOP_HOME%\libexec

Pig.cmd

STEP - 5: Start the Hadoop


Here need to start Hadoop first - 

Open command prompt and change directory to “D:\Hadoop\hadoop-2.8.0\sbin" and type "start-all.cmd" to start apache.

Start Hadoop


It will open four instances of cmd for following tasks – 
  • Hadoop Datanaode
  • Hadoop Namenode
  • Yarn Nodemanager
  • Yarn Resourcemanager
Hadoop Started

It can be verified via browser also as – 
  • Namenode (hdfs) - http://localhost:50070 
  • Datanode - http://localhost:50075
  • All Applications (cluster) - http://localhost:8088 etc.
Hadoop In Browser

Since the ‘start-all.cmd’ command has been deprecated so you can use below command in order wise - 
  • “start-dfs.cmd” and 
  • “start-yarn.cmd”

STEP - 6: Validate Pig installation


Post successful execution of Hadoop, change directory to “D:\Pig\pig-0.17.0\bin” and verify the installation.

Pig validation

Pig Version

STEP - 7: Execute Pig (Modes)


Pig has been installed and ready to execute so time to execute, you can run Apache Pig in two modes, namely -

[1] Local Mode 
In this mode, all the files are installed and run from your local host and local file system. There is no need of Hadoop or HDFS. This mode is generally used for testing purpose.


[2] HDFS mode
MapReduce mode is where we load or process the data that exists in the Hadoop File System (HDFS) using Apache Pig. In this mode, whenever we execute the Pig Latin statements to process the data, a MapReduce job is invoked in the back-end to perform a particular operation on the data that exists in the HDFS.


Apache Pig Execution Mechanisms


Apache Pig scripts can be executed in three ways, namely – 


[1] Interactive Mode (Grunt shell)
You can run Apache Pig in interactive mode using the Grunt shell. In this shell, you can enter the Pig Latin statements and get the output (using Dump operator).

[2] Batch Mode (Script)
You can run Apache Pig in Batch mode by writing the Pig Latin script in a single file with .pig extension.

[3] Embedded Mode (UDF)
Apache Pig provides the provision of defining our own functions (User Defined Functions) in programming languages such as Java, and using them in our script.


STEP - 8: Invoke Grunt shell



Now you can invoke the Grunt shell in a desired mode (local/MapReduce) using the −x option as shown below.

Local Mode
pig -x local

local mode

Local Mode executed

MapReduce
pig -x mapreduce

MapReduce Mode

MapReduce Mode executed

Either of these commands gives you the Grunt shell prompt as shown below.

Grunt Shell


Congratulations, Pig installed !!😊

STEP-9: Some hands on activities


[1] Create a text file ‘student.txt’ delimited by ‘,’

text file

[2] Load the file in a variable ‘input_file’
input_file = LOAD 'd:/student.text' USING PigStorage(',') 
as (Sr_No: Int, Student_Name:chararray, Student_Location:chararray);

Load query

Load text file


[3] Check result using DUMP operator (write result to the console)
DUMP input_file;


DUMP file

DUMP file result


[4] We can execute the pig script in batch mode also.


Step 1 - Write all the required Pig Latin statements in a single file. We can write all the Pig Latin statements and commands in a single file and save it as .pig file.

Pig script


Step 2 - You can execute it from the Grunt shell as well using the exec command as shown below.
exec /d:/student.pig


exec pig script

Executed the script

 Stay in touch for more posts.

1 comment:

  1. Short and to the point. The best part is you have given an example to check PIG is working as expected on Windows. Keep up the good work.

    ReplyDelete