See: Description
Pig runs on hadoop MapReduce, reading data from and writing data to HDFS, and doing processing via one or more MapReduce jobs.
Pig's design is guided by our pig philosophy.
Pig shares many similarities with a traditional RDBMS design. It has a parser, type checker, optimizer, and operators that perform the data processing. However, there are some significant differences. Pig does not have a data catalog, there are no transactions, pig does not directly manage data storage, nor does it implement the execution framework.
PhysicalPlan.
This Physical Plan contains the operators that will be applied to the data. This is then
divided into a set of MapReduce jobs by the
MRCompiler into an
MROperPlan. This
MROperPlan (aka the map reduce plan) is then optimized (for example, the combiner is used where
possible, jobs that scan the same input data are combined where possible, etc.). Finally a set of
MapReduce jobs are generated by the
JobControlCompiler. These are
submitted to Hadoop and monitored by the
MapReduceLauncher.
On the backend, each
PigGenericMapReduce.Map,
PigCombiner.Combine, and
PigGenericMapReduce.Reduce
use the pipeline of physical operators constructed in the front end to load, process, and store
data.
In addition to the command line and grunt interfaces, users can connect to
PigServer from a Java program.
Pig makes it easy for users to extend its functionality by implementing User Defined Functions
(UDFs). There are interfaces for defining functions to load data
LoadFunc, storing data StoreFunc, doing evaluations
on fields (including collections of data, so user defined aggregates are possible)
EvalFunc and filtering data FilterFunc.
Copyright © 2007-2017 The Apache Software Foundation