org.apache.pig.backend.hadoop.executionengine

Class HExecutionEngine

java.lang.Object

org.apache.pig.backend.hadoop.executionengine.HExecutionEngine

All Implemented Interfaces:

ExecutionEngine

Direct Known Subclasses:

MRExecutionEngine, SparkExecutionEngine, TezExecutionEngine

public abstract class HExecutionEngine
extends java.lang.Object
implements ExecutionEngine

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	CORE_DEFAULT_SITE
static java.lang.String	CORE_SITE
protected DataStorage	ds
static java.lang.String	HADOOP_SITE
protected Launcher	launcher
static java.lang.String	LOCAL
protected java.util.Map<OperatorKey,OperatorKey>	logicalToPhysicalKeys
static java.lang.String	MAPRED_DEFAULT_SITE
protected java.util.Map<Operator,PhysicalOperator>	newLogToPhyMap
protected PigContext	pigContext
static java.lang.String	YARN_DEFAULT_SITE
static java.lang.String	YARN_SITE

Constructor Summary

Constructors

Constructor and Description
HExecutionEngine(PigContext pigContext)

Method Summary

Modifier and Type	Method and Description
PhysicalPlan	compile(LogicalPlan plan, java.util.Properties properties)
void	destroy() Perform any cleanup operation
void	explain(LogicalPlan lp, PigContext pc, java.io.PrintStream ps, java.lang.String format, boolean verbose, java.io.File file, java.lang.String suffix) This method handles the backend processing of the Explain command.
java.util.Properties	getConfiguration() Returns the Properties representation of the ExecutionEngine configuration.
DataStorage	getDataStorage() Returns the DataStorage the ExecutionEngine is using.
org.apache.hadoop.mapred.JobConf	getExecConf(java.util.Properties properties)
ExecutableManager	getExecutableManager() Returns the ExecutableManager to be used in Pig Streaming.
java.util.Map<LOForEach,java.util.Map<LogicalRelationalOperator,PhysicalOperator>>	getForEachInnerLogToPhyMap(LogicalPlan plan)
org.apache.hadoop.mapred.JobConf	getJobConf() Deprecated.
org.apache.hadoop.mapred.JobConf	getLocalConf()
LogicalPlan	getLogicalPlan()
java.util.Map<Operator,PhysicalOperator>	getLogToPhyMap()
org.apache.hadoop.mapred.JobConf	getS3Conf()
void	init() This method is responsible for the initialization of the ExecutionEngine.
void	kill() This method is called when user requests to kill all jobs associated with the execution engine
void	killJob(java.lang.String jobID) This method is called when a user requests to kill a job associated with the given job id.
PigStats	launchPig(LogicalPlan lp, java.lang.String grpName, PigContext pc) This method is responsible for the actual execution of a LogicalPlan.
void	setConfiguration(java.util.Properties newConfiguration) Responsible for updating the properties for the ExecutionEngine.
void	setProperty(java.lang.String property, java.lang.String value) Responsible for setting a specific property and value.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.pig.backend.executionengine.ExecutionEngine

instantiatePigStats, instantiateScriptState

Field Detail

HADOOP_SITE

public static final java.lang.String HADOOP_SITE

See Also:

Constant Field Values

CORE_SITE

public static final java.lang.String CORE_SITE

See Also:

Constant Field Values

YARN_SITE

public static final java.lang.String YARN_SITE

See Also:

Constant Field Values

CORE_DEFAULT_SITE

public static final java.lang.String CORE_DEFAULT_SITE

See Also:

Constant Field Values

MAPRED_DEFAULT_SITE

public static final java.lang.String MAPRED_DEFAULT_SITE

See Also:

Constant Field Values

YARN_DEFAULT_SITE

public static final java.lang.String YARN_DEFAULT_SITE

See Also:

Constant Field Values

LOCAL

public static final java.lang.String LOCAL

See Also:

Constant Field Values

pigContext

protected PigContext pigContext

ds

protected DataStorage ds

launcher

protected Launcher launcher

logicalToPhysicalKeys

protected java.util.Map<OperatorKey,OperatorKey> logicalToPhysicalKeys

newLogToPhyMap

protected java.util.Map<Operator,PhysicalOperator> newLogToPhyMap

Constructor Detail

HExecutionEngine

public HExecutionEngine(PigContext pigContext)

Method Detail

getJobConf

@Deprecated
public org.apache.hadoop.mapred.JobConf getJobConf()

Deprecated.

getDataStorage

public DataStorage getDataStorage()

Description copied from interface: ExecutionEngine

Returns the DataStorage the ExecutionEngine is using.

Specified by:

getDataStorage in interface ExecutionEngine

Returns:

DataStorage the ExecutionEngine is using.

init

public void init()
          throws ExecException

Description copied from interface: ExecutionEngine

This method is responsible for the initialization of the ExecutionEngine. All necessary setup tasks and configuration should execute in the init() method. This method will be called via the PigContext object.

Specified by:

init in interface ExecutionEngine

Throws:

ExecException

getS3Conf

public org.apache.hadoop.mapred.JobConf getS3Conf()
                                           throws ExecException

Throws:

ExecException

getLocalConf

public org.apache.hadoop.mapred.JobConf getLocalConf()

getExecConf

public org.apache.hadoop.mapred.JobConf getExecConf(java.util.Properties properties)
                                             throws ExecException

Throws:

ExecException

compile

public PhysicalPlan compile(LogicalPlan plan,
                            java.util.Properties properties)
                     throws FrontendException

Throws:

FrontendException

getLogToPhyMap

public java.util.Map<Operator,PhysicalOperator> getLogToPhyMap()

getForEachInnerLogToPhyMap

public java.util.Map<LOForEach,java.util.Map<LogicalRelationalOperator,PhysicalOperator>> getForEachInnerLogToPhyMap(LogicalPlan plan)

launchPig

public PigStats launchPig(LogicalPlan lp,
                          java.lang.String grpName,
                          PigContext pc)
                   throws FrontendException,
                          ExecException

Description copied from interface: ExecutionEngine

This method is responsible for the actual execution of a LogicalPlan. No assumptions is made about the architecture of the ExecutionEngine, except that it is capable of executing the LogicalPlan representation of a script. The ExecutionEngine should take care of all cleanup after executing the logical plan and returns an implementation of PigStats that contains the relevant information/statistics of the execution of the script.

Specified by:

launchPig in interface ExecutionEngine

Parameters:

lp - -- plan to compile

grpName - -- group name for submission

pc - -- context for execution

Throws:

ExecException

FrontendException

explain

public void explain(LogicalPlan lp,
                    PigContext pc,
                    java.io.PrintStream ps,
                    java.lang.String format,
                    boolean verbose,
                    java.io.File file,
                    java.lang.String suffix)
             throws PlanException,
                    VisitorException,
                    java.io.IOException,
                    FrontendException

Description copied from interface: ExecutionEngine

This method handles the backend processing of the Explain command. Once again, no assumptions is made about the architecture of the ExecutionEngine, except that it is capable of "explaining" the LogicalPlan representation of a script. The ExecutionEngine should print all of it's explain statements in the PrintStream provided UNLESS the File object is NOT null. In that case, the file is the directory for which the ExecutionEngine must write out the explain statements into semantically distinct files. For example, if the ExecutionEngine operates on a PhysicalPlan and an ExecutionPlan then there would be two separate files detailing each. The suffix param indicates the suffix of each of these file names.

Specified by:

explain in interface ExecutionEngine

Parameters:

lp - -- plan to explain

pc - -- context for explain processing

ps - -- print stream to write all output to (if dir param is null)

format - -- format to print explain

file - -- directory to write output to. if not null, write to files

suffix - -- if writing to files, suffix to be used for each file

Throws:

PlanException

VisitorException

java.io.IOException

FrontendException

getLogicalPlan

public LogicalPlan getLogicalPlan()

getConfiguration

public java.util.Properties getConfiguration()

Description copied from interface: ExecutionEngine

Returns the Properties representation of the ExecutionEngine configuration. The Properties object returned does not have to be the same object between distinct calls to getConfiguration(). The ExecutionEngine may create a new Properties object populated with all the properties each time.

Specified by:

getConfiguration in interface ExecutionEngine

setConfiguration

public void setConfiguration(java.util.Properties newConfiguration)
                      throws ExecException

Description copied from interface: ExecutionEngine

Responsible for updating the properties for the ExecutionEngine. The update may require reinitialization of the engine, perhaps through another call to init() if appropriate. This decision is delegated to the ExecutionEngine -- that is, the caller will not call init() after updating the properties. The Properties passed in should replace any configuration that occurred from previous Properties object. The Properties object should also be updated to reflect the deprecation/modifications that the ExecutionEngine may trigger.

Specified by:

setConfiguration in interface ExecutionEngine

Parameters:

newConfiguration - -- Properties object holding all configuration vals

Throws:

ExecException

setProperty

public void setProperty(java.lang.String property,
                        java.lang.String value)

Description copied from interface: ExecutionEngine

Responsible for setting a specific property and value. This method may be called as a result of a user "SET" command in the script or elsewhere in Pig to set certain properties. The properties object of the PigContext should be updated with the property and value with deprecation/other configuration done by the ExecutionEngine reflected. The ExecutionEngine should also update its internal configuration view as well.

Specified by:

setProperty in interface ExecutionEngine

Parameters:

property - to update

value - to set for property

getExecutableManager

public ExecutableManager getExecutableManager()

Description copied from interface: ExecutionEngine

Returns the ExecutableManager to be used in Pig Streaming.

Specified by:

getExecutableManager in interface ExecutionEngine

Returns:

ExecutableManager to be used in Pig Streaming.

kill

public void kill()
          throws BackendException

Description copied from interface: ExecutionEngine

This method is called when user requests to kill all jobs associated with the execution engine

Specified by:

kill in interface ExecutionEngine

Throws:

BackendException

killJob

public void killJob(java.lang.String jobID)
             throws BackendException

Description copied from interface: ExecutionEngine

This method is called when a user requests to kill a job associated with the given job id. If it is not possible for a user to kill a job, throw a exception. It is imperative for the job id's being displayed to be unique such that the correct jobs are being killed when the user supplies the id.

Specified by:

killJob in interface ExecutionEngine

Throws:

BackendException

destroy

public void destroy()

Description copied from interface: ExecutionEngine

Perform any cleanup operation

Specified by:

destroy in interface ExecutionEngine