org.apache.pig.impl.util

Class Utils

java.lang.Object

org.apache.pig.impl.util.Utils

public class Utils
extends java.lang.Object

Class with utility static methods

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.hadoop.fs.PathFilter	VISIBLE_FILES A PathFilter that filters out invisible files.

Constructor Summary

Constructors

Constructor and Description
Utils()

Method Summary

Modifier and Type	Method and Description
static void	addShutdownHookWithPriority(java.lang.Runnable hook, int priority) Add shutdown hook that runs before the FileSystem cache shutdown happens.
static FuncSpec	buildSimpleFuncSpec(java.lang.String className, byte... types)
static boolean	checkNullAndClass(java.lang.Object obj1, java.lang.Object obj2) This method is a helper for classes to implement Object.equals(java.lang.Object) The method checks whether the two arguments are both null or both not null and whether they are of the same class
static boolean	checkNullEquals(java.lang.Object obj1, java.lang.Object obj2, boolean checkEquality) This method is a helper for classes to implement Object.equals(java.lang.Object) checks if two objects are equals - two levels of checks are made - first if both are null or not null.
static org.apache.hadoop.fs.Path	depthFirstSearchForFile(org.apache.hadoop.fs.FileStatus[] statusArray, org.apache.hadoop.fs.FileSystem fileSystem) Finds a valid path for a file from a FileStatus object.
static org.apache.hadoop.fs.Path	depthFirstSearchForFile(org.apache.hadoop.fs.FileStatus[] statusArray, org.apache.hadoop.fs.FileSystem fileSystem, org.apache.hadoop.fs.PathFilter filter) Finds a valid path for a file from an array of FileStatus objects.
static int	extractHeapSizeInMB(java.lang.String input)
static java.io.InputStream	getCompositeStream(java.io.InputStream in, java.util.Properties properties)
static ResourceSchema	getSchema(LoadFunc wrappedLoadFunc, java.lang.String location, boolean checkExistence, org.apache.hadoop.mapreduce.Job job)
static Schema	getSchemaFromBagSchemaString(java.lang.String schemaString) getSchemaFromBagSchemaString NOTE: use this call when you need to generate a Schema object from the representation generated by Schema#toString.
static Schema	getSchemaFromString(java.lang.String schemaString)
static ResourceSchema	getSchemaWithInputSourceTag(ResourceSchema schema, java.lang.String fieldName) This method adds FieldSchema of 'input source tag/path' as the first field.
static Schema	getScriptSchema(java.lang.String loadFuncSignature, org.apache.hadoop.conf.Configuration conf) A helper function for retrieving the script schema set by the LOLoad function.
static java.lang.String	getScriptSchemaKey(java.lang.String loadFuncSignature)
static java.lang.String	getStackStraceStr(java.lang.Throwable e)
static java.lang.String	getStringFromArray(java.lang.String[] arr)
static java.lang.String	getTmpFileCompressorName(PigContext pigContext)
static java.lang.Class<? extends FileInputLoadFunc>	getTmpFileStorageClass(java.util.Properties properties)
static FileInputLoadFunc	getTmpFileStorageObject(org.apache.hadoop.conf.Configuration conf)
static boolean	is64bitJVM()
static boolean	isLocal(org.apache.hadoop.conf.Configuration conf)
static boolean	isLocal(PigContext pigContext, org.apache.hadoop.conf.Configuration conf)
static boolean	isVendorIBM() This method checks whether JVM vendor is IBM
static <O> java.util.Collection<O>	mergeCollection(java.util.Collection<O> a, java.util.Collection<O> b)
static java.lang.Object	parseConstant(java.lang.String constantString)
static LogicalSchema	parseSchema(java.lang.String schemaString)
static void	recomputeProperties(org.apache.hadoop.mapred.JobConf jobConf, java.util.Properties properties) Method to apply pig properties to JobConf (replaces properties with resulting jobConf values).
static void	setDefaultTimeZone(org.apache.hadoop.conf.Configuration conf)
static void	setMapredCompressionCodecProps(org.apache.hadoop.conf.Configuration conf)
static void	setTmpFileCompressionOnConf(PigContext pigContext, org.apache.hadoop.conf.Configuration conf)
static java.lang.String	slashisize(java.lang.String str) Replace sequences of two slashes ("\\") with one slash ("\") (not escaping a slash in grunt is disallowed, but a double slash doesn't get converted into a regular slash, so we have to do it instead)
static java.lang.String	substituteVars(java.lang.String expr)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

VISIBLE_FILES

public static final org.apache.hadoop.fs.PathFilter VISIBLE_FILES

A PathFilter that filters out invisible files.

Constructor Detail

Utils

public Utils()

Method Detail

isVendorIBM

public static boolean isVendorIBM()

This method checks whether JVM vendor is IBM

Returns:

true if IBM JVM is being used false otherwise

is64bitJVM

public static boolean is64bitJVM()

checkNullEquals

public static boolean checkNullEquals(java.lang.Object obj1,
                                      java.lang.Object obj2,
                                      boolean checkEquality)

This method is a helper for classes to implement Object.equals(java.lang.Object) checks if two objects are equals - two levels of checks are made - first if both are null or not null. If either is null, check is made whether both are null. If both are non null, equality also is checked if so indicated

Parameters:

obj1 - first object to be compared

obj2 - second object to be compared

checkEquality - flag to indicate whether object equality should be checked if obj1 and obj2 are non-null

Returns:

true if the two objects are equal false otherwise

checkNullAndClass

public static boolean checkNullAndClass(java.lang.Object obj1,
                                        java.lang.Object obj2)

This method is a helper for classes to implement Object.equals(java.lang.Object) The method checks whether the two arguments are both null or both not null and whether they are of the same class

Parameters:

obj1 - first object to compare

obj2 - second object to compare

Returns:

true if both objects are null or both are not null and if both are of the same class if not null false otherwise

getScriptSchema

public static Schema getScriptSchema(java.lang.String loadFuncSignature,
                                     org.apache.hadoop.conf.Configuration conf)
                              throws java.io.IOException

A helper function for retrieving the script schema set by the LOLoad function.

Parameters:

loadFuncSignature -

conf -

Returns:

Schema

Throws:

java.io.IOException

getScriptSchemaKey

public static java.lang.String getScriptSchemaKey(java.lang.String loadFuncSignature)

getSchema

public static ResourceSchema getSchema(LoadFunc wrappedLoadFunc,
                                       java.lang.String location,
                                       boolean checkExistence,
                                       org.apache.hadoop.mapreduce.Job job)
                                throws java.io.IOException

Throws:

java.io.IOException

getSchemaFromString

public static Schema getSchemaFromString(java.lang.String schemaString)
                                  throws ParserException

Parameters:

schemaString - a String representation of the Schema without any enclosing curly-braces.Not for use with Schema#toString

Returns:

Schema instance

Throws:

ParserException

getSchemaFromBagSchemaString

public static Schema getSchemaFromBagSchemaString(java.lang.String schemaString)
                                           throws ParserException

getSchemaFromBagSchemaString NOTE: use this call when you need to generate a Schema object from the representation generated by Schema#toString. This call strips the enclosing outer curly braces from the toString representation, which are placed there because the actual representation of the schema data is as a Bag-type relation.

Parameters:

schemaString - a String representation of the Schema to instantiate, in the form generated by Schema.toString()

Returns:

Schema instance

Throws:

ParserException

parseSchema

public static LogicalSchema parseSchema(java.lang.String schemaString)
                                 throws ParserException

Throws:

ParserException

parseConstant

public static java.lang.Object parseConstant(java.lang.String constantString)
                                      throws ParserException

Throws:

ParserException

getSchemaWithInputSourceTag

public static ResourceSchema getSchemaWithInputSourceTag(ResourceSchema schema,
                                                         java.lang.String fieldName)

This method adds FieldSchema of 'input source tag/path' as the first field. This will be called only when PigStorage is invoked with '-tagFile' or '-tagPath' option and the schema file is present to be loaded.

Parameters:

schema -

fieldName -

Returns:

ResourceSchema

getTmpFileCompressorName

public static java.lang.String getTmpFileCompressorName(PigContext pigContext)

getTmpFileStorageObject

public static FileInputLoadFunc getTmpFileStorageObject(org.apache.hadoop.conf.Configuration conf)
                                                 throws java.io.IOException

Throws:

java.io.IOException

getTmpFileStorageClass

public static java.lang.Class<? extends FileInputLoadFunc> getTmpFileStorageClass(java.util.Properties properties)

setMapredCompressionCodecProps

public static void setMapredCompressionCodecProps(org.apache.hadoop.conf.Configuration conf)

setTmpFileCompressionOnConf

public static void setTmpFileCompressionOnConf(PigContext pigContext,
                                               org.apache.hadoop.conf.Configuration conf)
                                        throws java.io.IOException

Throws:

java.io.IOException

getStringFromArray

public static java.lang.String getStringFromArray(java.lang.String[] arr)

buildSimpleFuncSpec

public static FuncSpec buildSimpleFuncSpec(java.lang.String className,
                                           byte... types)

slashisize

public static java.lang.String slashisize(java.lang.String str)

Replace sequences of two slashes ("\\") with one slash ("\") (not escaping a slash in grunt is disallowed, but a double slash doesn't get converted into a regular slash, so we have to do it instead)

Parameters:

str -

Returns:

the resulting string

mergeCollection

public static <O> java.util.Collection<O> mergeCollection(java.util.Collection<O> a,
                                                          java.util.Collection<O> b)

getCompositeStream

public static java.io.InputStream getCompositeStream(java.io.InputStream in,
                                                     java.util.Properties properties)

recomputeProperties

public static void recomputeProperties(org.apache.hadoop.mapred.JobConf jobConf,
                                       java.util.Properties properties)

Method to apply pig properties to JobConf (replaces properties with resulting jobConf values).

Parameters:

conf - JobConf with appropriate hadoop resource files

properties - Pig properties that will override hadoop properties; properties might be modified

getStackStraceStr

public static java.lang.String getStackStraceStr(java.lang.Throwable e)

isLocal

public static boolean isLocal(PigContext pigContext,
                              org.apache.hadoop.conf.Configuration conf)

isLocal

public static boolean isLocal(org.apache.hadoop.conf.Configuration conf)

substituteVars

public static java.lang.String substituteVars(java.lang.String expr)

depthFirstSearchForFile

public static org.apache.hadoop.fs.Path depthFirstSearchForFile(org.apache.hadoop.fs.FileStatus[] statusArray,
                                                                org.apache.hadoop.fs.FileSystem fileSystem)
                                                         throws java.io.IOException

Finds a valid path for a file from a FileStatus object.

Parameters:

fileStatus - FileStatus object corresponding to a file, or a directory.

fileSystem - FileSystem in with the file should be found

Returns:

The first file found

Throws:

java.io.IOException

depthFirstSearchForFile

public static org.apache.hadoop.fs.Path depthFirstSearchForFile(org.apache.hadoop.fs.FileStatus[] statusArray,
                                                                org.apache.hadoop.fs.FileSystem fileSystem,
                                                                org.apache.hadoop.fs.PathFilter filter)
                                                         throws java.io.IOException

Finds a valid path for a file from an array of FileStatus objects.

Parameters:

statusArray - Array of FileStatus objects in which to search for the file.

fileSystem - FileSystem in which to search for the first file.

Returns:

The first file found.

Throws:

java.io.IOException

extractHeapSizeInMB

public static int extractHeapSizeInMB(java.lang.String input)

setDefaultTimeZone

public static void setDefaultTimeZone(org.apache.hadoop.conf.Configuration conf)

addShutdownHookWithPriority

public static void addShutdownHookWithPriority(java.lang.Runnable hook,
                                               int priority)

Add shutdown hook that runs before the FileSystem cache shutdown happens.

Parameters:

hook - code to execute during shutdown

priority - Priority over the FileSystem.SHUTDOWN_HOOK_PRIORITY