org.apache.pig.builtin

Class TextLoader

java.lang.Object

org.apache.pig.LoadFunc

org.apache.pig.builtin.TextLoader

All Implemented Interfaces:

LoadCaster

public class TextLoader
extends LoadFunc
implements LoadCaster

This load function simply creates a tuple for each line of text that has a single chararray field that contains the line of text.

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.mapreduce.RecordReader	in
protected org.apache.commons.logging.Log	mLog

Constructor Summary

Constructors

Constructor and Description
TextLoader()

Method Summary

Modifier and Type	Method and Description
DataBag	bytesToBag(byte[] b, ResourceSchema.ResourceFieldSchema schema) TextLoader does not support conversion to Bag
java.math.BigDecimal	bytesToBigDecimal(byte[] b) Cast data from bytearray to BigDecimal value.
java.math.BigInteger	bytesToBigInteger(byte[] b) Cast data from bytearray to BigInteger value.
java.lang.Boolean	bytesToBoolean(byte[] b) TextLoader does not support conversion to Boolean
java.lang.String	bytesToCharArray(byte[] b) Cast data from bytes to chararray value.
org.joda.time.DateTime	bytesToDateTime(byte[] b) TextLoader does not support conversion to DateTime
java.lang.Double	bytesToDouble(byte[] b) TextLoader does not support conversion to Double
java.lang.Float	bytesToFloat(byte[] b) TextLoader does not support conversion to Float
java.lang.Integer	bytesToInteger(byte[] b) TextLoader does not support conversion to Integer
java.lang.Long	bytesToLong(byte[] b) TextLoader does not support conversion to Long
java.util.Map<java.lang.String,java.lang.Object>	bytesToMap(byte[] b, ResourceSchema.ResourceFieldSchema schema) Cast data from bytearray to map value.
Tuple	bytesToTuple(byte[] b, ResourceSchema.ResourceFieldSchema schema) TextLoader does not support conversion to Tuple
org.apache.hadoop.mapreduce.InputFormat	getInputFormat() This will be called during planning on the front end.
LoadCaster	getLoadCaster() This will be called on both the front end and the back end during execution.
Tuple	getNext() Retrieves the next tuple to be processed.
void	prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader, PigSplit split) Initializes LoadFunc for reading data.
void	setLocation(java.lang.String location, org.apache.hadoop.mapreduce.Job job) Communicate to the loader the location of the object(s) being loaded.
byte[]	toBytes(java.lang.Boolean b)
byte[]	toBytes(DataBag bag)
byte[]	toBytes(org.joda.time.DateTime dt)
byte[]	toBytes(java.lang.Double d)
byte[]	toBytes(java.lang.Float f)
byte[]	toBytes(java.lang.Integer i)
byte[]	toBytes(java.lang.Long l)
byte[]	toBytes(java.util.Map<java.lang.String,java.lang.Object> m)
byte[]	toBytes(java.lang.String s)
byte[]	toBytes(Tuple t)

Methods inherited from class org.apache.pig.LoadFunc

getAbsolutePath, getCacheFiles, getPathStrings, getShipFiles, join, relativeToAbsolutePath, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

in

protected org.apache.hadoop.mapreduce.RecordReader in

mLog

protected final org.apache.commons.logging.Log mLog

Constructor Detail

TextLoader

public TextLoader()

Method Detail

getNext

public Tuple getNext()
              throws java.io.IOException

Description copied from class: LoadFunc

Retrieves the next tuple to be processed. Implementations should NOT reuse tuple objects (or inner member objects) they return across calls and should return a different tuple object in each call.

Specified by:

getNext in class LoadFunc

Returns:

the next tuple to be processed or null if there are no more tuples to be processed.

Throws:

java.io.IOException - if there is an exception while retrieving the next tuple

bytesToBoolean

public java.lang.Boolean bytesToBoolean(byte[] b)
                                 throws java.io.IOException

TextLoader does not support conversion to Boolean

Specified by:

bytesToBoolean in interface LoadCaster

Parameters:

b - bytearray to be cast.

Returns:

Boolean value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToInteger

public java.lang.Integer bytesToInteger(byte[] b)
                                 throws java.io.IOException

TextLoader does not support conversion to Integer

Specified by:

bytesToInteger in interface LoadCaster

Parameters:

b - bytearray to be cast.

Returns:

Double value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToLong

public java.lang.Long bytesToLong(byte[] b)
                           throws java.io.IOException

TextLoader does not support conversion to Long

Specified by:

bytesToLong in interface LoadCaster

Parameters:

b - bytearray to be cast.

Returns:

Long value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToFloat

public java.lang.Float bytesToFloat(byte[] b)
                             throws java.io.IOException

TextLoader does not support conversion to Float

Specified by:

bytesToFloat in interface LoadCaster

Parameters:

b - bytearray to be cast.

Returns:

Float value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToDouble

public java.lang.Double bytesToDouble(byte[] b)
                               throws java.io.IOException

TextLoader does not support conversion to Double

Specified by:

bytesToDouble in interface LoadCaster

Parameters:

b - bytearray to be cast.

Returns:

Double value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToDateTime

public org.joda.time.DateTime bytesToDateTime(byte[] b)
                                       throws java.io.IOException

TextLoader does not support conversion to DateTime

Specified by:

bytesToDateTime in interface LoadCaster

Parameters:

b - bytearray to be cast.

Returns:

datetime value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToCharArray

public java.lang.String bytesToCharArray(byte[] b)
                                  throws java.io.IOException

Cast data from bytes to chararray value.

Specified by:

bytesToCharArray in interface LoadCaster

Parameters:

b - byte array to be cast.

Returns:

String value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToMap

public java.util.Map<java.lang.String,java.lang.Object> bytesToMap(byte[] b,
                                                                   ResourceSchema.ResourceFieldSchema schema)
                                                            throws java.io.IOException

Description copied from interface: LoadCaster

Cast data from bytearray to map value.

Specified by:

bytesToMap in interface LoadCaster

Parameters:

b - bytearray to be cast.

schema - field schema for the output map

Returns:

Map value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToTuple

public Tuple bytesToTuple(byte[] b,
                          ResourceSchema.ResourceFieldSchema schema)
                   throws java.io.IOException

TextLoader does not support conversion to Tuple

Specified by:

bytesToTuple in interface LoadCaster

Parameters:

b - bytearray to be cast.

schema - field schema for the output tuple

Returns:

Tuple value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToBag

public DataBag bytesToBag(byte[] b,
                          ResourceSchema.ResourceFieldSchema schema)
                   throws java.io.IOException

TextLoader does not support conversion to Bag

Specified by:

bytesToBag in interface LoadCaster

Parameters:

b - bytearray to be cast.

schema - field schema for the output bag

Returns:

Bag value.

Throws:

java.io.IOException - if the value cannot be cast.

toBytes

public byte[] toBytes(DataBag bag)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(java.lang.String s)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(java.lang.Double d)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(java.lang.Float f)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(java.lang.Boolean b)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(java.lang.Integer i)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(java.lang.Long l)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(org.joda.time.DateTime dt)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(java.util.Map<java.lang.String,java.lang.Object> m)
               throws java.io.IOException

Throws:

java.io.IOException

toBytes

public byte[] toBytes(Tuple t)
               throws java.io.IOException

Throws:

java.io.IOException

bytesToBigInteger

public java.math.BigInteger bytesToBigInteger(byte[] b)
                                       throws java.io.IOException

Description copied from interface: LoadCaster

Cast data from bytearray to BigInteger value.

Specified by:

bytesToBigInteger in interface LoadCaster

Parameters:

b - bytearray to be cast.

Returns:

BigInteger value.

Throws:

java.io.IOException - if the value cannot be cast.

bytesToBigDecimal

public java.math.BigDecimal bytesToBigDecimal(byte[] b)
                                       throws java.io.IOException

Description copied from interface: LoadCaster

Cast data from bytearray to BigDecimal value.

Specified by:

bytesToBigDecimal in interface LoadCaster

Parameters:

b - bytearray to be cast.

Returns:

BigInteger value.

Throws:

java.io.IOException - if the value cannot be cast.

getInputFormat

public org.apache.hadoop.mapreduce.InputFormat getInputFormat()

Description copied from class: LoadFunc

This will be called during planning on the front end. This is the instance of InputFormat (rather than the class name) because the load function may need to instantiate the InputFormat in order to control how it is constructed.

Specified by:

getInputFormat in class LoadFunc

Returns:

the InputFormat associated with this loader.

getLoadCaster

public LoadCaster getLoadCaster()

Description copied from class: LoadFunc

This will be called on both the front end and the back end during execution.

Overrides:

getLoadCaster in class LoadFunc

Returns:

the LoadCaster associated with this loader. Returning null indicates that casts from byte array are not supported for this loader. construction

prepareToRead

public void prepareToRead(org.apache.hadoop.mapreduce.RecordReader reader,
                          PigSplit split)

Description copied from class: LoadFunc

Initializes LoadFunc for reading data. This will be called during execution before any calls to getNext. The RecordReader needs to be passed here because it has been instantiated for a particular InputSplit.

Specified by:

prepareToRead in class LoadFunc

Parameters:

reader - RecordReader to be used by this instance of the LoadFunc

split - The input PigSplit to process

setLocation

public void setLocation(java.lang.String location,
                        org.apache.hadoop.mapreduce.Job job)
                 throws java.io.IOException

Description copied from class: LoadFunc

Communicate to the loader the location of the object(s) being loaded. The location string passed to the LoadFunc here is the return value of LoadFunc.relativeToAbsolutePath(String, Path). Implementations should use this method to communicate the location (and any other information) to its underlying InputFormat through the Job object. This method will be called in the frontend and backend multiple times. Implementations should bear in mind that this method is called multiple times and should ensure there are no inconsistent side effects due to the multiple calls.

Specified by:

setLocation in class LoadFunc

Parameters:

location - Location as returned by LoadFunc.relativeToAbsolutePath(String, Path)

job - the Job object store or retrieve earlier stored information from the UDFContext

Throws:

java.io.IOException - if the location is not valid.