org.apache.hadoop.hbase.io.hfile

Class HFileWriterImpl

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFileWriterImpl

All Implemented Interfaces:

Closeable, AutoCloseable, HFile.Writer, CellSink, ShipperListener

@InterfaceAudience.Private
public class HFileWriterImpl
extends Object
implements HFile.Writer

Common functionality needed by all versions of HFile writers.

Field Summary

Fields

Modifier and Type	Field and Description
protected HFileDataBlockEncoder	blockEncoder The data block encoding which will be used.
protected org.apache.hadoop.hbase.io.hfile.HFileBlock.Writer	blockWriter block writer
protected CacheConfig	cacheConf Cache configuration for caching data on write.
protected boolean	closeOutputStream True if we opened the outputStream (and so will close it).
protected CellComparator	comparator Key comparator.
protected long	entryCount Total # of key/value entries, i.e.
protected HFile.FileInfo	fileInfo A "file info" block: a key-value map of file-wide metadata.
protected Cell	firstCellInBlock First cell in a block.
protected HFileContext	hFileContext
static int	KEY_VALUE_VER_WITH_MEMSTORE Version for KeyValue which includes memstore timestamp
static byte[]	KEY_VALUE_VERSION KeyValue version in FileInfo
protected Cell	lastCell The Cell previously appended.
protected long	lastDataBlockOffset The offset of the last data block or 0 if the file is empty.
protected long	maxMemstoreTS
protected List<org.apache.hadoop.io.Writable>	metaData Writables representing meta block data.
protected List<byte[]>	metaNames Meta block names.
protected String	name Name for this object used when logging or in toString.
protected org.apache.hadoop.fs.FSDataOutputStream	outputStream FileSystem stream to write into.
protected org.apache.hadoop.fs.Path	path May be null if we were passed a stream.
protected long	totalKeyLength Used for calculating the average key length.
protected long	totalUncompressedBytes Total uncompressed bytes, maybe calculate a compression ratio later.
protected long	totalValueLength Used for calculating the average value length.
static String	UNIFIED_ENCODED_BLOCKSIZE_RATIO if this feature is enabled, preCalculate encoded data size before real encoding happens

Fields inherited from interface org.apache.hadoop.hbase.io.hfile.HFile.Writer

MAX_MEMSTORE_TS_KEY

Constructor Summary

Constructors

Constructor and Description
HFileWriterImpl(org.apache.hadoop.conf.Configuration conf, CacheConfig cacheConf, org.apache.hadoop.fs.Path path, org.apache.hadoop.fs.FSDataOutputStream outputStream, CellComparator comparator, HFileContext fileContext)

Method Summary

Modifier and Type	Method and Description
void	addDeleteFamilyBloomFilter(BloomFilterWriter bfw) Store delete family Bloom filter in the file, which is only supported in HFile V2.
void	addGeneralBloomFilter(BloomFilterWriter bfw) Store general Bloom filter in the file.
void	addInlineBlockWriter(InlineBlockWriter ibw) Adds an inline block writer such as a multi-level block index writer or a compound Bloom filter writer.
void	append(Cell cell) Add key/value to file.
void	appendFileInfo(byte[] k, byte[] v) Add to the file info.
void	appendMetaBlock(String metaBlockName, org.apache.hadoop.io.Writable content) Add a meta block to the end of the file.
void	beforeShipped() The action that needs to be performed before Shipper.shipped() is performed
protected void	checkBlockBoundary() At a block boundary, write all the inline blocks and opens new block.
protected boolean	checkKey(Cell cell) Checks that the given Cell's key does not violate the key order.
protected void	checkValue(byte[] value, int offset, int length) Checks the given value for validity.
void	close()
static Compression.Algorithm	compressionByName(String algoName)
protected static org.apache.hadoop.fs.FSDataOutputStream	createOutputStream(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, InetSocketAddress[] favoredNodes) A helper method to create HFile output streams in constructors
protected void	finishClose(FixedFileTrailer trailer)
protected void	finishFileInfo()
protected void	finishInit(org.apache.hadoop.conf.Configuration conf) Additional initialization steps
HFileContext	getFileContext() Return the file context for the HFile this writer belongs to
protected int	getMajorVersion()
static Cell	getMidpoint(CellComparator comparator, Cell left, Cell right) Try to return a Cell that falls between left and right but that is shorter; i.e.
protected int	getMinorVersion()
org.apache.hadoop.fs.Path	getPath()
protected void	newBlock() Ready a new block for writing.
String	toString()
protected void	writeFileInfo(FixedFileTrailer trailer, DataOutputStream out) Sets the file info offset in the trailer, finishes up populating fields in the file info, and writes the file info into the given data output.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

UNIFIED_ENCODED_BLOCKSIZE_RATIO

public static final String UNIFIED_ENCODED_BLOCKSIZE_RATIO

if this feature is enabled, preCalculate encoded data size before real encoding happens

See Also:

Constant Field Values

lastCell

protected Cell lastCell

The Cell previously appended. Becomes the last cell in the file.

outputStream

protected org.apache.hadoop.fs.FSDataOutputStream outputStream

FileSystem stream to write into.

closeOutputStream

protected final boolean closeOutputStream

True if we opened the outputStream (and so will close it).

fileInfo

protected HFile.FileInfo fileInfo

A "file info" block: a key-value map of file-wide metadata.

entryCount

protected long entryCount

Total # of key/value entries, i.e. how many times add() was called.

totalKeyLength

protected long totalKeyLength

Used for calculating the average key length.

totalValueLength

protected long totalValueLength

Used for calculating the average value length.

totalUncompressedBytes

protected long totalUncompressedBytes

Total uncompressed bytes, maybe calculate a compression ratio later.

comparator

protected final CellComparator comparator

Key comparator. Used to ensure we write in order.

metaNames

protected List<byte[]> metaNames

Meta block names.

metaData

protected List<org.apache.hadoop.io.Writable> metaData

Writables representing meta block data.

firstCellInBlock

protected Cell firstCellInBlock

First cell in a block. This reference should be short-lived since we write hfiles in a burst.

path

protected final org.apache.hadoop.fs.Path path

May be null if we were passed a stream.

cacheConf

protected final CacheConfig cacheConf

Cache configuration for caching data on write.

name

protected final String name

Name for this object used when logging or in toString. Is either the result of a toString on stream or else name of passed file Path.

blockEncoder

protected final HFileDataBlockEncoder blockEncoder

The data block encoding which will be used. NoOpDataBlockEncoder.INSTANCE if there is no encoding.

hFileContext

protected final HFileContext hFileContext

KEY_VALUE_VERSION

public static final byte[] KEY_VALUE_VERSION

KeyValue version in FileInfo

KEY_VALUE_VER_WITH_MEMSTORE

public static final int KEY_VALUE_VER_WITH_MEMSTORE

Version for KeyValue which includes memstore timestamp

See Also:

Constant Field Values

blockWriter

protected org.apache.hadoop.hbase.io.hfile.HFileBlock.Writer blockWriter

block writer

lastDataBlockOffset

protected long lastDataBlockOffset

The offset of the last data block or 0 if the file is empty.

maxMemstoreTS

protected long maxMemstoreTS

Constructor Detail

HFileWriterImpl

public HFileWriterImpl(org.apache.hadoop.conf.Configuration conf,
                       CacheConfig cacheConf,
                       org.apache.hadoop.fs.Path path,
                       org.apache.hadoop.fs.FSDataOutputStream outputStream,
                       CellComparator comparator,
                       HFileContext fileContext)

Method Detail

appendFileInfo

public void appendFileInfo(byte[] k,
                           byte[] v)
                    throws IOException

Add to the file info. All added key/value pairs can be obtained using HFile.Reader.loadFileInfo().

Specified by:

appendFileInfo in interface HFile.Writer

Parameters:

k - Key

v - Value

Throws:

IOException - in case the key or the value are invalid

writeFileInfo

protected final void writeFileInfo(FixedFileTrailer trailer,
                                   DataOutputStream out)
                            throws IOException

Sets the file info offset in the trailer, finishes up populating fields in the file info, and writes the file info into the given data output. The reason the data output is not always outputStream is that we store file info as a block in version 2.

Parameters:

trailer - fixed file trailer

out - the data output to write the file info to

Throws:

IOException

checkKey

protected boolean checkKey(Cell cell)
                    throws IOException

Checks that the given Cell's key does not violate the key order.

Parameters:

cell - Cell whose key to check.

Returns:

true if the key is duplicate

Throws:

IOException - if the key or the key order is wrong

checkValue

protected void checkValue(byte[] value,
                          int offset,
                          int length)
                   throws IOException

Checks the given value for validity.

Throws:

IOException

getPath

public org.apache.hadoop.fs.Path getPath()

Specified by:

getPath in interface HFile.Writer

Returns:

Path or null if we were passed a stream rather than a Path.

toString

public String toString()

Overrides:

toString in class Object

compressionByName

public static Compression.Algorithm compressionByName(String algoName)

createOutputStream

protected static org.apache.hadoop.fs.FSDataOutputStream createOutputStream(org.apache.hadoop.conf.Configuration conf,
                                                                            org.apache.hadoop.fs.FileSystem fs,
                                                                            org.apache.hadoop.fs.Path path,
                                                                            InetSocketAddress[] favoredNodes)
                                                                     throws IOException

A helper method to create HFile output streams in constructors

Throws:

IOException

finishInit

protected void finishInit(org.apache.hadoop.conf.Configuration conf)

Additional initialization steps

checkBlockBoundary

protected void checkBlockBoundary()
                           throws IOException

At a block boundary, write all the inline blocks and opens new block.

Throws:

IOException

getMidpoint

public static Cell getMidpoint(CellComparator comparator,
                               Cell left,
                               Cell right)

Try to return a Cell that falls between left and right but that is shorter; i.e. takes up less space. This trick is used building HFile block index. Its an optimization. It does not always work. In this case we'll just return the right cell.

Parameters:

comparator - Comparator to use.

left -

right -

Returns:

A cell that sorts between left and right.

newBlock

protected void newBlock()
                 throws IOException

Ready a new block for writing.

Throws:

IOException

appendMetaBlock

public void appendMetaBlock(String metaBlockName,
                            org.apache.hadoop.io.Writable content)

Add a meta block to the end of the file. Call before close(). Metadata blocks are expensive. Fill one with a bunch of serialized data rather than do a metadata block per metadata instance. If metadata is small, consider adding to file info using appendFileInfo(byte[], byte[])

Specified by:

appendMetaBlock in interface HFile.Writer

Parameters:

metaBlockName - name of the block

content - will call readFields to get data later (DO NOT REUSE)

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Throws:

IOException

addInlineBlockWriter

public void addInlineBlockWriter(InlineBlockWriter ibw)

Description copied from interface: HFile.Writer

Adds an inline block writer such as a multi-level block index writer or a compound Bloom filter writer.

Specified by:

addInlineBlockWriter in interface HFile.Writer

addGeneralBloomFilter

public void addGeneralBloomFilter(BloomFilterWriter bfw)

Description copied from interface: HFile.Writer

Store general Bloom filter in the file. This does not deal with Bloom filter internals but is necessary, since Bloom filters are stored differently in HFile version 1 and version 2.

Specified by:

addGeneralBloomFilter in interface HFile.Writer

addDeleteFamilyBloomFilter

public void addDeleteFamilyBloomFilter(BloomFilterWriter bfw)

Description copied from interface: HFile.Writer

Store delete family Bloom filter in the file, which is only supported in HFile V2.

Specified by:

addDeleteFamilyBloomFilter in interface HFile.Writer

getFileContext

public HFileContext getFileContext()

Description copied from interface: HFile.Writer

Return the file context for the HFile this writer belongs to

Specified by:

getFileContext in interface HFile.Writer

append

public void append(Cell cell)
            throws IOException

Add key/value to file. Keys must be added in an order that agrees with the Comparator passed on construction.

Specified by:

append in interface CellSink

Parameters:

cell - Cell to add. Cannot be empty nor null.

Throws:

IOException

beforeShipped

public void beforeShipped()
                   throws IOException

Description copied from interface: ShipperListener

The action that needs to be performed before Shipper.shipped() is performed

Specified by:

beforeShipped in interface ShipperListener

Throws:

IOException

finishFileInfo

protected void finishFileInfo()
                       throws IOException

Throws:

IOException

getMajorVersion

protected int getMajorVersion()

getMinorVersion

protected int getMinorVersion()

finishClose

protected void finishClose(FixedFileTrailer trailer)
                    throws IOException

Throws:

IOException