org.apache.hadoop.hbase.io.hfile

Class HFileBlockIndex.BlockIndexWriter

java.lang.Object

org.apache.hadoop.hbase.io.hfile.HFileBlockIndex.BlockIndexWriter

All Implemented Interfaces:

InlineBlockWriter

Enclosing class:

HFileBlockIndex

public static class HFileBlockIndex.BlockIndexWriter
extends Object
implements InlineBlockWriter

Writes the block index into the output stream. Generate the tree from bottom up. The leaf level is written to disk as a sequence of inline blocks, if it is larger than a certain number of bytes. If the leaf level is not large enough, we write all entries to the root level instead. After all leaf blocks have been written, we end up with an index referencing the resulting leaf index blocks. If that index is larger than the allowed root index size, the writer will break it up into reasonable-size intermediate-level index block chunks write those chunks out, and create another index referencing those chunks. This will be repeated until the remaining index is small enough to become the root index. However, in most practical cases we will only have leaf-level blocks and the root index, or just the root index.

Constructor Summary

Constructors

Constructor and Description
BlockIndexWriter() Creates a single-level block index writer
BlockIndexWriter(org.apache.hadoop.hbase.io.hfile.HFileBlock.Writer blockWriter, CacheConfig cacheConf, String nameForCaching) Creates a multi-level block index writer.

Method Summary

Modifier and Type	Method and Description
void	addEntry(byte[] firstKey, long blockOffset, int blockDataSize) Add one index entry to the current leaf-level block.
void	blockWritten(long offset, int onDiskSize, int uncompressedSize) Called after an inline block has been written so that we can add an entry referring to that block to the parent-level index.
void	ensureSingleLevel()
boolean	getCacheOnWrite()
BlockType	getInlineBlockType() The type of blocks this block writer produces.
int	getNumLevels()
int	getNumRootEntries()
long	getTotalUncompressedSize() The total uncompressed size of the root index block, intermediate-level index blocks, and leaf-level index blocks.
void	setMaxChunkSize(int maxChunkSize)
void	setMinIndexNumEntries(int minIndexNumEntries)
boolean	shouldWriteBlock(boolean closing) Whether there is an inline block ready to be written.
long	writeIndexBlocks(org.apache.hadoop.fs.FSDataOutputStream out) Writes the root level and intermediate levels of the block index into the output stream, generating the tree from bottom up.
void	writeInlineBlock(DataOutput out) Write out the current inline index block.
void	writeSingleLevelIndex(DataOutput out, String description) Writes the block index data as a single level only.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

BlockIndexWriter

public BlockIndexWriter()

Creates a single-level block index writer

BlockIndexWriter

public BlockIndexWriter(org.apache.hadoop.hbase.io.hfile.HFileBlock.Writer blockWriter,
                        CacheConfig cacheConf,
                        String nameForCaching)

Creates a multi-level block index writer.

Parameters:

blockWriter - the block writer to use to write index blocks

cacheConf - used to determine when and how a block should be cached-on-write.

Method Detail

setMaxChunkSize

public void setMaxChunkSize(int maxChunkSize)

setMinIndexNumEntries

public void setMinIndexNumEntries(int minIndexNumEntries)

writeIndexBlocks

public long writeIndexBlocks(org.apache.hadoop.fs.FSDataOutputStream out)
                      throws IOException

Writes the root level and intermediate levels of the block index into the output stream, generating the tree from bottom up. Assumes that the leaf level has been inline-written to the disk if there is enough data for more than one leaf block. We iterate by breaking the current level of the block index, starting with the index of all leaf-level blocks, into chunks small enough to be written to disk, and generate its parent level, until we end up with a level small enough to become the root level. If the leaf level is not large enough, there is no inline block index anymore, so we only write that level of block index to disk as the root level.

Parameters:

out - FSDataOutputStream

Returns:

position at which we entered the root-level index.

Throws:

IOException

writeSingleLevelIndex

public void writeSingleLevelIndex(DataOutput out,
                                  String description)
                           throws IOException

Writes the block index data as a single level only. Does not do any block framing.

Parameters:

out - the buffered output stream to write the index to. Typically a stream writing into an HFile block.

description - a short description of the index being written. Used in a log message.

Throws:

IOException

getNumRootEntries

public final int getNumRootEntries()

Returns:

how many block index entries there are in the root level

getNumLevels

public int getNumLevels()

Returns:

the number of levels in this block index.

shouldWriteBlock

public boolean shouldWriteBlock(boolean closing)

Whether there is an inline block ready to be written. In general, we write an leaf-level index block as an inline block as soon as its size as serialized in the non-root format reaches a certain threshold.

Specified by:

shouldWriteBlock in interface InlineBlockWriter

Parameters:

closing - whether the file is being closed, in which case we need to write out all available data and not wait to accumulate another block

writeInlineBlock

public void writeInlineBlock(DataOutput out)
                      throws IOException

Write out the current inline index block. Inline blocks are non-root blocks, so the non-root index format is used.

Specified by:

writeInlineBlock in interface InlineBlockWriter

Parameters:

out -

Throws:

IOException

blockWritten

public void blockWritten(long offset,
                         int onDiskSize,
                         int uncompressedSize)

Called after an inline block has been written so that we can add an entry referring to that block to the parent-level index.

Specified by:

blockWritten in interface InlineBlockWriter

Parameters:

offset - the offset of the block in the stream

onDiskSize - the on-disk size of the block

uncompressedSize - the uncompressed size of the block

getInlineBlockType

public BlockType getInlineBlockType()

Description copied from interface: InlineBlockWriter

The type of blocks this block writer produces.

Specified by:

getInlineBlockType in interface InlineBlockWriter

addEntry

public void addEntry(byte[] firstKey,
                     long blockOffset,
                     int blockDataSize)

Add one index entry to the current leaf-level block. When the leaf-level block gets large enough, it will be flushed to disk as an inline block.

Parameters:

firstKey - the first key of the data block

blockOffset - the offset of the data block

blockDataSize - the on-disk size of the data block (HFile format version 2), or the uncompressed size of the data block ( HFile format version 1).

ensureSingleLevel

public void ensureSingleLevel()
                       throws IOException

Throws:

IOException - if we happened to write a multi-level index.

getCacheOnWrite

public boolean getCacheOnWrite()

Specified by:

getCacheOnWrite in interface InlineBlockWriter

Returns:

true if we are using cache-on-write. This is configured by the caller of the constructor by either passing a valid block cache or null.

getTotalUncompressedSize

public long getTotalUncompressedSize()

The total uncompressed size of the root index block, intermediate-level index blocks, and leaf-level index blocks.

Returns:

the total uncompressed size of all index blocks