org.apache.hadoop.hbase.regionserver

Class HStoreFile

java.lang.Object

org.apache.hadoop.hbase.regionserver.HStoreFile

All Implemented Interfaces:

StoreFile, StoreFileReader.Listener

@InterfaceAudience.Private
public class HStoreFile
extends Object
implements StoreFile, StoreFileReader.Listener

A Store data file. Stores usually have one or more of these files. They are produced by flushing the memstore to disk. To create, instantiate a writer using StoreFileWriter.Builder and append data. Be sure to add any metadata before calling close on the Writer (Use the appendMetadata convenience methods). On close, a StoreFile is sitting in the Filesystem. To refer to it, create a StoreFile instance passing filesystem and path. To read, call initReader()

StoreFiles may also reference store files in another Store. The reason for this weird pattern where you use a different instance for the writer and a reader is that we write once but read a lot more.

Field Summary

Fields

Modifier and Type	Field and Description
static byte[]	BLOOM_FILTER_TYPE_KEY Bloom filter Type in FileInfo
static byte[]	BULKLOAD_TASK_KEY Meta key set when store file is a result of a bulk load
static byte[]	BULKLOAD_TIME_KEY
static byte[]	DELETE_FAMILY_COUNT Delete Family Count in FileInfo
static byte[]	EARLIEST_PUT_TS Key for timestamp of earliest-put in metadata
static byte[]	EXCLUDE_FROM_MINOR_COMPACTION_KEY Minor compaction flag in FileInfo
static byte[]	LAST_BLOOM_KEY Last Bloom filter key in FileInfo
static byte[]	MAJOR_COMPACTION_KEY Major compaction flag in FileInfo
static byte[]	MAX_SEQ_ID_KEY Max Sequence ID in FileInfo
static byte[]	MOB_CELLS_COUNT Key for the number of mob cells in metadata
static byte[]	SKIP_RESET_SEQ_ID Key for skipping resetting sequence id in metadata.
static String	STORE_FILE_READER_NO_READAHEAD
static byte[]	TIMERANGE_KEY Key for Timerange information in metadata

Constructor Summary

Constructors

Constructor and Description
HStoreFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p, org.apache.hadoop.conf.Configuration conf, CacheConfig cacheConf, BloomType cfBloomType, boolean primaryReplica) Constructor, loads a reader and it's indices, etc.
HStoreFile(org.apache.hadoop.fs.FileSystem fs, StoreFileInfo fileInfo, org.apache.hadoop.conf.Configuration conf, CacheConfig cacheConf, BloomType cfBloomType, boolean primaryReplica) Constructor, loads a reader and it's indices, etc.

Method Summary

Modifier and Type	Method and Description
void	closeStoreFile(boolean evictOnClose)
void	closeStreamReaders(boolean evictOnClose)
void	deleteStoreFile() Delete this file
boolean	excludeFromMinorCompaction()
OptionalLong	getBulkLoadTimestamp() Return the timestamp at which this bulk load file was generated.
CacheConfig	getCacheConf()
CellComparator	getComparator() Get the comparator for comparing two cells.
StoreFileInfo	getFileInfo()
Optional<Cell>	getFirstKey() Get the first key in this store file.
HDFSBlocksDistribution	getHDFSBlockDistribution()
Optional<Cell>	getLastKey() Get the last key in this store file.
OptionalLong	getMaximumTimestamp() Get the max timestamp of all the cells in the store file.
long	getMaxMemStoreTS() Get max of the MemstoreTS in the KV's in this store file.
long	getMaxSequenceId()
byte[]	getMetadataValue(byte[] key) Only used by the Striped Compaction Policy
OptionalLong	getMinimumTimestamp() Get the min timestamp of all the cells in the store file.
long	getModificationTimestamp() Get the modification time of this store file.
long	getModificationTimeStamp() Get the modification time of this store file.
org.apache.hadoop.fs.Path	getPath()
StoreFileScanner	getPreadScanner(boolean cacheBlocks, long readPt, long scannerOrder, boolean canOptimizeForNonNullColumn) Get a scanner which uses pread.
org.apache.hadoop.fs.Path	getQualifiedPath()
StoreFileReader	getReader()
int	getRefCount()
StoreFileScanner	getStreamScanner(boolean canUseDropBehind, boolean cacheBlocks, boolean isCompaction, long readPt, long scannerOrder, boolean canOptimizeForNonNullColumn) Get a scanner which uses streaming read.
void	initReader() Initialize the reader used for pread.
boolean	isBulkLoadResult() Check if this storefile was created by bulk load.
boolean	isCompactedAway()
boolean	isHFile()
boolean	isMajorCompactionResult()
boolean	isReference()
boolean	isReferencedInReads()
void	markCompactedAway()
void	storeFileReaderClosed(StoreFileReader reader)
String	toString()
String	toStringDetailed()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

STORE_FILE_READER_NO_READAHEAD

public static final String STORE_FILE_READER_NO_READAHEAD

See Also:

Constant Field Values

MAX_SEQ_ID_KEY

public static final byte[] MAX_SEQ_ID_KEY

Max Sequence ID in FileInfo

MAJOR_COMPACTION_KEY

public static final byte[] MAJOR_COMPACTION_KEY

Major compaction flag in FileInfo

EXCLUDE_FROM_MINOR_COMPACTION_KEY

public static final byte[] EXCLUDE_FROM_MINOR_COMPACTION_KEY

Minor compaction flag in FileInfo

BLOOM_FILTER_TYPE_KEY

public static final byte[] BLOOM_FILTER_TYPE_KEY

Bloom filter Type in FileInfo

DELETE_FAMILY_COUNT

public static final byte[] DELETE_FAMILY_COUNT

Delete Family Count in FileInfo

LAST_BLOOM_KEY

public static final byte[] LAST_BLOOM_KEY

Last Bloom filter key in FileInfo

TIMERANGE_KEY

public static final byte[] TIMERANGE_KEY

Key for Timerange information in metadata

EARLIEST_PUT_TS

public static final byte[] EARLIEST_PUT_TS

Key for timestamp of earliest-put in metadata

MOB_CELLS_COUNT

public static final byte[] MOB_CELLS_COUNT

Key for the number of mob cells in metadata

BULKLOAD_TASK_KEY

public static final byte[] BULKLOAD_TASK_KEY

Meta key set when store file is a result of a bulk load

BULKLOAD_TIME_KEY

public static final byte[] BULKLOAD_TIME_KEY

SKIP_RESET_SEQ_ID

public static final byte[] SKIP_RESET_SEQ_ID

Key for skipping resetting sequence id in metadata. For bulk loaded hfiles, the scanner resets the cell seqId with the latest one, if this metadata is set as true, the reset is skipped.

Constructor Detail

HStoreFile

public HStoreFile(org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path p,
                  org.apache.hadoop.conf.Configuration conf,
                  CacheConfig cacheConf,
                  BloomType cfBloomType,
                  boolean primaryReplica)
           throws IOException

Constructor, loads a reader and it's indices, etc. May allocate a substantial amount of ram depending on the underlying files (10-20MB?).

Parameters:

fs - The current file system to use.

p - The path of the file.

conf - The current configuration.

cacheConf - The cache configuration and block cache reference.

cfBloomType - The bloom type to use for this store file as specified by column family configuration. This may or may not be the same as the Bloom filter type actually present in the HFile, because column family configuration might change. If this is BloomType.NONE, the existing Bloom filter is ignored.

primaryReplica - true if this is a store file for primary replica, otherwise false.

Throws:

IOException

HStoreFile

public HStoreFile(org.apache.hadoop.fs.FileSystem fs,
                  StoreFileInfo fileInfo,
                  org.apache.hadoop.conf.Configuration conf,
                  CacheConfig cacheConf,
                  BloomType cfBloomType,
                  boolean primaryReplica)

Constructor, loads a reader and it's indices, etc. May allocate a substantial amount of ram depending on the underlying files (10-20MB?).

Parameters:

fs - fs The current file system to use.

fileInfo - The store file information.

conf - The current configuration.

cacheConf - The cache configuration and block cache reference.

cfBloomType - The bloom type to use for this store file as specified by column family configuration. This may or may not be the same as the Bloom filter type actually present in the HFile, because column family configuration might change. If this is BloomType.NONE, the existing Bloom filter is ignored.

primaryReplica - true if this is a store file for primary replica, otherwise false.

Method Detail

getCacheConf

public CacheConfig getCacheConf()

getFirstKey

public Optional<Cell> getFirstKey()

Description copied from interface: StoreFile

Get the first key in this store file.

Specified by:

getFirstKey in interface StoreFile

getLastKey

public Optional<Cell> getLastKey()

Description copied from interface: StoreFile

Get the last key in this store file.

Specified by:

getLastKey in interface StoreFile

getComparator

public CellComparator getComparator()

Description copied from interface: StoreFile

Get the comparator for comparing two cells.

Specified by:

getComparator in interface StoreFile

getMaxMemStoreTS

public long getMaxMemStoreTS()

Description copied from interface: StoreFile

Get max of the MemstoreTS in the KV's in this store file.

Specified by:

getMaxMemStoreTS in interface StoreFile

getFileInfo

public StoreFileInfo getFileInfo()

Returns:

the StoreFile object associated to this StoreFile. null if the StoreFile is not a reference.

getPath

public org.apache.hadoop.fs.Path getPath()

Specified by:

getPath in interface StoreFile

Returns:

Path or null if this StoreFile was made with a Stream.

getQualifiedPath

public org.apache.hadoop.fs.Path getQualifiedPath()

Specified by:

getQualifiedPath in interface StoreFile

Returns:

Returns the qualified path of this StoreFile

isReference

public boolean isReference()

Specified by:

isReference in interface StoreFile

Returns:

True if this is a StoreFile Reference.

isHFile

public boolean isHFile()

Specified by:

isHFile in interface StoreFile

Returns:

True if this is HFile.

isMajorCompactionResult

public boolean isMajorCompactionResult()

Specified by:

isMajorCompactionResult in interface StoreFile

Returns:

True if this file was made by a major compaction.

excludeFromMinorCompaction

public boolean excludeFromMinorCompaction()

Specified by:

excludeFromMinorCompaction in interface StoreFile

Returns:

True if this file should not be part of a minor compaction.

getMaxSequenceId

public long getMaxSequenceId()

Specified by:

getMaxSequenceId in interface StoreFile

Returns:

This files maximum edit sequence id.

getModificationTimeStamp

public long getModificationTimeStamp()
                              throws IOException

Description copied from interface: StoreFile

Get the modification time of this store file. Usually will access the file system so throws IOException.

Specified by:

getModificationTimeStamp in interface StoreFile

Throws:

IOException

See Also:

StoreFile.getModificationTimestamp()

getModificationTimestamp

public long getModificationTimestamp()
                              throws IOException

Description copied from interface: StoreFile

Get the modification time of this store file. Usually will access the file system so throws IOException.

Specified by:

getModificationTimestamp in interface StoreFile

Throws:

IOException

getMetadataValue

public byte[] getMetadataValue(byte[] key)

Only used by the Striped Compaction Policy

Parameters:

key -

Returns:

value associated with the metadata key

isBulkLoadResult

public boolean isBulkLoadResult()

Description copied from interface: StoreFile

Check if this storefile was created by bulk load. When a hfile is bulk loaded into HBase, we append '_SeqId_<id-when-loaded>' to the hfile name, unless "hbase.mapreduce.bulkload.assign.sequenceNumbers" is explicitly turned off. If "hbase.mapreduce.bulkload.assign.sequenceNumbers" is turned off, fall back to BULKLOAD_TIME_KEY.

Specified by:

isBulkLoadResult in interface StoreFile

Returns:

true if this storefile was created by bulk load.

isCompactedAway

public boolean isCompactedAway()

getRefCount

public int getRefCount()

isReferencedInReads

public boolean isReferencedInReads()

Returns:

true if the file is still used in reads

getBulkLoadTimestamp

public OptionalLong getBulkLoadTimestamp()

Description copied from interface: StoreFile

Return the timestamp at which this bulk load file was generated.

Specified by:

getBulkLoadTimestamp in interface StoreFile

getHDFSBlockDistribution

public HDFSBlocksDistribution getHDFSBlockDistribution()

Returns:

the cached value of HDFS blocks distribution. The cached value is calculated when store file is opened.

initReader

public void initReader()
                throws IOException

Initialize the reader used for pread.

Throws:

IOException

getPreadScanner

public StoreFileScanner getPreadScanner(boolean cacheBlocks,
                                        long readPt,
                                        long scannerOrder,
                                        boolean canOptimizeForNonNullColumn)

Get a scanner which uses pread.

Must be called after initReader.

getStreamScanner

public StoreFileScanner getStreamScanner(boolean canUseDropBehind,
                                         boolean cacheBlocks,
                                         boolean isCompaction,
                                         long readPt,
                                         long scannerOrder,
                                         boolean canOptimizeForNonNullColumn)
                                  throws IOException

Get a scanner which uses streaming read.

Must be called after initReader.

Throws:

IOException

getReader

public StoreFileReader getReader()

Returns:

Current reader. Must call initReader first else returns null.

See Also:

initReader()

closeStoreFile

public void closeStoreFile(boolean evictOnClose)
                    throws IOException

Parameters:

evictOnClose - whether to evict blocks belonging to this file

Throws:

IOException

closeStreamReaders

public void closeStreamReaders(boolean evictOnClose)
                        throws IOException

Throws:

IOException

deleteStoreFile

public void deleteStoreFile()
                     throws IOException

Delete this file

Throws:

IOException

markCompactedAway

public void markCompactedAway()

toString

public String toString()

Overrides:

toString in class Object

toStringDetailed

public String toStringDetailed()

Specified by:

toStringDetailed in interface StoreFile

Returns:

a length description of this StoreFile, suitable for debug output

getMinimumTimestamp

public OptionalLong getMinimumTimestamp()

Description copied from interface: StoreFile

Get the min timestamp of all the cells in the store file.

Specified by:

getMinimumTimestamp in interface StoreFile

getMaximumTimestamp

public OptionalLong getMaximumTimestamp()

Description copied from interface: StoreFile

Get the max timestamp of all the cells in the store file.

Specified by:

getMaximumTimestamp in interface StoreFile

storeFileReaderClosed

public void storeFileReaderClosed(StoreFileReader reader)

Specified by:

storeFileReaderClosed in interface StoreFileReader.Listener