Package org.apache.hadoop.hbase.io.hfile

Provides implementations of HFile and HFile BlockCache.

See: Description

Interface Summary

Interface	Description
BlockCache	Block cache interface.
Cacheable	Cacheable is an interface that allows for an object to be cached.
CacheableDeserializer<T extends Cacheable>	Interface for a deserializer.
CachedBlock
HFile.CachingBlockReader	An abstraction used by the block index.
HFile.Reader	An interface used by clients to open and iterate an HFile.
HFile.Writer	API required to write an HFile
HFileDataBlockEncoder	Controls what kind of data block encoding is used.
HFileScanner	A scanner allows you to position yourself within a HFile and scan through it.
InlineBlockWriter	A way to write "inline" blocks into an HFile.
ResizableBlockCache	BlockCache which is resizable.

Class Summary

Class	Description
AgeSnapshot	Snapshot of block cache age in cache.
BlockCacheKey	Cache Key for use with implementations of BlockCache
BlockCacheUtil	Utilty for aggregating counts in CachedBlocks and toString/toJSON CachedBlocks and BlockCaches.
BlockCacheUtil.CachedBlocksByFile	Use one of these to keep a running account of cached blocks by file.
BlockWithScanInfo	BlockWithScanInfo is wrapper class for HFileBlock with other attributes.
CacheableDeserializerIdManager	This class is used to manage the identifiers for CacheableDeserializer
CacheConfig	Stores all of the cache objects and configuration for a single HFile.
CacheStats	Class that implements cache metrics.
ChecksumUtil	Utility methods to compute and validate checksums.
CombinedBlockCache	CombinedBlockCache is an abstraction layer that combines LruBlockCache and BucketCache.
CombinedBlockCache.CombinedCacheStats
CompoundBloomFilter	A Bloom filter implementation built on top of BloomFilterChunk, encapsulating a set of fixed-size Bloom filters written out at the time of HFile generation into the data block stream, and loaded on demand at query time.
CompoundBloomFilterBase
CompoundBloomFilterWriter	Adds methods required for writing a compound Bloom filter to the data section of an HFile to the CompoundBloomFilter class.
FixedFileTrailer	The HFile has a fixed trailer which contains offsets to other variable parts of the file.
HFile	File format for hbase.
HFile.FileInfo	Metadata for this file.
HFile.WriterFactory	This variety of ways to construct writers is used throughout the code, and we want to be able to swap writer implementations.
HFileBlock	Cacheable Blocks of an HFile version 2 file.
HFileBlockIndex	Provides functionality to write (HFileBlockIndex.BlockIndexWriter) and read BlockIndexReader single-level and multi-level block indexes.
HFileBlockIndex.BlockIndexWriter	Writes the block index into the output stream.
HFileContext	This carries the information on some of the meta data about the HFile.
HFileContextBuilder	A builder that helps in building up the HFileContext
HFileDataBlockEncoderImpl	Do different kinds of data block encoding according to column family options.
HFilePrettyPrinter	Implements pretty-printing functionality for HFiles.
HFileReaderImpl	Implementation that can handle all hfile versions of HFile.Reader.
HFileReaderImpl.EncodedScanner	Scanner that operates on encoded data blocks.
HFileReaderImpl.HFileScannerImpl
HFileWriterImpl	Common functionality needed by all versions of HFile writers.
InclusiveCombinedBlockCache
LruBlockCache	A block cache implementation that is memory-aware using HeapSize, memory-bound using an LRU eviction algorithm, and concurrent: backed by a ConcurrentHashMap and with a non-blocking eviction thread giving constant-time LruBlockCache.cacheBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, org.apache.hadoop.hbase.io.hfile.Cacheable, boolean) and LruBlockCache.getBlock(org.apache.hadoop.hbase.io.hfile.BlockCacheKey, boolean, boolean, boolean) operations.
LruCachedBlock	Represents an entry in the LruBlockCache.
LruCachedBlockQueue	A memory-bound queue that will grow until an element brings total size >= maxSize.
MemcachedBlockCache	Class to store blocks into memcached.
NoOpDataBlockEncoder	Does not perform any kind of encoding/decoding.
PrefetchExecutor

Enum Summary

Enum	Description
BlockPriority
BlockType	Various types of HFile blocks.
BlockType.BlockCategory
Cacheable.MemoryType	SHARED means when this Cacheable is read back from cache it refers to the same memory area as used by the cache for caching it.

Exception Summary

Exception	Description
CorruptHFileException	This exception is thrown when attempts to read an HFile fail due to corruption or truncation issues.
HFileReaderImpl.BlockIndexNotLoadedException
HFileReaderImpl.NotSeekedException	An exception thrown when an operation requiring a scanner to be seeked is invoked on a scanner that is not seeked.
InvalidHFileException	Thrown when an invalid HFile format is detected

Package org.apache.hadoop.hbase.io.hfile Description

Provides implementations of HFile and HFile BlockCache. Caches are configured (and instantiated) by CacheConfig. See head of the CacheConfig class for constants that define cache options and configuration keys to use setting cache options. Cache implementations include the default, native on-heap LruBlockCache and a BucketCache that has a bunch of deploy formats including acting as a L2 for LruBlockCache -- when a block is evicted from LruBlockCache, it goes to the BucketCache and when we search a block, we look in both places -- or, the most common deploy type, using CombinedBlockCache, BucketCache is used as a host for data blocks with meta blocks in an instance of LruBlockCache. BucketCache can also be onheap, offheap, and file-backed.

Which BlockCache should I use?

By default LruBlockCache is on. If you would like to cache more, and offheap (offheap usually means less GC headache), try enabling * BucketCache. Fetching will always be slower when fetching from BucketCache but latencies tend to be less erratic over time (roughly because GC is less). See Nick Dimiduk's BlockCache 101 for some numbers.

Enabling BucketCache

See the HBase Reference Guide Enable BucketCache.