org.apache.hadoop.hbase.regionserver

Class StoreScanner

java.lang.Object

org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner

org.apache.hadoop.hbase.regionserver.NonReversedNonLazyKeyValueScanner

org.apache.hadoop.hbase.regionserver.StoreScanner

All Implemented Interfaces:

Closeable, AutoCloseable, ChangedReadersObserver, InternalScanner, KeyValueScanner, Shipper

Direct Known Subclasses:

MobStoreScanner, ReversedStoreScanner

@InterfaceAudience.Private
public class StoreScanner
extends NonReversedNonLazyKeyValueScanner
implements KeyValueScanner, InternalScanner, ChangedReadersObserver

Scanner scans both the memstore and the Store. Coalesce KeyValue stream into List<KeyValue> for a single row.

The implementation is not thread safe. So there will be no race between next and close. The only exception is updateReaders, it will be called in the memstore flush thread to indicate that there is a flush.

Field Summary

Fields

Modifier and Type	Field and Description
static long	DEFAULT_HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK Default value of HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK.
static String	HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK The number of cells scanned in between timeout checks.
protected KeyValueHeap	heap
protected long	readPt
protected HStore	store
static String	STORESCANNER_PARALLEL_SEEK_ENABLE
static String	STORESCANNER_PREAD_MAX_BYTES If the read type if Scan.ReadType.DEFAULT, we will start with pread, and if the kvs we scanned reaches this limit, we will reopen the scanner with stream.

Fields inherited from interface org.apache.hadoop.hbase.regionserver.KeyValueScanner

NO_NEXT_INDEXED_KEY

Constructor Summary

Constructors

Constructor and Description
StoreScanner(HStore store, ScanInfo scanInfo, List<? extends KeyValueScanner> scanners, long smallestReadPoint, long earliestPutTs, byte[] dropDeletesFromRow, byte[] dropDeletesToRow) Used for compactions that drop deletes from a limited range of rows.
StoreScanner(HStore store, ScanInfo scanInfo, List<? extends KeyValueScanner> scanners, ScanType scanType, long smallestReadPoint, long earliestPutTs) Used for store file compaction and memstore compaction.
StoreScanner(HStore store, ScanInfo scanInfo, Scan scan, NavigableSet<byte[]> columns, long readPt) Opens a scanner across memstore, snapshot, and all StoreFiles.
StoreScanner(ScanInfo scanInfo, ScanType scanType, List<? extends KeyValueScanner> scanners)

Method Summary

Modifier and Type	Method and Description
protected boolean	checkFlushed()
protected void	checkScanOrder(Cell prevKV, Cell kv, CellComparator comparator) Check whether scan as expected order
void	close() Close the KeyValue scanner.
long	getEstimatedNumberOfKvsScanned()
Cell	getNextIndexedKey()
long	getReadPoint()
protected KeyValueHeap	newKVHeap(List<? extends KeyValueScanner> scanners, CellComparator comparator)
KeyValue	next() Return the next Cell in this scanner, iterating the scanner
boolean	next(List<Cell> outResult, ScannerContext scannerContext) Get the next row of values from this Store.
Cell	peek() Look at the next Cell in this scanner, but do not iterate scanner.
protected boolean	reopenAfterFlush()
boolean	reseek(Cell kv) Reseek the scanner at or after the specified KeyValue.
protected void	resetKVHeap(List<? extends KeyValueScanner> scanners, CellComparator comparator)
boolean	seek(Cell key) Seek the scanner at or after the specified KeyValue.
protected boolean	seekAsDirection(Cell kv) Do a reseek in a normal StoreScanner(scan forward)
protected void	seekScanners(List<? extends KeyValueScanner> scanners, Cell seekKey, boolean isLazy, boolean isParallelSeek) Seek the specified scanners with the given key
protected boolean	seekToNextRow(Cell c)
protected List<KeyValueScanner>	selectScannersFrom(HStore store, List<? extends KeyValueScanner> allScanners) Filters the given list of scanners using Bloom filter, time range, and TTL.
void	shipped() Called after a batch of rows scanned and set to be returned to client.
protected boolean	trySkipToNextColumn(Cell cell) See trySkipToNextRow(Cell)
protected boolean	trySkipToNextRow(Cell cell) See if we should actually SEEK or rather just SKIP to the next Cell (see HBASE-13109).
void	updateReaders(List<HStoreFile> sfs, List<KeyValueScanner> memStoreScanners) Notify observers.

Methods inherited from class org.apache.hadoop.hbase.regionserver.NonReversedNonLazyKeyValueScanner

backwardSeek, seekToLastRow, seekToPreviousRow

Methods inherited from class org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner

doRealSeek, enforceSeek, getFilePath, isFileScanner, realSeekDone, requestSeek, shouldUseScanner

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.hadoop.hbase.regionserver.KeyValueScanner

backwardSeek, enforceSeek, getFilePath, getScannerOrder, isFileScanner, realSeekDone, requestSeek, seekToLastRow, seekToPreviousRow, shouldUseScanner

Methods inherited from interface org.apache.hadoop.hbase.regionserver.InternalScanner

next

Field Detail

store

protected final HStore store

heap

protected KeyValueHeap heap

STORESCANNER_PARALLEL_SEEK_ENABLE

public static final String STORESCANNER_PARALLEL_SEEK_ENABLE

See Also:

Constant Field Values

HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK

public static final String HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK

The number of cells scanned in between timeout checks. Specifying a larger value means that timeout checks will occur less frequently. Specifying a small value will lead to more frequent timeout checks.

See Also:

Constant Field Values

DEFAULT_HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK

public static final long DEFAULT_HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK

Default value of HBASE_CELLS_SCANNED_PER_HEARTBEAT_CHECK.

See Also:

Constant Field Values

STORESCANNER_PREAD_MAX_BYTES

public static final String STORESCANNER_PREAD_MAX_BYTES

If the read type if Scan.ReadType.DEFAULT, we will start with pread, and if the kvs we scanned reaches this limit, we will reopen the scanner with stream. The default value is 4 times of block size for this store.

See Also:

Constant Field Values

readPt

protected final long readPt

Constructor Detail

StoreScanner

public StoreScanner(HStore store,
                    ScanInfo scanInfo,
                    Scan scan,
                    NavigableSet<byte[]> columns,
                    long readPt)
             throws IOException

Opens a scanner across memstore, snapshot, and all StoreFiles. Assumes we are not in a compaction.

Parameters:

store - who we scan

scan - the spec

columns - which columns we are scanning

Throws:

IOException

StoreScanner

public StoreScanner(HStore store,
                    ScanInfo scanInfo,
                    List<? extends KeyValueScanner> scanners,
                    ScanType scanType,
                    long smallestReadPoint,
                    long earliestPutTs)
             throws IOException

Used for store file compaction and memstore compaction.

Opens a scanner across specified StoreFiles/MemStoreSegments.

Parameters:

store - who we scan

scanners - ancillary scanners

smallestReadPoint - the readPoint that we should use for tracking versions

Throws:

IOException

StoreScanner

public StoreScanner(HStore store,
                    ScanInfo scanInfo,
                    List<? extends KeyValueScanner> scanners,
                    long smallestReadPoint,
                    long earliestPutTs,
                    byte[] dropDeletesFromRow,
                    byte[] dropDeletesToRow)
             throws IOException

Used for compactions that drop deletes from a limited range of rows.

Opens a scanner across specified StoreFiles.

Parameters:

store - who we scan

scanners - ancillary scanners

smallestReadPoint - the readPoint that we should use for tracking versions

dropDeletesFromRow - The inclusive left bound of the range; can be EMPTY_START_ROW.

dropDeletesToRow - The exclusive right bound of the range; can be EMPTY_END_ROW.

Throws:

IOException

StoreScanner

public StoreScanner(ScanInfo scanInfo,
                    ScanType scanType,
                    List<? extends KeyValueScanner> scanners)
             throws IOException

Throws:

IOException

Method Detail

seekScanners

protected void seekScanners(List<? extends KeyValueScanner> scanners,
                            Cell seekKey,
                            boolean isLazy,
                            boolean isParallelSeek)
                     throws IOException

Seek the specified scanners with the given key

Parameters:

scanners -

seekKey -

isLazy - true if using lazy seek

isParallelSeek - true if using parallel seek

Throws:

IOException

resetKVHeap

protected void resetKVHeap(List<? extends KeyValueScanner> scanners,
                           CellComparator comparator)
                    throws IOException

Throws:

IOException

newKVHeap

protected KeyValueHeap newKVHeap(List<? extends KeyValueScanner> scanners,
                                 CellComparator comparator)
                          throws IOException

Throws:

IOException

selectScannersFrom

protected List<KeyValueScanner> selectScannersFrom(HStore store,
                                                   List<? extends KeyValueScanner> allScanners)

Filters the given list of scanners using Bloom filter, time range, and TTL.

Will be overridden by testcase so declared as protected.

peek

public Cell peek()

Description copied from interface: KeyValueScanner

Look at the next Cell in this scanner, but do not iterate scanner. NOTICE: The returned cell has not been passed into ScanQueryMatcher. So it may not be what the user need.

Specified by:

peek in interface KeyValueScanner

Returns:

the next Cell

next

public KeyValue next()

Description copied from interface: KeyValueScanner

Return the next Cell in this scanner, iterating the scanner

Specified by:

next in interface KeyValueScanner

Returns:

the next Cell

close

public void close()

Description copied from interface: KeyValueScanner

Close the KeyValue scanner.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface InternalScanner

Specified by:

close in interface KeyValueScanner

seek

public boolean seek(Cell key)
             throws IOException

Description copied from interface: KeyValueScanner

Seek the scanner at or after the specified KeyValue.

Specified by:

seek in interface KeyValueScanner

Parameters:

key - seek value

Returns:

true if scanner has values left, false if end of scanner

Throws:

IOException

next

public boolean next(List<Cell> outResult,
                    ScannerContext scannerContext)
             throws IOException

Get the next row of values from this Store.

Specified by:

next in interface InternalScanner

Parameters:

outResult -

scannerContext -

Returns:

true if there are more rows, false if scanner is done

Throws:

IOException - e

trySkipToNextRow

protected boolean trySkipToNextRow(Cell cell)
                            throws IOException

See if we should actually SEEK or rather just SKIP to the next Cell (see HBASE-13109). ScanQueryMatcher may issue SEEK hints, such as seek to next column, next row, or seek to an arbitrary seek key. This method decides whether a seek is the most efficient _actual_ way to get us to the requested cell (SEEKs are more expensive than SKIP, SKIP, SKIP inside the current, loaded block). It does this by looking at the next indexed key of the current HFile. This key is then compared with the _SEEK_ key, where a SEEK key is an artificial 'last possible key on the row' (only in here, we avoid actually creating a SEEK key; in the compare we work with the current Cell but compare as though it were a seek key; see down in matcher.compareKeyForNextRow, etc). If the compare gets us onto the next block we *_SEEK, otherwise we just SKIP to the next requested cell.

Other notes:

• Rows can straddle block boundaries
• Versions of columns can straddle block boundaries (i.e. column C1 at T1 might be in a different block than column C1 at T2)
• We want to SKIP if the chance is high that we'll find the desired Cell after a few SKIPs...
• We want to SEEK when the chance is high that we'll be able to seek past many Cells, especially if we know we need to go to the next block.

A good proxy (best effort) to determine whether SKIP is better than SEEK is whether we'll likely end up seeking to the next block (or past the next block) to get our next column. Example:

 |    BLOCK 1              |     BLOCK 2                   |
 |  r1/c1, r1/c2, r1/c3    |    r1/c4, r1/c5, r2/c1        |
                                   ^         ^
                                   |         |
                           Next Index Key   SEEK_NEXT_ROW (before r2/c1)


 |    BLOCK 1                       |     BLOCK 2                      |
 |  r1/c1/t5, r1/c1/t4, r1/c1/t3    |    r1/c1/t2, r1/c1/T1, r1/c2/T3  |
                                            ^              ^
                                            |              |
                                    Next Index Key        SEEK_NEXT_COL
 

Now imagine we want columns c1 and c3 (see first diagram above), the 'Next Index Key' of r1/c4 is > r1/c3 so we should seek to get to the c1 on the next row, r2. In second case, say we only want one version of c1, after we have it, a SEEK_COL will be issued to get to c2. Looking at the 'Next Index Key', it would land us in the next block, so we should SEEK. In other scenarios where the SEEK will not land us in the next block, it is very likely better to issues a series of SKIPs.

Parameters:

cell - current cell

Returns:

true means skip to next row, false means not

Throws:

IOException

trySkipToNextColumn

protected boolean trySkipToNextColumn(Cell cell)
                               throws IOException

See trySkipToNextRow(Cell)

Parameters:

cell - current cell

Returns:

true means skip to next column, false means not

Throws:

IOException

getReadPoint

public long getReadPoint()

Specified by:

getReadPoint in interface ChangedReadersObserver

Returns:

the read point of the current scan

updateReaders

public void updateReaders(List<HStoreFile> sfs,
                          List<KeyValueScanner> memStoreScanners)
                   throws IOException

Description copied from interface: ChangedReadersObserver

Notify observers.

Specified by:

updateReaders in interface ChangedReadersObserver

Parameters:

sfs - The new files

memStoreScanners - scanner of current memstore

Throws:

IOException - e

reopenAfterFlush

protected final boolean reopenAfterFlush()
                                  throws IOException

Returns:

if top of heap has changed (and KeyValueHeap has to try the next KV)

Throws:

IOException

checkScanOrder

protected void checkScanOrder(Cell prevKV,
                              Cell kv,
                              CellComparator comparator)
                       throws IOException

Check whether scan as expected order

Parameters:

prevKV -

kv -

comparator -

Throws:

IOException

seekToNextRow

protected boolean seekToNextRow(Cell c)
                         throws IOException

Throws:

IOException

seekAsDirection

protected boolean seekAsDirection(Cell kv)
                           throws IOException

Do a reseek in a normal StoreScanner(scan forward)

Parameters:

kv -

Returns:

true if scanner has values left, false if end of scanner

Throws:

IOException

reseek

public boolean reseek(Cell kv)
               throws IOException

Description copied from interface: KeyValueScanner

Reseek the scanner at or after the specified KeyValue. This method is guaranteed to seek at or after the required key only if the key comes after the current position of the scanner. Should not be used to seek to a key which may come before the current position.

Specified by:

reseek in interface KeyValueScanner

Parameters:

kv - seek value (should be non-null)

Returns:

true if scanner has values left, false if end of scanner

Throws:

IOException

checkFlushed

protected final boolean checkFlushed()

getEstimatedNumberOfKvsScanned

public long getEstimatedNumberOfKvsScanned()

Returns:

The estimated number of KVs seen by this scanner (includes some skipped KVs).

getNextIndexedKey

public Cell getNextIndexedKey()

Specified by:

getNextIndexedKey in interface KeyValueScanner

Overrides:

getNextIndexedKey in class NonLazyKeyValueScanner

Returns:

the next key in the index, usually the first key of next block OR a key that falls between last key of current block and first key of next block.. see HFileWriterImpl#getMidpoint, or null if not known.

shipped

public void shipped()
             throws IOException

Description copied from interface: Shipper

Called after a batch of rows scanned and set to be returned to client. Any in between cleanup can be done here.

Specified by:

shipped in interface Shipper

Overrides:

shipped in class NonLazyKeyValueScanner

Throws:

IOException