org.apache.orc

Class OrcFile.WriterOptions

java.lang.Object

org.apache.orc.OrcFile.WriterOptions

All Implemented Interfaces:

Cloneable

Enclosing class:

OrcFile

public static class OrcFile.WriterOptions
extends Object
implements Cloneable

Options for creating ORC file writers.

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WriterOptions(Properties tableProperties, org.apache.hadoop.conf.Configuration conf)

Method Summary

Modifier and Type	Method and Description
OrcFile.WriterOptions	blockPadding(boolean value) Sets whether the HDFS blocks are padded to prevent stripes from straddling blocks.
OrcFile.WriterOptions	blockSize(long value) Set the file system block size for the file.
OrcFile.WriterOptions	bloomFilterColumns(String columns) Comma separated values of column names for which bloom filter is to be created.
OrcFile.WriterOptions	bloomFilterFpp(double fpp) Specify the false positive probability for bloom filter.
OrcFile.WriterOptions	bloomFilterVersion(OrcFile.BloomFilterVersion version) Set the version of the bloom filters to write.
OrcFile.WriterOptions	bufferSize(int value) The size of the memory buffers used for compressing and storing the stripe in memory.
OrcFile.WriterOptions	callback(OrcFile.WriterCallback callback) Add a listener for when the stripe and file are about to be closed.
OrcFile.WriterOptions	clone()
OrcFile.WriterOptions	compress(CompressionKind value) Sets the generic compression that is used to compress the data.
OrcFile.WriterOptions	directEncodingColumns(String value) Set the comma-separated list of columns that should be direct encoded.
OrcFile.WriterOptions	encodingStrategy(OrcFile.EncodingStrategy strategy) Sets the encoding strategy that is used to encode the data.
OrcFile.WriterOptions	encrypt(String value) Encrypt a set of columns with a key.
OrcFile.WriterOptions	enforceBufferSize() Enforce writer to use requested buffer size instead of estimating buffer size based on stripe size and number of columns.
OrcFile.WriterOptions	fileSystem(org.apache.hadoop.fs.FileSystem value) Provide the filesystem for the path, if the client has it available.
boolean	getBlockPadding()
long	getBlockSize()
String	getBloomFilterColumns()
double	getBloomFilterFpp()
OrcFile.BloomFilterVersion	getBloomFilterVersion()
int	getBufferSize()
OrcFile.WriterCallback	getCallback()
CompressionKind	getCompress()
OrcFile.CompressionStrategy	getCompressionStrategy()
org.apache.hadoop.conf.Configuration	getConfiguration()
String	getDirectEncodingColumns()
OrcFile.EncodingStrategy	getEncodingStrategy()
String	getEncryption()
org.apache.hadoop.fs.FileSystem	getFileSystem()
org.apache.orc.impl.HadoopShims	getHadoopShims()
Map<String,org.apache.orc.impl.HadoopShims.KeyMetadata>	getKeyOverrides()
org.apache.orc.impl.KeyProvider	getKeyProvider()
String	getMasks()
org.apache.orc.MemoryManager	getMemoryManager()
boolean	getOverwrite()
double	getPaddingTolerance()
org.apache.orc.PhysicalWriter	getPhysicalWriter()
boolean	getProlepticGregorian()
int	getRowIndexStride()
org.apache.orc.TypeDescription	getSchema()
long	getStripeRowCountValue()
long	getStripeSize()
boolean	getUseUTCTimestamp()
OrcFile.Version	getVersion()
OrcFile.WriterVersion	getWriterVersion()
boolean	getWriteVariableLengthBlocks()
OrcFile.ZstdCompressOptions	getZstdCompressOptions()
boolean	isBuildIndex()
boolean	isEnforceBufferSize()
OrcFile.WriterOptions	masks(String value) Set the masks for the unencrypted data.
OrcFile.WriterOptions	memory(org.apache.orc.MemoryManager value) A public option to set the memory manager.
OrcFile.WriterOptions	overwrite(boolean value) If the output file already exists, should it be overwritten? If it is not provided, write operation will fail if the file already exists.
OrcFile.WriterOptions	paddingTolerance(double value) Sets the tolerance for block padding as a percentage of stripe size.
OrcFile.WriterOptions	physicalWriter(org.apache.orc.PhysicalWriter writer) Change the physical writer of the ORC file.
OrcFile.WriterOptions	rowIndexStride(int value) Set the distance between entries in the row index.
OrcFile.WriterOptions	setKeyProvider(org.apache.orc.impl.KeyProvider provider) Set the key provider for column encryption.
OrcFile.WriterOptions	setKeyVersion(String keyName, int version, org.apache.orc.EncryptionAlgorithm algorithm) For users that need to override the current version of a key, this method allows them to define the version and algorithm for a given key.
OrcFile.WriterOptions	setProlepticGregorian(boolean newValue) Should the writer use the proleptic Gregorian calendar for times and dates.
OrcFile.WriterOptions	setSchema(org.apache.orc.TypeDescription schema) Set the schema for the file.
OrcFile.WriterOptions	setShims(org.apache.orc.impl.HadoopShims value) Set the HadoopShims to use.
OrcFile.WriterOptions	stripeSize(long value) Set the stripe size for the file.
OrcFile.WriterOptions	useUTCTimestamp(boolean value) Manually set the time zone for the writer to utc.
OrcFile.WriterOptions	version(OrcFile.Version value) Sets the version of the file that will be written.
protected OrcFile.WriterOptions	writerVersion(OrcFile.WriterVersion version) Manually set the writer version.
OrcFile.WriterOptions	writeVariableLengthBlocks(boolean value) Should the ORC file writer use HDFS variable length blocks, if they are available?

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WriterOptions

protected WriterOptions(Properties tableProperties,
                        org.apache.hadoop.conf.Configuration conf)

Method Detail

clone

public OrcFile.WriterOptions clone()

Overrides:

clone in class Object

Returns:

a SHALLOW clone

fileSystem

public OrcFile.WriterOptions fileSystem(org.apache.hadoop.fs.FileSystem value)

Provide the filesystem for the path, if the client has it available. If it is not provided, it will be found from the path.

overwrite

public OrcFile.WriterOptions overwrite(boolean value)

If the output file already exists, should it be overwritten? If it is not provided, write operation will fail if the file already exists.

stripeSize

public OrcFile.WriterOptions stripeSize(long value)

Set the stripe size for the file. The writer stores the contents of the stripe in memory until this memory limit is reached and the stripe is flushed to the HDFS file and the next stripe started.

blockSize

public OrcFile.WriterOptions blockSize(long value)

Set the file system block size for the file. For optimal performance, set the block size to be multiple factors of stripe size.

rowIndexStride

public OrcFile.WriterOptions rowIndexStride(int value)

Set the distance between entries in the row index. The minimum value is 1000 to prevent the index from overwhelming the data. If the stride is set to 0, no indexes will be included in the file.

bufferSize

public OrcFile.WriterOptions bufferSize(int value)

The size of the memory buffers used for compressing and storing the stripe in memory. NOTE: ORC writer may choose to use smaller buffer size based on stripe size and number of columns for efficient stripe writing and memory utilization. To enforce writer to use the requested buffer size use enforceBufferSize().

enforceBufferSize

public OrcFile.WriterOptions enforceBufferSize()

Enforce writer to use requested buffer size instead of estimating buffer size based on stripe size and number of columns. See bufferSize() method for more info. Default: false

blockPadding

public OrcFile.WriterOptions blockPadding(boolean value)

Sets whether the HDFS blocks are padded to prevent stripes from straddling blocks. Padding improves locality and thus the speed of reading, but costs space.

encodingStrategy

public OrcFile.WriterOptions encodingStrategy(OrcFile.EncodingStrategy strategy)

Sets the encoding strategy that is used to encode the data.

paddingTolerance

public OrcFile.WriterOptions paddingTolerance(double value)

Sets the tolerance for block padding as a percentage of stripe size.

bloomFilterColumns

public OrcFile.WriterOptions bloomFilterColumns(String columns)

Comma separated values of column names for which bloom filter is to be created.

bloomFilterFpp

public OrcFile.WriterOptions bloomFilterFpp(double fpp)

Specify the false positive probability for bloom filter.

Parameters:

fpp - - false positive probability

Returns:

this

compress

public OrcFile.WriterOptions compress(CompressionKind value)

Sets the generic compression that is used to compress the data.

setSchema

public OrcFile.WriterOptions setSchema(org.apache.orc.TypeDescription schema)

Set the schema for the file. This is a required parameter.

Parameters:

schema - the schema for the file.

Returns:

this

version

public OrcFile.WriterOptions version(OrcFile.Version value)

Sets the version of the file that will be written.

callback

public OrcFile.WriterOptions callback(OrcFile.WriterCallback callback)

Add a listener for when the stripe and file are about to be closed.

Parameters:

callback - the object to be called when the stripe is closed

Returns:

this

bloomFilterVersion

public OrcFile.WriterOptions bloomFilterVersion(OrcFile.BloomFilterVersion version)

Set the version of the bloom filters to write.

physicalWriter

public OrcFile.WriterOptions physicalWriter(org.apache.orc.PhysicalWriter writer)

Change the physical writer of the ORC file.

SHOULD ONLY BE USED BY LLAP.

Parameters:

writer - the writer to control the layout and persistence

Returns:

this

memory

public OrcFile.WriterOptions memory(org.apache.orc.MemoryManager value)

A public option to set the memory manager.

writeVariableLengthBlocks

public OrcFile.WriterOptions writeVariableLengthBlocks(boolean value)

Should the ORC file writer use HDFS variable length blocks, if they are available?

Parameters:

value - the new value

Returns:

this

setShims

public OrcFile.WriterOptions setShims(org.apache.orc.impl.HadoopShims value)

Set the HadoopShims to use. This is only for testing.

Parameters:

value - the new value

Returns:

this

writerVersion

protected OrcFile.WriterOptions writerVersion(OrcFile.WriterVersion version)

Manually set the writer version. This is an internal API.

Parameters:

version - the version to write

Returns:

this

useUTCTimestamp

public OrcFile.WriterOptions useUTCTimestamp(boolean value)

Manually set the time zone for the writer to utc. If not defined, system time zone is assumed.

directEncodingColumns

public OrcFile.WriterOptions directEncodingColumns(String value)

Set the comma-separated list of columns that should be direct encoded.

Parameters:

value - the value to set

Returns:

this

encrypt

public OrcFile.WriterOptions encrypt(String value)

Encrypt a set of columns with a key.

Format of the string is a key-list.

• key-list = key (';' key-list)?
• key = key-name ':' field-list
• field-list = field-name ( ',' field-list )?
• field-name = number | field-part ('.' field-name)?
• field-part = quoted string | simple name

Parameters:

value - a key-list of which columns to encrypt

Returns:

this

masks

public OrcFile.WriterOptions masks(String value)

Set the masks for the unencrypted data.

Format of the string is a mask-list.

• mask-list = mask (';' mask-list)?
• mask = mask-name (',' parameter)* ':' field-list
• field-list = field-name ( ',' field-list )?
• field-name = number | field-part ('.' field-name)?
• field-part = quoted string | simple name

Parameters:

value - a list of the masks and column names

Returns:

this

setKeyVersion

public OrcFile.WriterOptions setKeyVersion(String keyName,
                                           int version,
                                           org.apache.orc.EncryptionAlgorithm algorithm)

For users that need to override the current version of a key, this method allows them to define the version and algorithm for a given key.

This will mostly be used for ORC file merging where the writer has to use the same version of the key that the original files used.

Parameters:

keyName - the key name

version - the version of the key to use

algorithm - the algorithm for the given key version

Returns:

this

setKeyProvider

public OrcFile.WriterOptions setKeyProvider(org.apache.orc.impl.KeyProvider provider)

Set the key provider for column encryption.

Parameters:

provider - the object that holds the master secrets

Returns:

this

setProlepticGregorian

public OrcFile.WriterOptions setProlepticGregorian(boolean newValue)

Should the writer use the proleptic Gregorian calendar for times and dates.

Parameters:

newValue - true if we should use the proleptic calendar

Returns:

this

getKeyProvider

public org.apache.orc.impl.KeyProvider getKeyProvider()

getBlockPadding

public boolean getBlockPadding()

getBlockSize

public long getBlockSize()

getBloomFilterColumns

public String getBloomFilterColumns()

getOverwrite

public boolean getOverwrite()

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem()

getConfiguration

public org.apache.hadoop.conf.Configuration getConfiguration()

getSchema

public org.apache.orc.TypeDescription getSchema()

getStripeSize

public long getStripeSize()

getStripeRowCountValue

public long getStripeRowCountValue()

getCompress

public CompressionKind getCompress()

getCallback

public OrcFile.WriterCallback getCallback()

getVersion

public OrcFile.Version getVersion()

getMemoryManager

public org.apache.orc.MemoryManager getMemoryManager()

getBufferSize

public int getBufferSize()

isEnforceBufferSize

public boolean isEnforceBufferSize()

getRowIndexStride

public int getRowIndexStride()

isBuildIndex

public boolean isBuildIndex()

getCompressionStrategy

public OrcFile.CompressionStrategy getCompressionStrategy()

getEncodingStrategy

public OrcFile.EncodingStrategy getEncodingStrategy()

getZstdCompressOptions

public OrcFile.ZstdCompressOptions getZstdCompressOptions()

getPaddingTolerance

public double getPaddingTolerance()

getBloomFilterFpp

public double getBloomFilterFpp()

getBloomFilterVersion

public OrcFile.BloomFilterVersion getBloomFilterVersion()

getPhysicalWriter

public org.apache.orc.PhysicalWriter getPhysicalWriter()

getWriterVersion

public OrcFile.WriterVersion getWriterVersion()

getWriteVariableLengthBlocks

public boolean getWriteVariableLengthBlocks()

getHadoopShims

public org.apache.orc.impl.HadoopShims getHadoopShims()

getUseUTCTimestamp

public boolean getUseUTCTimestamp()

getDirectEncodingColumns

public String getDirectEncodingColumns()

getEncryption

public String getEncryption()

getMasks

public String getMasks()

getKeyOverrides

public Map<String,org.apache.orc.impl.HadoopShims.KeyMetadata> getKeyOverrides()

getProlepticGregorian

public boolean getProlepticGregorian()