org.apache.orc.impl

Class WriterImpl

java.lang.Object

org.apache.orc.impl.WriterImpl

All Implemented Interfaces:

Closeable, AutoCloseable, org.apache.orc.impl.WriterInternal, org.apache.orc.MemoryManager.Callback, org.apache.orc.Writer

public class WriterImpl
extends Object
implements org.apache.orc.impl.WriterInternal, org.apache.orc.MemoryManager.Callback

An ORC file writer. The file is divided into stripes, which is the natural unit of work when reading. Each stripe is buffered in memory until the memory reaches the stripe size and then it is written out broken down by columns. Each column is written by a TreeWriter that is specific to that type of column. TreeWriters may have children TreeWriters that handle the sub-types. Each of the TreeWriters writes the column's data as a set of streams.

This class is unsynchronized like most Stream objects, so from the creation of an OrcFile and all access to a single instance has to be from a single thread.

There are no known cases where these happen between different threads today.

Caveat: the MemoryManager is created during WriterOptions create, that has to be confined to a single thread as well.

NOTE: The file was copied and modified to support zstd-jni. This feature is only supported in ORC 2.0, but 2.0 only supports JDK17. We need to support JDK8.

Constructor Summary

Constructors

Constructor and Description
WriterImpl(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, OrcFile.WriterOptions opts)

Method Summary

Modifier and Type	Method and Description
void	addRowBatch(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch batch)
void	addUserMetadata(String name, ByteBuffer value)
void	appendStripe(byte[] stripe, int offset, int length, org.apache.orc.StripeInformation stripeInfo, org.apache.orc.OrcProto.StripeStatistics stripeStatistics)
void	appendStripe(byte[] stripe, int offset, int length, org.apache.orc.StripeInformation stripeInfo, org.apache.orc.StripeStatistics[] stripeStatistics)
void	appendUserMetadata(List<org.apache.orc.OrcProto.UserMetadataItem> userMetadata)
boolean	checkMemory(double newScale)
void	close()
static org.apache.orc.CompressionCodec	createCodec(CompressionKind kind)
long	estimateMemory()
org.apache.orc.CompressionCodec	getCompressionCodec()
static int	getEstimatedBufferSize(long stripeSize, int numColumns, int bs)
long	getNumberOfRows() Row count gets updated when flushing the stripes.
long	getRawDataSize() Raw data size will be compute when writing the file footer.
org.apache.orc.TypeDescription	getSchema()
org.apache.orc.ColumnStatistics[]	getStatistics()
List<org.apache.orc.StripeInformation>	getStripes()
void	increaseCompressionSize(int newSize)
long	writeIntermediateFooter()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

WriterImpl

public WriterImpl(org.apache.hadoop.fs.FileSystem fs,
                  org.apache.hadoop.fs.Path path,
                  OrcFile.WriterOptions opts)
           throws IOException

Throws:

IOException

Method Detail

getEstimatedBufferSize

public static int getEstimatedBufferSize(long stripeSize,
                                         int numColumns,
                                         int bs)

increaseCompressionSize

public void increaseCompressionSize(int newSize)

Specified by:

increaseCompressionSize in interface org.apache.orc.impl.WriterInternal

createCodec

public static org.apache.orc.CompressionCodec createCodec(CompressionKind kind)

checkMemory

public boolean checkMemory(double newScale)
                    throws IOException

Specified by:

checkMemory in interface org.apache.orc.MemoryManager.Callback

Throws:

IOException

getSchema

public org.apache.orc.TypeDescription getSchema()

Specified by:

getSchema in interface org.apache.orc.Writer

addUserMetadata

public void addUserMetadata(String name,
                            ByteBuffer value)

Specified by:

addUserMetadata in interface org.apache.orc.Writer

addRowBatch

public void addRowBatch(org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch batch)
                 throws IOException

Specified by:

addRowBatch in interface org.apache.orc.Writer

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface org.apache.orc.Writer

Throws:

IOException

getRawDataSize

public long getRawDataSize()

Raw data size will be compute when writing the file footer. Hence raw data size value will be available only after closing the writer.

Specified by:

getRawDataSize in interface org.apache.orc.Writer

getNumberOfRows

public long getNumberOfRows()

Row count gets updated when flushing the stripes. To get accurate row count call this method after writer is closed.

Specified by:

getNumberOfRows in interface org.apache.orc.Writer

writeIntermediateFooter

public long writeIntermediateFooter()
                             throws IOException

Specified by:

writeIntermediateFooter in interface org.apache.orc.Writer

Throws:

IOException

appendStripe

public void appendStripe(byte[] stripe,
                         int offset,
                         int length,
                         org.apache.orc.StripeInformation stripeInfo,
                         org.apache.orc.OrcProto.StripeStatistics stripeStatistics)
                  throws IOException

Specified by:

appendStripe in interface org.apache.orc.Writer

Throws:

IOException

appendStripe

public void appendStripe(byte[] stripe,
                         int offset,
                         int length,
                         org.apache.orc.StripeInformation stripeInfo,
                         org.apache.orc.StripeStatistics[] stripeStatistics)
                  throws IOException

Throws:

IOException

appendUserMetadata

public void appendUserMetadata(List<org.apache.orc.OrcProto.UserMetadataItem> userMetadata)

Specified by:

appendUserMetadata in interface org.apache.orc.Writer

getStatistics

public org.apache.orc.ColumnStatistics[] getStatistics()

Specified by:

getStatistics in interface org.apache.orc.Writer

getStripes

public List<org.apache.orc.StripeInformation> getStripes()
                                                  throws IOException

Throws:

IOException

getCompressionCodec

public org.apache.orc.CompressionCodec getCompressionCodec()

estimateMemory

public long estimateMemory()