org.apache.orc.impl

Class PhysicalFsWriter

java.lang.Object

org.apache.orc.impl.PhysicalFsWriter

All Implemented Interfaces:

org.apache.orc.PhysicalWriter

public class PhysicalFsWriter
extends Object
implements org.apache.orc.PhysicalWriter

An orc PhysicalFsWriter.

NOTE: The file was copied and modified to support zstd-jni. This feature is only supported in ORC 2.0, but 2.0 only supports JDK17. We need to support JDK8.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	PhysicalFsWriter.VariantTracker Record the information about each column encryption variant.

Nested classes/interfaces inherited from interface org.apache.orc.PhysicalWriter

org.apache.orc.PhysicalWriter.OutputReceiver

Constructor Summary

Constructors

Constructor and Description
PhysicalFsWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, OrcFile.WriterOptions opts)
PhysicalFsWriter(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, OrcFile.WriterOptions opts, org.apache.orc.impl.writer.WriterEncryptionVariant[] encryption)
PhysicalFsWriter(org.apache.hadoop.fs.FSDataOutputStream outputStream, OrcFile.WriterOptions opts, org.apache.orc.impl.writer.WriterEncryptionVariant[] encryption)

Method Summary

Modifier and Type	Method and Description
void	appendRawStripe(ByteBuffer buffer, org.apache.orc.OrcProto.StripeInformation.Builder dirEntry)
void	close()
org.apache.orc.impl.PhysicalFsWriter.BufferedStream	createDataStream(org.apache.orc.impl.StreamName name)
protected OutputStream	createIndexStream(org.apache.orc.impl.StreamName name)
void	finalizeStripe(org.apache.orc.OrcProto.StripeFooter.Builder footerBuilder, org.apache.orc.OrcProto.StripeInformation.Builder dirEntry)
void	flush()
long	getFileBytes(int column, org.apache.orc.impl.writer.WriterEncryptionVariant variant) Get the number of bytes for a file in a given column by finding all the streams (not suppressed) for a given column and returning the sum of their sizes.
org.apache.orc.impl.writer.StreamOptions	getStreamOptions()
String	toString()
void	writeBloomFilter(org.apache.orc.impl.StreamName name, org.apache.orc.OrcProto.BloomFilterIndex.Builder bloom)
void	writeFileFooter(org.apache.orc.OrcProto.Footer.Builder builder)
void	writeFileMetadata(org.apache.orc.OrcProto.Metadata.Builder builder)
void	writeHeader()
void	writeIndex(org.apache.orc.impl.StreamName name, org.apache.orc.OrcProto.RowIndex.Builder index)
long	writePostScript(org.apache.orc.OrcProto.PostScript.Builder builder)
void	writeStatistics(org.apache.orc.impl.StreamName name, org.apache.orc.OrcProto.ColumnStatistics.Builder statistics)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.orc.PhysicalWriter

getCompressionCodec, getFileBytes, writeBloomFilter, writeIndex

Constructor Detail

PhysicalFsWriter

public PhysicalFsWriter(org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.fs.Path path,
                        OrcFile.WriterOptions opts)
                 throws IOException

Throws:

IOException

PhysicalFsWriter

public PhysicalFsWriter(org.apache.hadoop.fs.FileSystem fs,
                        org.apache.hadoop.fs.Path path,
                        OrcFile.WriterOptions opts,
                        org.apache.orc.impl.writer.WriterEncryptionVariant[] encryption)
                 throws IOException

Throws:

IOException

PhysicalFsWriter

public PhysicalFsWriter(org.apache.hadoop.fs.FSDataOutputStream outputStream,
                        OrcFile.WriterOptions opts,
                        org.apache.orc.impl.writer.WriterEncryptionVariant[] encryption)
                 throws IOException

Throws:

IOException

Method Detail

getFileBytes

public long getFileBytes(int column,
                         org.apache.orc.impl.writer.WriterEncryptionVariant variant)

Get the number of bytes for a file in a given column by finding all the streams (not suppressed) for a given column and returning the sum of their sizes. excludes index

Parameters:

column - column from which to get file size

Returns:

number of bytes for the given column

getStreamOptions

public org.apache.orc.impl.writer.StreamOptions getStreamOptions()

writeFileMetadata

public void writeFileMetadata(org.apache.orc.OrcProto.Metadata.Builder builder)
                       throws IOException

Specified by:

writeFileMetadata in interface org.apache.orc.PhysicalWriter

Throws:

IOException

writeFileFooter

public void writeFileFooter(org.apache.orc.OrcProto.Footer.Builder builder)
                     throws IOException

Specified by:

writeFileFooter in interface org.apache.orc.PhysicalWriter

Throws:

IOException

writePostScript

public long writePostScript(org.apache.orc.OrcProto.PostScript.Builder builder)
                     throws IOException

Specified by:

writePostScript in interface org.apache.orc.PhysicalWriter

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface org.apache.orc.PhysicalWriter

Throws:

IOException

flush

public void flush()
           throws IOException

Specified by:

flush in interface org.apache.orc.PhysicalWriter

Throws:

IOException

appendRawStripe

public void appendRawStripe(ByteBuffer buffer,
                            org.apache.orc.OrcProto.StripeInformation.Builder dirEntry)
                     throws IOException

Specified by:

appendRawStripe in interface org.apache.orc.PhysicalWriter

Throws:

IOException

finalizeStripe

public void finalizeStripe(org.apache.orc.OrcProto.StripeFooter.Builder footerBuilder,
                           org.apache.orc.OrcProto.StripeInformation.Builder dirEntry)
                    throws IOException

Specified by:

finalizeStripe in interface org.apache.orc.PhysicalWriter

Throws:

IOException

writeHeader

public void writeHeader()
                 throws IOException

Specified by:

writeHeader in interface org.apache.orc.PhysicalWriter

Throws:

IOException

createDataStream

public org.apache.orc.impl.PhysicalFsWriter.BufferedStream createDataStream(org.apache.orc.impl.StreamName name)

Specified by:

createDataStream in interface org.apache.orc.PhysicalWriter

createIndexStream

protected OutputStream createIndexStream(org.apache.orc.impl.StreamName name)

writeIndex

public void writeIndex(org.apache.orc.impl.StreamName name,
                       org.apache.orc.OrcProto.RowIndex.Builder index)
                throws IOException

Throws:

IOException

writeBloomFilter

public void writeBloomFilter(org.apache.orc.impl.StreamName name,
                             org.apache.orc.OrcProto.BloomFilterIndex.Builder bloom)
                      throws IOException

Throws:

IOException

writeStatistics

public void writeStatistics(org.apache.orc.impl.StreamName name,
                            org.apache.orc.OrcProto.ColumnStatistics.Builder statistics)

toString

public String toString()

Overrides:

toString in class Object