org.apache.paimon.s3

Class HadoopCompliantFileIO

java.lang.Object

org.apache.paimon.s3.HadoopCompliantFileIO

All Implemented Interfaces:

Closeable, Serializable, AutoCloseable, FileIO

Direct Known Subclasses:

S3FileIO

public abstract class HadoopCompliantFileIO
extends Object
implements FileIO

Hadoop FileIO.

Important: copy this class from HadoopFileIO here to avoid class loader conflicts.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<String,org.apache.hadoop.fs.FileSystem>	fsMap

Fields inherited from interface org.apache.paimon.fs.FileIO

LOG

Constructor Summary

Constructors

Constructor and Description
HadoopCompliantFileIO()

Method Summary

Modifier and Type	Method and Description
protected abstract org.apache.hadoop.fs.FileSystem	createFileSystem(org.apache.hadoop.fs.Path path)
boolean	delete(Path path, boolean recursive) Delete a file.
boolean	exists(Path path) Check if exists.
FileStatus	getFileStatus(Path path) Return a file status object that represents the path.
RemoteIterator<FileStatus>	listFilesIterative(Path path, boolean recursive) List the statuses of the files iteratively in the given path if the path is a directory.
FileStatus[]	listStatus(Path path) List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(Path path) Make the given file and all non-existent parents into directories.
SeekableInputStream	newInputStream(Path path) Opens an SeekableInputStream at the indicated Path.
PositionOutputStream	newOutputStream(Path path, boolean overwrite) Opens an PositionOutputStream at the indicated Path.
boolean	rename(Path src, Path dst) Renames the file/directory src to dst.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.fs.FileIO

checkOrMkdirs, close, configure, copyFile, copyFiles, deleteDirectoryQuietly, deleteFilesQuietly, deleteQuietly, discoverLoaders, get, getFileSize, isDir, isObjectStore, listDirectories, listFiles, overwriteFileUtf8, overwriteHintFile, readFileUtf8, readOverwrittenFileUtf8, tryToWriteAtomic, writeFile

Field Detail

fsMap

protected transient volatile Map<String,org.apache.hadoop.fs.FileSystem> fsMap

Constructor Detail

HadoopCompliantFileIO

public HadoopCompliantFileIO()

Method Detail

newInputStream

public SeekableInputStream newInputStream(Path path)
                                   throws IOException

Description copied from interface: FileIO

Opens an SeekableInputStream at the indicated Path.

Specified by:

newInputStream in interface FileIO

Parameters:

path - the file to open

Throws:

IOException

newOutputStream

public PositionOutputStream newOutputStream(Path path,
                                            boolean overwrite)
                                     throws IOException

Description copied from interface: FileIO

Opens an PositionOutputStream at the indicated Path.

Specified by:

newOutputStream in interface FileIO

Parameters:

path - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

Throws:

IOException - Thrown, if the stream could not be opened because of an I/O, or because a file already exists at that path and the write mode indicates to not overwrite the file.

getFileStatus

public FileStatus getFileStatus(Path path)
                         throws IOException

Description copied from interface: FileIO

Return a file status object that represents the path.

Specified by:

getFileStatus in interface FileIO

Parameters:

path - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

listStatus

public FileStatus[] listStatus(Path path)
                        throws IOException

Description copied from interface: FileIO

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in interface FileIO

Parameters:

path - given path

Returns:

the statuses of the files/directories in the given path

Throws:

IOException

listFilesIterative

public RemoteIterator<FileStatus> listFilesIterative(Path path,
                                                     boolean recursive)
                                              throws IOException

Description copied from interface: FileIO

List the statuses of the files iteratively in the given path if the path is a directory.

Specified by:

listFilesIterative in interface FileIO

Parameters:

path - given path

recursive - if set to true will recursively list files in subdirectories, otherwise only files in the current directory will be listed

Returns:

an RemoteIterator over FileStatus of the files in the given path

Throws:

IOException

exists

public boolean exists(Path path)
               throws IOException

Description copied from interface: FileIO

Check if exists.

Specified by:

exists in interface FileIO

Parameters:

path - source file

Throws:

IOException

delete

public boolean delete(Path path,
                      boolean recursive)
               throws IOException

Description copied from interface: FileIO

Delete a file.

Specified by:

delete in interface FileIO

Parameters:

path - the path to delete

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false

Returns:

true if delete is successful, false otherwise

Throws:

IOException

mkdirs

public boolean mkdirs(Path path)
               throws IOException

Description copied from interface: FileIO

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in interface FileIO

Parameters:

path - the directory/directories to be created

Returns:

true if at least one new directory has been created, false otherwise

Throws:

IOException - thrown if an I/O error occurs while creating the directory

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Description copied from interface: FileIO

Renames the file/directory src to dst.

Specified by:

rename in interface FileIO

Parameters:

src - the file/directory to rename

dst - the new name of the file/directory

Returns:

true if the renaming was successful, false otherwise

Throws:

IOException

createFileSystem

protected abstract org.apache.hadoop.fs.FileSystem createFileSystem(org.apache.hadoop.fs.Path path)
                                                             throws IOException

Throws:

IOException