org.apache.paimon.fs.hadoop

Class HadoopFileIO

java.lang.Object

org.apache.paimon.fs.hadoop.HadoopFileIO

All Implemented Interfaces:

Serializable, FileIO

public class HadoopFileIO
extends Object
implements FileIO

Hadoop FileIO.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Map<Pair<String,String>,org.apache.hadoop.fs.FileSystem>	fsMap
protected SerializableConfiguration	hadoopConf

Fields inherited from interface org.apache.paimon.fs.FileIO

LOG

Constructor Summary

Constructors

Constructor and Description
HadoopFileIO()

Method Summary

Modifier and Type	Method and Description
void	configure(CatalogContext context) Configure by CatalogContext.
protected org.apache.hadoop.fs.FileSystem	createFileSystem(org.apache.hadoop.fs.Path path)
boolean	delete(Path path, boolean recursive) Delete a file.
boolean	exists(Path path) Check if exists.
FileStatus	getFileStatus(Path path) Return a file status object that represents the path.
boolean	isObjectStore()
FileStatus[]	listStatus(Path path) List the statuses of the files/directories in the given path if the path is a directory.
boolean	mkdirs(Path path) Make the given file and all non-existent parents into directories.
SeekableInputStream	newInputStream(Path path) Opens an SeekableInputStream at the indicated Path.
PositionOutputStream	newOutputStream(Path path, boolean overwrite) Opens an PositionOutputStream at the indicated Path.
void	overwriteFileUtf8(Path path, String content) Overwrite file by content atomically, different FileIOs have different atomic implementations.
boolean	rename(Path src, Path dst) Renames the file/directory src to dst.
void	setFileSystem(Path path, org.apache.hadoop.fs.FileSystem fs)
boolean	tryAtomicOverwriteViaRename(Path dst, String content)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.paimon.fs.FileIO

checkOrMkdirs, copyFile, copyFiles, deleteDirectoryQuietly, deleteFilesQuietly, deleteQuietly, discoverLoaders, get, getFileSize, isDir, listDirectories, readFileUtf8, readOverwrittenFileUtf8, tryToWriteAtomic, writeFile

Field Detail

hadoopConf

protected SerializableConfiguration hadoopConf

fsMap

protected transient volatile Map<Pair<String,String>,org.apache.hadoop.fs.FileSystem> fsMap

Constructor Detail

HadoopFileIO

public HadoopFileIO()

Method Detail

setFileSystem

@VisibleForTesting
public void setFileSystem(Path path,
                                             org.apache.hadoop.fs.FileSystem fs)
                                      throws IOException

Throws:

IOException

isObjectStore

public boolean isObjectStore()

Specified by:

isObjectStore in interface FileIO

configure

public void configure(CatalogContext context)

Description copied from interface: FileIO

Configure by CatalogContext.

Specified by:

configure in interface FileIO

newInputStream

public SeekableInputStream newInputStream(Path path)
                                   throws IOException

Description copied from interface: FileIO

Opens an SeekableInputStream at the indicated Path.

Specified by:

newInputStream in interface FileIO

Parameters:

path - the file to open

Throws:

IOException

newOutputStream

public PositionOutputStream newOutputStream(Path path,
                                            boolean overwrite)
                                     throws IOException

Description copied from interface: FileIO

Opens an PositionOutputStream at the indicated Path.

Specified by:

newOutputStream in interface FileIO

Parameters:

path - the file name to open

overwrite - if a file with this name already exists, then if true, the file will be overwritten, and if false an error will be thrown.

Throws:

IOException - Thrown, if the stream could not be opened because of an I/O, or because a file already exists at that path and the write mode indicates to not overwrite the file.

getFileStatus

public FileStatus getFileStatus(Path path)
                         throws IOException

Description copied from interface: FileIO

Return a file status object that represents the path.

Specified by:

getFileStatus in interface FileIO

Parameters:

path - The path we want information from

Returns:

a FileStatus object

Throws:

FileNotFoundException - when the path does not exist; IOException see specific implementation

IOException

listStatus

public FileStatus[] listStatus(Path path)
                        throws IOException

Description copied from interface: FileIO

List the statuses of the files/directories in the given path if the path is a directory.

Specified by:

listStatus in interface FileIO

Parameters:

path - given path

Returns:

the statuses of the files/directories in the given path

Throws:

IOException

exists

public boolean exists(Path path)
               throws IOException

Description copied from interface: FileIO

Check if exists.

Specified by:

exists in interface FileIO

Parameters:

path - source file

Throws:

IOException

delete

public boolean delete(Path path,
                      boolean recursive)
               throws IOException

Description copied from interface: FileIO

Delete a file.

Specified by:

delete in interface FileIO

Parameters:

path - the path to delete

recursive - if path is a directory and set to true, the directory is deleted else throws an exception. In case of a file the recursive can be set to either true or false

Returns:

true if delete is successful, false otherwise

Throws:

IOException

mkdirs

public boolean mkdirs(Path path)
               throws IOException

Description copied from interface: FileIO

Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.

Specified by:

mkdirs in interface FileIO

Parameters:

path - the directory/directories to be created

Returns:

true if at least one new directory has been created, false otherwise

Throws:

IOException - thrown if an I/O error occurs while creating the directory

rename

public boolean rename(Path src,
                      Path dst)
               throws IOException

Description copied from interface: FileIO

Renames the file/directory src to dst.

Specified by:

rename in interface FileIO

Parameters:

src - the file/directory to rename

dst - the new name of the file/directory

Returns:

true if the renaming was successful, false otherwise

Throws:

IOException

overwriteFileUtf8

public void overwriteFileUtf8(Path path,
                              String content)
                       throws IOException

Description copied from interface: FileIO

Overwrite file by content atomically, different FileIOs have different atomic implementations.

Specified by:

overwriteFileUtf8 in interface FileIO

Throws:

IOException

createFileSystem

protected org.apache.hadoop.fs.FileSystem createFileSystem(org.apache.hadoop.fs.Path path)
                                                    throws IOException

Throws:

IOException

tryAtomicOverwriteViaRename

public boolean tryAtomicOverwriteViaRename(Path dst,
                                           String content)
                                    throws IOException

Throws:

IOException