org.apache.paimon.append

Class AppendCompactCoordinator

java.lang.Object

org.apache.paimon.append.AppendCompactCoordinator

public class AppendCompactCoordinator
extends Object

Compact coordinator for append only tables.

Note: AppendCompactCoordinator scan files in snapshot, read APPEND and COMPACT snapshot then load those new files. It will try it best to generate compaction task for the restored files scanned in snapshot, but to reduce memory usage, it won't remain single file for a long time. After ten times scan, single file with one partition will be ignored and removed from memory, which means, it will not participate in compaction again until restart the compaction job.

When a third task delete file in latest snapshot(including batch delete/update and overwrite), the file in coordinator will still remain and participate in compaction task. When this happens, compaction job will fail in commit stage, and fail-over to rescan the restored files in latest snapshot.

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	COMPACT_AGE
protected static int	REMOVE_AGE

Constructor Summary

Constructors

Constructor and Description
AppendCompactCoordinator(FileStoreTable table, boolean isStreaming)
AppendCompactCoordinator(FileStoreTable table, boolean isStreaming, Predicate filter)

Method Summary

Modifier and Type	Method and Description
List<AppendCompactTask>	run()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

REMOVE_AGE

protected static final int REMOVE_AGE

See Also:

Constant Field Values

COMPACT_AGE

protected static final int COMPACT_AGE

See Also:

Constant Field Values

Constructor Detail

AppendCompactCoordinator

public AppendCompactCoordinator(FileStoreTable table,
                                boolean isStreaming)

AppendCompactCoordinator

public AppendCompactCoordinator(FileStoreTable table,
                                boolean isStreaming,
                                @Nullable
                                Predicate filter)

Method Detail

run

public List<AppendCompactTask> run()