Cleanup and Repair

Inspect orphan files or repair missing files, manifests, and table metadata.

See Procedures for Flink version requirements, argument conventions, and catalog selection.

remove_orphan_files

To remove the orphan data files and metadata files. Arguments:

table: the target table identifier. Cannot be empty, you can use database_name.* to clean whole database.
olderThan: to avoid deleting newly written files, this procedure only deletes orphan files older than 1 day by default. This argument can modify the interval.
dryRun: when true, view only orphan files, don't actually remove files. Default is false.
parallelism: The maximum number of concurrent deleting files. By default is the number of processors available to the Java virtual machine.
mode: The mode of remove orphan clean procedure (local or distributed) . By default is distributed.

Syntax

-- Use named argument
CALL [catalog.]sys.remove_orphan_files(
    `table` => 'identifier',
    older_than => 'olderThan',
    dry_run => 'dryRun',
    mode => 'mode'
);

-- Use indexed argument
CALL [catalog.]sys.remove_orphan_files('identifier');

CALL [catalog.]sys.remove_orphan_files('identifier', 'olderThan');

CALL [catalog.]sys.remove_orphan_files('identifier', 'olderThan', 'dryRun');

CALL [catalog.]sys.remove_orphan_files('identifier', 'olderThan', 'dryRun','parallelism');

CALL [catalog.]sys.remove_orphan_files('identifier', 'olderThan', 'dryRun','parallelism','mode');

Example

CALL sys.remove_orphan_files(`table` => 'default.T', older_than => '2023-10-31 12:00:00');

CALL sys.remove_orphan_files(`table` => 'default.*', older_than => '2023-10-31 12:00:00');

CALL sys.remove_orphan_files(`table` => 'default.T', older_than => '2023-10-31 12:00:00', dry_run => true);

CALL sys.remove_orphan_files(
    `table` => 'default.T',
    older_than => '2023-10-31 12:00:00',
    dry_run => false,
    parallelism => 5
);

CALL sys.remove_orphan_files(
    `table` => 'default.T',
    older_than => '2023-10-31 12:00:00',
    dry_run => false,
    parallelism => 5,
    mode => 'local'
);

This procedure does not delete primary-key .managed.blob packs. Use remove_orphan_blobs.

remove_orphan_blobs

Remove unreferenced primary-key .managed.blob packs.

Arguments

table: the target table identifier. Cannot be empty, you can use database_name.* to clean the whole database.
olderThan: an absolute timestamp cutoff. Only packs whose modification time is earlier than this timestamp are candidates. The default cutoff is 1 day before the procedure starts.
dryRun: when true, calculate the candidate file count and total bytes without deleting files. The procedure returns aggregate counts, not individual pack paths. Default is false.
parallelism: per-table concurrency. In distributed mode this is the Flink task parallelism of each table job. In local mode this is the per-table file-operation thread limit (default: the number of processors available to the Java virtual machine). For database_name.*, distributed mode runs tables one after another, so cluster concurrency stays within this per-table value; local mode may run several tables at once, so total threads can exceed this value.
mode: The mode of remove orphan blob procedure (local or distributed). By default is distributed.

Syntax

-- Use named argument
CALL [catalog.]sys.remove_orphan_blobs(
    `table` => 'identifier',
    older_than => 'olderThan',
    dry_run => 'dryRun',
    parallelism => parallelism,
    mode => 'mode'
);

-- Use indexed argument
CALL [catalog.]sys.remove_orphan_blobs('identifier');

CALL [catalog.]sys.remove_orphan_blobs('identifier', 'olderThan');

CALL [catalog.]sys.remove_orphan_blobs('identifier', 'olderThan', 'dryRun');

CALL [catalog.]sys.remove_orphan_blobs('identifier', 'olderThan', 'dryRun','parallelism');

CALL [catalog.]sys.remove_orphan_blobs('identifier', 'olderThan', 'dryRun','parallelism','mode');

Example

CALL sys.remove_orphan_blobs(`table` => 'default.T', older_than => '2023-10-31 12:00:00');

CALL sys.remove_orphan_blobs(`table` => 'default.*', older_than => '2023-10-31 12:00:00');

CALL sys.remove_orphan_blobs(`table` => 'default.T', older_than => '2023-10-31 12:00:00', dry_run => true);

CALL sys.remove_orphan_blobs(
    `table` => 'default.T',
    older_than => '2023-10-31 12:00:00',
    dry_run => false,
    parallelism => 5
);

CALL sys.remove_orphan_blobs(
    `table` => 'default.T',
    older_than => '2023-10-31 12:00:00',
    dry_run => false,
    parallelism => 5,
    mode => 'local'
);

remove_unexisting_files

Procedure to remove unexisting data files from manifest entries. See Java docs for detailed use cases. Arguments:

table: the target table identifier. Cannot be empty, you can use database_name.* to clean whole database.
dry_run (optional): only check what files will be removed, but not really remove them. Default is false.
parallelism (optional): number of parallelisms to check files in the manifests.

Note that user is on his own risk using this procedure, which may cause data loss when used outside from the use cases listed in Java docs.

Syntax

-- Use named argument
CALL [catalog.]sys.remove_unexisting_files(
    `table` => 'identifier',
    dry_run => 'dryRun',
    parallelism => parallelism
);

-- Use indexed argument
CALL [catalog.]sys.remove_unexisting_files('identifier');

CALL [catalog.]sys.remove_unexisting_files('identifier', 'dryRun', 'parallelism');

Example

-- remove unexisting data files in the table `mydb.myt`
CALL sys.remove_unexisting_files(`table` => 'mydb.myt');

-- only check what files will be removed, but not really remove them (dry run)
CALL sys.remove_unexisting_files(`table` => 'mydb.myt', `dry_run` => true);

remove_unexisting_manifests

Procedure to remove unexisting manifest file from manifset-list. for detailed use cases. Arguments:

tableId: the target table identifier. Cannot be empty, you can use database.table$branch_xx to remove branch table unexisting manifest file.

Note that user is on his own risk using this procedure, which may cause data loss when used outside from the use cases listed in Java docs.

Syntax

-- Use indexed argument
CALL [catalog.]sys.remove_unexisting_manifests('identifier');

Example

-- remove unexisting manifest file in the table `mydb.myt`
CALL sys.remove_unexisting_manifests('mydb.myt');

-- remove unexisting manifest file in the branch table `mydb.myt$branch_rt`
CALL sys.remove_unexisting_manifests('mydb.myt$branch_rt');

repair

Synchronize information from the file system to Metastore. Argument:

empty: all databases and tables in catalog.
databaseName: the target database name.
tableName: the target table identifier.

Syntax

-- repair all databases and tables in catalog
CALL [catalog.]sys.repair();

-- repair all tables in a specific database
CALL [catalog.]sys.repair('databaseName');

-- repair a table
CALL [catalog.]sys.repair('databaseName.tableName');

-- repair database and table in a string if you specify multiple tags, delimiter is ','
CALL [catalog.]sys.repair('databaseName01,database02.tableName01,database03');

Example

CALL sys.repair(`table` => 'test_db.T');

repair_earliest_snapshot

Repair the earliest snapshot hint for a table. Arguments:

table: the target table identifier. Cannot be empty.
snapshot_id: the snapshot ID to set as the earliest snapshot.

Syntax

CALL [catalog.]sys.repair_earliest_snapshot(`table` => 'identifier', snapshot_id => snapshotId);

Example

CALL sys.repair_earliest_snapshot(`table` => 'default.T', snapshot_id => 10);

remove_orphan_files​

remove_orphan_blobs​

remove_unexisting_files​

remove_unexisting_manifests​

repair​

repair_earliest_snapshot​

remove_orphan_files

remove_orphan_blobs

remove_unexisting_files

remove_unexisting_manifests

repair

repair_earliest_snapshot