org.apache.paimon.schema

Class SchemaEvolutionUtil

java.lang.Object

org.apache.paimon.schema.SchemaEvolutionUtil

public class SchemaEvolutionUtil
extends Object

Utils for schema evolution.

Constructor Summary

Constructors

Constructor and Description
SchemaEvolutionUtil()

Method Summary

Modifier and Type	Method and Description
static CastExecutor<?,?>[]	createConvertMapping(List<DataField> tableFields, List<DataField> dataFields, int[] indexMapping) Create converter mapping from table fields to underlying data fields.
static List<Predicate>	createDataFilters(List<DataField> tableFields, List<DataField> dataFields, List<Predicate> filters) Create predicate list from data fields.
static int[][]	createDataProjection(List<DataField> tableFields, List<DataField> dataFields, int[][] tableProjection) Create data projection from table projection.
static IndexCastMapping	createIndexCastMapping(int[] tableProjection, List<DataField> tableFields, int[] dataProjection, List<DataField> dataFields) Create index mapping from table projection to underlying data projection.
static IndexCastMapping	createIndexCastMapping(int[] tableProjection, List<DataField> tableKeyFields, List<DataField> tableValueFields, int[] dataProjection, List<DataField> dataKeyFields, List<DataField> dataValueFields) Create index mapping from table projection to data with key and value fields.
static IndexCastMapping	createIndexCastMapping(List<DataField> tableFields, List<DataField> dataFields) Create index mapping from table fields to underlying data fields.
static int[]	createIndexMapping(List<DataField> tableFields, List<DataField> dataFields) Create index mapping from table fields to underlying data fields.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SchemaEvolutionUtil

public SchemaEvolutionUtil()

Method Detail

createIndexMapping

@Nullable
public static int[] createIndexMapping(List<DataField> tableFields,
                                                 List<DataField> dataFields)

Create index mapping from table fields to underlying data fields. For example, the table and data fields are as follows

• table fields: 1->c, 6->b, 3->a
• data fields: 1->a, 3->c

We can get the index mapping [0, -1, 1], in which 0 is the index of table field 1->c in data fields, -1 is the index of 6->b in data fields and 1 is the index of 3->a in data fields.

/// TODO should support nest index mapping when nest schema evolution is supported.

Parameters:

tableFields - the fields of table

dataFields - the fields of underlying data

Returns:

the index mapping

createIndexCastMapping

public static IndexCastMapping createIndexCastMapping(int[] tableProjection,
                                                      List<DataField> tableFields,
                                                      int[] dataProjection,
                                                      List<DataField> dataFields)

Create index mapping from table projection to underlying data projection. For example, the table and data fields are as follows

• table fields: 1->c, 3->a, 4->e, 5->d, 6->b
• data fields: 1->a, 2->b, 3->c, 4->d

The table and data top projections are as follows

• table projection: [0, 4, 1]
• data projection: [0, 2]

We can first get fields list for table and data projections from their fields as follows

• table projection field list: [1->c, 6->b, 3->a]
• data projection field list: [1->a, 3->c]

Then create index mapping based on the fields list and create cast mapping based on index mapping.

/// TODO should support nest index mapping when nest schema evolution is supported.

Parameters:

tableProjection - the table projection

tableFields - the fields in table

dataProjection - the underlying data projection

dataFields - the fields in underlying data

Returns:

the index mapping

createIndexCastMapping

public static IndexCastMapping createIndexCastMapping(List<DataField> tableFields,
                                                      List<DataField> dataFields)

Create index mapping from table fields to underlying data fields.

createIndexCastMapping

public static IndexCastMapping createIndexCastMapping(int[] tableProjection,
                                                      List<DataField> tableKeyFields,
                                                      List<DataField> tableValueFields,
                                                      int[] dataProjection,
                                                      List<DataField> dataKeyFields,
                                                      List<DataField> dataValueFields)

Create index mapping from table projection to data with key and value fields. We should first create table and data fields with their key/value fields, then create index mapping with their projections and fields. For example, the table and data projections and fields are as follows

• Table key fields: 1->ka, 3->kb, 5->kc, 6->kd; value fields: 0->a, 2->d, 4->b; projection: [0, 2, 3, 4, 5, 7] where 0 is 1->ka, 2 is 5->kc, 3 is 5->kc, 4/5 are seq and kind, 7 is 2->d
• Data key fields: 1->kb, 5->ka; value fields: 2->aa, 4->f; projection: [0, 1, 2, 3, 4] where 0 is 1->kb, 1 is 5->ka, 2/3 are seq and kind, 4 is 2->aa

First we will get max key id from table and data fields which is 6, then create table and data fields on it

• Table fields: 1->ka, 3->kb, 5->kc, 6->kd, 7->seq, 8->kind, 9->a, 11->d, 13->b
• Data fields: 1->kb, 5->ka, 7->seq, 8->kind, 11->aa, 13->f

Finally we can create index mapping with table/data projections and fields, and create cast mapping based on index mapping.

/// TODO should support nest index mapping when nest schema evolution is supported.

Parameters:

tableProjection - the table projection

tableKeyFields - the table key fields

tableValueFields - the table value fields

dataProjection - the data projection

dataKeyFields - the data key fields

dataValueFields - the data value fields

Returns:

the result index and cast mapping

createDataProjection

public static int[][] createDataProjection(List<DataField> tableFields,
                                           List<DataField> dataFields,
                                           int[][] tableProjection)

Create data projection from table projection. For example, the table and data fields are as follows

• table fields: 1->c, 3->a, 4->e, 5->d, 6->b
• data fields: 1->a, 2->b, 3->c, 4->d

When we project 1->c, 6->b, 3->a from table fields, the table projection is [[0], [4], [1]], in which 0 is the index of field 1->c, 4 is the index of field 6->b, 1 is the index of field 3->a in table fields. We need to create data projection from [[0], [4], [1]] as follows:

• Get field id of each index in table projection from table fields
• Get index of each field above from data fields

The we can create table projection as follows: [[0], [-1], [2]], in which 0, -1 and 2 are the index of fields [1->c, 6->b, 3->a] in data fields. When we project column from underlying data, we need to specify the field index and name. It is difficult to assign a proper field id and name for 6->b in data projection and add it to data fields, and we can't use 6->b directly because the field index of b in underlying is 2. We can remove the -1 field index in data projection, then the result data projection is: [[0], [2]].

We create InternalRow for 1->a, 3->c after projecting them from underlying data, then create ProjectedRow with a index mapping and return null for 6->b in table fields.

Parameters:

tableFields - the fields of table

dataFields - the fields of underlying data

tableProjection - the projection of table

Returns:

the projection of data

createDataFilters

@Nullable
public static List<Predicate> createDataFilters(List<DataField> tableFields,
                                                          List<DataField> dataFields,
                                                          List<Predicate> filters)

Create predicate list from data fields. We will visit all predicate in filters, reset it's field index, name and type, and ignore predicate if the field is not exist.

Parameters:

tableFields - the table fields

dataFields - the underlying data fields

filters - the filters

Returns:

the data filters

createConvertMapping

@Nullable
public static CastExecutor<?,?>[] createConvertMapping(List<DataField> tableFields,
                                                                 List<DataField> dataFields,
                                                                 int[] indexMapping)

Create converter mapping from table fields to underlying data fields. For example, the table and data fields are as follows

• table fields: 1->c INT, 6->b STRING, 3->a BIGINT
• data fields: 1->a BIGINT, 3->c DOUBLE

We can get the column types (1->a BIGINT), (3->c DOUBLE) from data fields for (1->c INT) and (3->a BIGINT) in table fields through index mapping [0, -1, 1], then compare the data type and create converter mapping.

/// TODO should support nest index mapping when nest schema evolution is supported.

Parameters:

tableFields - the fields of table

dataFields - the fields of underlying data

indexMapping - the index mapping from table fields to data fields

Returns:

the index mapping