This documentation is for an unreleased version of Apache Paimon. We recommend you use the latest stable version.

Functions #

Paimon introduces a Function abstraction designed to support functions in a standard format for compute engine, addressing:

Unified Column-Level Filtering and Processing: Facilitates operations at the column level, including tasks such as encryption and decryption of data.
Parameterized View Capabilities: Supports parameterized operations within views, enhancing the dynamism and usability of data retrieval processes.

Types of Functions Supported #

Currently, Paimon supports three types of functions:

File Function: Users can define functions within a file, providing flexibility and modular support for function definition.
Lambda Function: Empowering users to define functions using Java lambda expressions, enabling inline, concise, and functional-style operations.
SQL Function: Users can define functions directly within SQL, which integrates seamlessly with SQL-based data processing.

File Function Usage in Flink #

Paimon functions can be utilized within Apache Flink to execute complex data operations. Below are the SQL commands for creating, altering, and dropping functions in Flink environments.

Create Function #

To create a new function in Flink SQL:

-- Flink SQL
CREATE FUNCTION mydb.parse_str
    AS 'com.streaming.flink.udf.StrUdf' 
    LANGUAGE JAVA
    USING JAR 'oss://my_bucket/my_location/udf.jar' [, JAR 'oss://my_bucket/my_location/a.jar'];

This statement creates a Java-based user-defined function named parse_str within the mydb database, utilizing specified JAR files from an object storage location.

Alter Function #

To modify an existing function in Flink SQL:

-- Flink SQL
ALTER FUNCTION mydb.parse_str
    AS 'com.streaming.flink.udf.StrUdf2' 
    LANGUAGE JAVA;

This command changes the implementation of the parse_str function to use a new Java class definition.

Drop Function #

To remove a function from Flink SQL:

-- Flink SQL
DROP FUNCTION mydb.parse_str;

This statement deletes the existing parse_str function from the mydb database, relinquishing its functionality.

Functions in Spark #

see SQL Functions