Semantic Views
A semantic view is a named catalog object that stores a complete semantic model definition. The model describes datasets, relationships, dimensions, measures, and related business semantics. This experimental REST Catalog extension manages the object and its definition. The Java client preserves the original document text. Servers validate the model formats, syntax versions, and features they support; consumers provide query execution.
Registration alone does not grant access to source data or guarantee engine support. The Spark 4.2 connector supports the Ossie query profile below. This implementation supplies Java clients, wire models, and the Catalog OpenAPI contract. REST server providers must implement persistence, validation, authorization, and the atomic operations described below.
REST operations
GET /v1/{prefix}/databases/{database}/semantic-views
GET /v1/{prefix}/databases/{database}/semantic-views/{semanticView}
POST /v1/{prefix}/databases/{database}/semantic-views/{semanticView}
DELETE /v1/{prefix}/databases/{database}/semantic-views/{semanticView}
Use the opaque prefix returned by config. Database and object names are independent UTF-8 encoded path segments, including slashes, percent signs, spaces, Unicode, and dot-only names. Names are never split on dots, and a catalog name inside model content does not determine the REST prefix.
Upsert and read
POST creates or atomically replaces one complete definition in an existing database:
{
"definition": {
"format": "databricks-yaml",
"content": "version: '1.1'\nsource: main.sales.orders\nmeasures:\n - name: revenue\n expr: SUM(paid_amount)\n"
}
}
format and content are required nonblank strings. format identifies both the model syntax
and document encoding, for example databricks-yaml, snowflake-yaml, or ossie-yaml. These are
Paimon format identifiers, not a closed enumeration; each server decides which formats it supports.
The example requires server support for databricks-yaml. Model syntax versions and expression
SQL dialects belong inside the content, following the chosen format's specification.
Clients do not parse, normalize, or discard fields in the content.
Content is limited to 1 MiB of UTF-8 bytes, checked before sending by the Java client.
POST and GET return HTTP 200 with the full object:
{
"name": "order_metrics",
"definition": {
"format": "databricks-yaml",
"content": "version: '1.1'\nsource: main.sales.orders\nmeasures:\n - name: revenue\n expr: SUM(paid_amount)\n"
}
}
name is the local object name within the database addressed by the path. POST returns the
committed object, and subsequent reads must expose the complete committed definition.
Upserts are unconditional: the last successful write takes effect. Each update replaces the whole definition, so concurrent edits can overwrite one another. Replacement preserves identity, creation metadata, owner, permissions, and labels. Definition-level comments and synonyms are part of the complete replacement.
List and delete
List accepts optional maxResults (1–1000; omitted uses the server default) and opaque pageToken:
{"semanticViews":["order_metrics"],"nextPageToken":"opaque-token"}
Lists return visible names only; GET retrieves a full definition. The last page omits the token. An empty page terminates pagination and must not carry a continuation token. A missing database returns 404; an existing database with no visible models returns an empty array.
DELETE uses the object path with no query parameters or request body. It returns 200 without a body; an absent object returns 404, and a blocking dependency returns 409.
Java catalog access
import org.apache.paimon.catalog.Identifier;
import org.apache.paimon.management.SemanticViewManagement;
import org.apache.paimon.view.SemanticView;
import org.apache.paimon.view.SemanticViewDefinition;
SemanticViewManagement models = restCatalog.semanticViewManagement();
Identifier id = Identifier.create("sales", "order_metrics");
SemanticViewDefinition definition = new SemanticViewDefinition("databricks-yaml", yamlText);
SemanticView saved = models.upsertSemanticView(id, definition);
SemanticView current = models.getSemanticView(id);
models.upsertSemanticView(id, replacementDefinition);
models.listSemanticViews("sales"); // Follows all pages.
models.listSemanticViewsPaged("sales", 100, null);
models.deleteSemanticView(id);
The accessor reuses the catalog's REST client, prefix, authentication, and configured headers.
It is specific to RESTCatalog, not the generic Catalog interface. RESTApi also exposes the same
upsert/get/list/paged-list/delete operations; its read and upsert methods return GetSemanticViewResponse.
SemanticView does not implement ordinary View.query() or View.rowType().
Server integration contract
- Ordinary SQL views and semantic views share the database's view namespace. Both creation paths must reject a cross-type name collision with 409. They cannot implicitly convert each other.
- Existing
/views,/view-details, and global view lists return ordinary SQL views only. Existing get/alter/drop SQL view routes treat a semantic view name as a missing ordinary view (404). Existing table/function naming rules continue to apply. - Reuse permission
ResourceType.VIEWand structured database/view identities. Creation checks the parent database'sCREATEVIEW; replacement checksALTER; deletion checksDROP; reads checkSELECT. Discovery follows databaseLISTand object visibility rules. Server resource resolution must understand the semantic subtype. Source access is checked separately at execution. - Labels use
entityType=VIEWand a canonicalentityNamesupplied according to the provider's naming and escaping rules; the semantic view response does not supply that label identity. Clients must not assume that concatenating database and view names with a dot produces it. The label resolver must support semantic views. Definition replacement preserves bindings; deletion cleans up direct labels and permissions without deleting referenced sources. - Semantic views count when determining whether a database is empty. Database cascade deletion follows existing catalog rules and cleans up semantic metadata and its direct bindings.
- Servers validate the model formats and metadata they accept without dropping fields. Execution adapters additionally validate supported model semantics, expressions, and source access when querying an object. The management client exposes no generic executable-validation status.
The HTTP client tests verify wire behavior and error propagation. They do not prove a provider's storage atomicity, permission enforcement, namespace isolation, or database lifecycle behavior.
Errors and scope
Errors use ErrorResponse; semantic view errors use resourceType=SEMANTIC_VIEW.
| HTTP | Meaning | Java client exception |
|---|---|---|
| 400 | Invalid input or unsupported model format, syntax version, or feature | BadRequestException |
| 401 / 403 | Authentication or permission failure | NotAuthorizedException / ForbiddenException |
| 404 | Missing database or model | NoSuchResourceException |
| 409 | Name conflict or dependency blocks deletion | AlreadyExistsException (existing REST mapping) |
| 413 | Content exceeds the size limit | RESTException |
| 501 | Server does not implement semantic views | NotImplementedException |
Client-side invalid input throws IllegalArgumentException before HTTP. An unsupported endpoint
never falls back to SQL views, labels, or table options. Materialization, rename, and per-measure
or per-field REST resources are outside this version.
Spark 4.2 Ossie queries
The paimon-spark-4.2_2.13 connector can query an ossie-yaml definition through the configured
Paimon REST Catalog. Enable PaimonSparkSessionExtensions as described in the
Spark installation guide. The REST server must implement semantic
view management; the connector does not provide production REST persistence or validation.
Register a definition whose content is a flat Ossie document, for example:
version: 0.3.0.dev0
name: order_metrics
datasets:
- name: orders
source: orders
fields:
- name: region
expression:
sql: region_code
- name: net_amount
expression:
sql: amount - discount
metrics:
- name: revenue
expression:
sql: SUM(net_amount)
For a definition registered as sales.order_metrics in Spark catalog paimon, query it with:
SELECT region, MEASURE(revenue) AS revenue
FROM paimon.sales.order_metrics
WHERE region = 'CN'
GROUP BY region;
The connector reads the definition at analysis time, compiles it into a Spark Metric View, and
uses Spark's native MEASURE rewrite and execution. It does not persist a converted copy or cache
definitions across queries. Reanalyzing a query sees the current committed definition; an already
analyzed DataFrame keeps its existing plan.
This experimental execution profile supports:
- Exactly one dataset backed by a physical Paimon table in the same Spark catalog. A one-part source resolves in the semantic view's database; two parts specify database and table; three parts specify catalog, database and table. SQL queries as sources are not supported.
- Declared scalar fields and dataset-scoped metrics. A dataset metric's unqualified reference
resolves to a declared field;
source.columnreaches the original source column. A field namedsourceprevents that source-column qualification, following Ossie's scoping rule. - Model-scoped metrics referencing that dataset's qualified fields. Field and metric names must be unique, ignoring case, in the resulting Spark relation.
- Portable
expression.sqlor anOSSIE_SQL_2026dialect expression; arithmetic, comparisons, boolean logic,CASE,IN, numeric/string/boolean/date casts,SUM,COUNT(including distinct),AVG,MIN,MAX,ABS,ROUND,FLOOR,CEIL,CEILING,COALESCE, andDATE_TRUNC. DATE_TRUNC(unit, input)requires a case-insensitive string literal for the unit. The supported units areyear,quarter,month,week,day,hour,minute, andsecond. Weeks start on Monday. The result isTIMESTAMP_LTZ. Spark uses the session time zone to truncateTIMESTAMP_LTZvalues and implicitly convertDATEvalues and timestamp strings toTIMESTAMP_LTZ. ANULLinput returnsNULL.TIMESTAMP_NTZinputs are rejected during analysis.
Other expressions, relationships, multiple datasets, external joins, streaming reads and writes
are rejected. SQL Semantic View DDL and SQL discovery are not added; use the management API to
register and list definitions. Ordinary tables and SQL views retain their existing lookup paths.
Source table authorization is required independently of permission to read the semantic definition.
When using a CTE, evaluate measures inside its definition (for example,
WITH m AS (SELECT MEASURE(revenue) AS revenue FROM paimon.sales.order_metrics) SELECT revenue FROM m).
Forwarding raw measure columns through a CTE or temporary view is not supported by Spark 4.2's
Metric View resolver; evaluate measures inside the defining query block before reusing its results.
The bundled schema is pinned to Ossie commit
2692b1cc4708fdcf6a91e597ca928a9bc9d0da60.
The 0.3.0.dev0 version string is a development version and does not imply compatibility with
future changes under that same string. ai_context and nonportable_extensions remain in the
original document and do not alter query execution.
Extension data must contain JSON values with string keys. Quote date and timestamp strings;
unquoted dates, timestamps, non-finite numbers, and recursive YAML aliases are rejected.
Alias expansion is also bounded before JSON validation to prevent small documents from expanding
into unbounded in-memory models. Portable SQL string literals follow ANSI quoting, independently
of Spark's spark.sql.parser.escapedStringLiterals setting.