Skip to main content

Semantic Views

A semantic view is a named catalog object that stores a complete semantic model definition. The model describes datasets, relationships, dimensions, measures, and related business semantics. This experimental REST Catalog extension manages the object and its definition. The Java client preserves the original document text. Servers validate the model formats, syntax versions, and features they support; consumers provide query execution.

Registration alone does not grant access to source data or guarantee engine support. The Spark 4.2 connector supports the Ossie query profile below. This implementation supplies Java clients, wire models, and the Catalog OpenAPI contract. REST server providers must implement persistence, validation, authorization, and the atomic operations described below.

REST operations​

GET /v1/{prefix}/databases/{database}/semantic-views
GET /v1/{prefix}/databases/{database}/semantic-views/{semanticView}
POST /v1/{prefix}/databases/{database}/semantic-views/{semanticView}
DELETE /v1/{prefix}/databases/{database}/semantic-views/{semanticView}

Use the opaque prefix returned by config. Database and object names are independent UTF-8 encoded path segments, including slashes, percent signs, spaces, Unicode, and dot-only names. Names are never split on dots, and a catalog name inside model content does not determine the REST prefix.

Upsert and read​

POST creates or atomically replaces one complete definition in an existing database:

{
"definition": {
"format": "databricks-yaml",
"content": "version: '1.1'\nsource: main.sales.orders\nmeasures:\n - name: revenue\n expr: SUM(paid_amount)\n"
}
}

format and content are required nonblank strings. format identifies both the model syntax and document encoding, for example databricks-yaml, snowflake-yaml, or ossie-yaml. These are Paimon format identifiers, not a closed enumeration; each server decides which formats it supports. The example requires server support for databricks-yaml. Model syntax versions and expression SQL dialects belong inside the content, following the chosen format's specification. Clients do not parse, normalize, or discard fields in the content. Content is limited to 1 MiB of UTF-8 bytes, checked before sending by the Java client.

POST and GET return HTTP 200 with the full object:

{
"name": "order_metrics",
"definition": {
"format": "databricks-yaml",
"content": "version: '1.1'\nsource: main.sales.orders\nmeasures:\n - name: revenue\n expr: SUM(paid_amount)\n"
}
}

name is the local object name within the database addressed by the path. POST returns the committed object, and subsequent reads must expose the complete committed definition.

Upserts are unconditional: the last successful write takes effect. Each update replaces the whole definition, so concurrent edits can overwrite one another. Replacement preserves identity, creation metadata, owner, permissions, and labels. Definition-level comments and synonyms are part of the complete replacement.

List and delete​

List accepts optional maxResults (1–1000; omitted uses the server default) and opaque pageToken:

{"semanticViews":["order_metrics"],"nextPageToken":"opaque-token"}

Lists return visible names only; GET retrieves a full definition. The last page omits the token. An empty page terminates pagination and must not carry a continuation token. A missing database returns 404; an existing database with no visible models returns an empty array.

DELETE uses the object path with no query parameters or request body. It returns 200 without a body; an absent object returns 404, and a blocking dependency returns 409.

Java catalog access​

import org.apache.paimon.catalog.Identifier;
import org.apache.paimon.management.SemanticViewManagement;
import org.apache.paimon.view.SemanticView;
import org.apache.paimon.view.SemanticViewDefinition;

SemanticViewManagement models = restCatalog.semanticViewManagement();
Identifier id = Identifier.create("sales", "order_metrics");
SemanticViewDefinition definition = new SemanticViewDefinition("databricks-yaml", yamlText);
SemanticView saved = models.upsertSemanticView(id, definition);
SemanticView current = models.getSemanticView(id);
models.upsertSemanticView(id, replacementDefinition);
models.listSemanticViews("sales"); // Follows all pages.
models.listSemanticViewsPaged("sales", 100, null);
models.deleteSemanticView(id);

The accessor reuses the catalog's REST client, prefix, authentication, and configured headers. It is specific to RESTCatalog, not the generic Catalog interface. RESTApi also exposes the same upsert/get/list/paged-list/delete operations; its read and upsert methods return GetSemanticViewResponse. SemanticView does not implement ordinary View.query() or View.rowType().

Server integration contract​

  • Ordinary SQL views and semantic views share the database's view namespace. Both creation paths must reject a cross-type name collision with 409. They cannot implicitly convert each other.
  • Existing /views, /view-details, and global view lists return ordinary SQL views only. Existing get/alter/drop SQL view routes treat a semantic view name as a missing ordinary view (404). Existing table/function naming rules continue to apply.
  • Reuse permission ResourceType.VIEW and structured database/view identities. Creation checks the parent database's CREATEVIEW; replacement checks ALTER; deletion checks DROP; reads check SELECT. Discovery follows database LIST and object visibility rules. Server resource resolution must understand the semantic subtype. Source access is checked separately at execution.
  • Labels use entityType=VIEW and a canonical entityName supplied according to the provider's naming and escaping rules; the semantic view response does not supply that label identity. Clients must not assume that concatenating database and view names with a dot produces it. The label resolver must support semantic views. Definition replacement preserves bindings; deletion cleans up direct labels and permissions without deleting referenced sources.
  • Semantic views count when determining whether a database is empty. Database cascade deletion follows existing catalog rules and cleans up semantic metadata and its direct bindings.
  • Servers validate the model formats and metadata they accept without dropping fields. Execution adapters additionally validate supported model semantics, expressions, and source access when querying an object. The management client exposes no generic executable-validation status.

The HTTP client tests verify wire behavior and error propagation. They do not prove a provider's storage atomicity, permission enforcement, namespace isolation, or database lifecycle behavior.

Errors and scope​

Errors use ErrorResponse; semantic view errors use resourceType=SEMANTIC_VIEW.

HTTPMeaningJava client exception
400Invalid input or unsupported model format, syntax version, or featureBadRequestException
401 / 403Authentication or permission failureNotAuthorizedException / ForbiddenException
404Missing database or modelNoSuchResourceException
409Name conflict or dependency blocks deletionAlreadyExistsException (existing REST mapping)
413Content exceeds the size limitRESTException
501Server does not implement semantic viewsNotImplementedException

Client-side invalid input throws IllegalArgumentException before HTTP. An unsupported endpoint never falls back to SQL views, labels, or table options. Materialization, rename, and per-measure or per-field REST resources are outside this version.

Spark 4.2 Ossie queries​

The paimon-spark-4.2_2.13 connector can query an ossie-yaml definition through the configured Paimon REST Catalog. Enable PaimonSparkSessionExtensions as described in the Spark installation guide. The REST server must implement semantic view management; the connector does not provide production REST persistence or validation.

Register a definition whose content is a flat Ossie document, for example:

version: 0.3.0.dev0
name: order_metrics
datasets:
- name: orders
source: orders
fields:
- name: region
expression:
sql: region_code
- name: net_amount
expression:
sql: amount - discount
metrics:
- name: revenue
expression:
sql: SUM(net_amount)

For a definition registered as sales.order_metrics in Spark catalog paimon, query it with:

SELECT region, MEASURE(revenue) AS revenue
FROM paimon.sales.order_metrics
WHERE region = 'CN'
GROUP BY region;

The connector reads the definition at analysis time, compiles it into a Spark Metric View, and uses Spark's native MEASURE rewrite and execution. It does not persist a converted copy or cache definitions across queries. Reanalyzing a query sees the current committed definition; an already analyzed DataFrame keeps its existing plan.

This experimental execution profile supports:

  • Exactly one dataset backed by a physical Paimon table in the same Spark catalog. A one-part source resolves in the semantic view's database; two parts specify database and table; three parts specify catalog, database and table. SQL queries as sources are not supported.
  • Declared scalar fields and dataset-scoped metrics. A dataset metric's unqualified reference resolves to a declared field; source.column reaches the original source column. A field named source prevents that source-column qualification, following Ossie's scoping rule.
  • Model-scoped metrics referencing that dataset's qualified fields. Field and metric names must be unique, ignoring case, in the resulting Spark relation.
  • Portable expression.sql or an OSSIE_SQL_2026 dialect expression; arithmetic, comparisons, boolean logic, CASE, IN, numeric/string/boolean/date casts, SUM, COUNT (including distinct), AVG, MIN, MAX, ABS, ROUND, FLOOR, CEIL, CEILING, COALESCE, and DATE_TRUNC.
  • DATE_TRUNC(unit, input) requires a case-insensitive string literal for the unit. The supported units are year, quarter, month, week, day, hour, minute, and second. Weeks start on Monday. The result is TIMESTAMP_LTZ. Spark uses the session time zone to truncate TIMESTAMP_LTZ values and implicitly convert DATE values and timestamp strings to TIMESTAMP_LTZ. A NULL input returns NULL. TIMESTAMP_NTZ inputs are rejected during analysis.

Other expressions, relationships, multiple datasets, external joins, streaming reads and writes are rejected. SQL Semantic View DDL and SQL discovery are not added; use the management API to register and list definitions. Ordinary tables and SQL views retain their existing lookup paths. Source table authorization is required independently of permission to read the semantic definition. When using a CTE, evaluate measures inside its definition (for example, WITH m AS (SELECT MEASURE(revenue) AS revenue FROM paimon.sales.order_metrics) SELECT revenue FROM m). Forwarding raw measure columns through a CTE or temporary view is not supported by Spark 4.2's Metric View resolver; evaluate measures inside the defining query block before reusing its results.

The bundled schema is pinned to Ossie commit 2692b1cc4708fdcf6a91e597ca928a9bc9d0da60. The 0.3.0.dev0 version string is a development version and does not imply compatibility with future changes under that same string. ai_context and nonportable_extensions remain in the original document and do not alter query execution. Extension data must contain JSON values with string keys. Quote date and timestamp strings; unquoted dates, timestamps, non-finite numbers, and recursive YAML aliases are rejected. Alias expansion is also bounded before JSON validation to prevent small documents from expanding into unbounded in-memory models. Portable SQL string literals follow ANSI quoting, independently of Spark's spark.sql.parser.escapedStringLiterals setting.