Skip to main content

Vector Storage

Overview​

Use VECTOR for fixed-length dense numeric data such as model embeddings. This page shows how to create, write, and read vector columns, including dedicated storage using the Vortex columnar format.

Storage and similarity search are configured separately. After loading vectors, build a Vector Index to use ANN search. Vector indexes can also index ARRAY<FLOAT> columns without dedicated vector files.

Vector Data Type​

Paimon supports defining columns of type VECTOR<t, n>, which represents a fixed-length, dense vector column, where:

  • t: The element type. Supports: BOOLEAN, TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE;
  • n: The vector dimension, must be a positive integer not exceeding 2,147,483,647.

Compared to variable-length arrays, dense vectors provide:

  • More natural semantic constraints, preventing mismatched lengths and null elements at the storage layer;
  • Better point-lookup performance, eliminating offset array storage and access;
  • Closer alignment with type representations in specialized vector engines, avoiding memory copies and type conversions.

Notes:

  • Columns of VECTOR type cannot be used as primary key columns, partition columns, or for sorting.
  • If a VECTOR value itself is not null, its elements are not allowed to be null.

Dedicated Vector File Storage​

Paimon stores vector columns in separate .vector.vortex files within Data Evolution tables, keeping scalar and vector data independently optimized.

File layout:

table/
├── bucket-0/
│   ├── data-uuid-0.parquet       # Scalar columns (id, name, ...)
│   ├── data-uuid-1.blob          # Blob data
│   ├── data-uuid-2.vector.vortex # Vector columns in Vortex format
│   └── ...
├── manifest/
├── schema/
└── snapshot/
OptionDescription
vector.file.formatFile format for dedicated vector files. Set to vortex to enable dedicated vector storage.
vector.target-file-sizeTarget file size for vector files. Defaults to target-file-size.
row-tracking.enabledMust be true for dedicated vector storage.
data-evolution.enabledMust be true for dedicated vector storage.

Create Table​

The single-vector examples use dimension 3 so the write examples are complete. Use the dimension produced by your embedding model in an application.

The recommended way to create a vector table in SQL is to use the comment directive __VECTOR_FIELD;dim on the column. Paimon automatically converts the ARRAY type to VECTOR and registers the field in the vector-field option.

-- Comment directive: __VECTOR_FIELD;{dim}; optional comment
CREATE TABLE vector_table (
    id BIGINT,
    embed ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;3; product embedding'
) WITH (
    'vector.file.format' = 'vortex',
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

-- Multiple vector columns
CREATE TABLE multi_vector_table (
    id BIGINT,
    embed1 ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;128',
    embed2 ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;768'
) WITH (
    'vector.file.format' = 'vortex',
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true'
);

Write Data​

INSERT INTO vector_table VALUES (1, ARRAY[CAST(1.0 AS FLOAT), CAST(2.0 AS FLOAT), CAST(3.0 AS FLOAT)]);

Read Data​

SELECT id, embed FROM vector_table;

Adding a Vector Column​

This is a separate schema-change example to run after the create/write/read walkthrough. Add the column, then backfill it with Data Evolution.

ALTER TABLE vector_table
    ADD embed2 ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;768; text embedding';