String Operations

Complete reference for string manipulation in Rayforce — from basic transforms to pattern matching, covering both RAY_SYM and RAY_STR column types.

RAY_SYM vs RAY_STR

Rayforce provides two distinct string column representations, each optimized for different workloads. Choosing the right type is critical for performance.

RAY_SYM — Dictionary-Encoded Symbols

RAY_SYM columns store strings as integer indices into a global intern table. Ideal for low-cardinality categorical data (country codes, status flags, product categories).

• Adaptive-width indices — 8, 16, 32, or 64-bit integers depending on dictionary size
• Equality comparison is a single integer compare — O(1)
• Group-by on SYM columns uses direct index lookup instead of hashing
• Memory efficient — each row stores only 1–8 bytes regardless of string length
• Global intern table — symbols are shared across all columns and tables

; Create a table with a SYM column (default for short repeated strings in CSV)
ray> (set t (read-csv "trades.csv"))
; region column is automatically SYM — only 4 unique values across 1M rows

RAY_STR — Variable-Length Strings

RAY_STR columns store variable-length strings with a hybrid inline/pool layout. Best for high-cardinality or unique text data (names, descriptions, URLs).

• SSO (Small String Optimization) — strings of 12 bytes or fewer are stored inline in the 16-byte ray_str_t element, requiring zero indirection
• Pool storage — strings longer than 12 bytes are written to a per-vector pool; the element stores a 4-byte offset and 4-byte length, plus a 4-byte prefix copied inline for fast comparison rejection
• Fast comparison rejection — the 4-byte prefix allows most unequal comparisons to short-circuit without following the pool pointer
• Per-vector pool — each RAY_STR vector has its own pool; col_propagate_str_pool() shares pools between source and destination during execution

; STR columns are used for unique/high-cardinality text
ray> (set names (vec-str ["Alice" "Bob" "Charlie"]))
; "Alice" (5 bytes) → stored inline (SSO)
; "A longer description here" (26 bytes) → stored in pool with 4-byte prefix

Null Propagation

All string operations in Rayforce follow strict null propagation semantics:

• Null input produces null output — if any required input row is null, the output row is null
• CONCAT is null if any argument is null — concat("hello", NULL, "world") returns NULL, not "helloworld"
• Null propagation applies uniformly to both RAY_SYM and RAY_STR columns
• Null bitmaps are carried through the execution pipeline per morsel (1024 elements)

ray> (set t (table [name] (list ["Alice" 0N "Charlie"])))
ray> (select {from:t cols: {upper_name: (upper name)}})
upper_name
----------
ALICE
0N
CHARLIE

String Functions

upper

lower

strlen

trim

concat

substr

replace

like

ilike

split

format

String Operations in the DAG

When using the C API, string operations are available as DAG opcodes. These are fused into morsel-driven execution alongside arithmetic and comparison operations.

C API Example

/* Filter rows where upper(name) LIKE "A%" and compute strlen */
ray_graph_t* g = ray_graph_new(table);

ray_op_t* name    = ray_scan(g, "name");
ray_op_t* up_name = ray_upper(g, name);
ray_op_t* pattern = ray_const_str(g, "A%", 2);
ray_op_t* pred    = ray_like(g, up_name, pattern);

ray_op_t* filt_name = ray_filter(g, name, pred);
ray_op_t* name_len  = ray_strlen(g, filt_name);

/* Execute — upper, like, filter, strlen all fused into one morsel pass */
ray_t* result = ray_execute(g, ray_optimize(g, name_len));

String Pool Internals

Understanding the internal layout helps explain performance characteristics of string operations.

ray_str_t Element Layout (16 bytes)

Hash and Comparison

String hashing uses ray_str_t_hash() which operates directly on the element bytes. Comparison via ray_str_t_cmp() / ray_str_t_eq() first compares the 4-byte prefix for fast rejection, then falls through to a full byte comparison only when prefixes match. This makes hash joins and group-by on string columns significantly faster than naive approaches.

Dictionary-Encoded Symbol Width

RAY_SYM columns use adaptive-width integer indices to minimize memory:

Width is set at column creation via ray_sym_vec_new(sym_width, capacity) where sym_width is 1, 2, 4, or 8. The CSV reader picks the narrowest width that fits the observed cardinality.