parquet_strings Module

Independent, self-contained module for Parquet BYTE_ARRAY/STRING column storage.

Provides two public types:

  • parquet_string_column -- an owning, contiguous Arrow-LargeUtf8-compatible string column (int64 offsets + a packed byte payload + a lazily allocated, bit-packed validity bitmap). Designed for row-group-chunked appends, tens-to-hundreds of millions of rows and multi-gigabyte payloads with a minimal allocation count and good cache locality.
  • parquet_string -- a lightweight, non-owning handle referring to one element of a column (a column pointer + a 1-based index), resolved lazily on access so it survives appends to the column. It owns nothing and frees nothing.

This module depends only on iso_fortran_env and iso_c_binding; it has no dependency on any other module in this library (the read/write integration layer depends on it, never the reverse). See raw_buffers/append_buffers for the buffer-level interop hooks the future Parquet read/write path consumes.


Uses


Interfaces

public interface parquet_string_column_reserve

Typed form of %reserve, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_reserve_i32(self, n_rows, n_characters)

    int32 specific of reserve; see the reserve generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int32), intent(in) :: n_rows

    required row capacity.
    integer(kind=int32), intent(in) :: n_characters

    required character capacity (bytes).

  • private subroutine parquet_string_column_reserve_i64(self, n_rows, n_characters)

    int64 specific of reserve: grows capacity to hold at least n_rows/n_characters (never shrinks; pass 0 for "no requirement on this dimension").

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int64), intent(in) :: n_rows

    required row capacity.
    integer(kind=int64), intent(in) :: n_characters

    required character capacity (bytes).

public interface parquet_string_column_get

Typed form of %get, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_get_i32(self, i, res, null_value, allow_null)

    int32 specific of get; see the get generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(in) :: self

    the column.
    integer(kind=int32), intent(in) :: i

    1-based element index.
    character(len=:), intent(out), allocatable :: res

    element i (unallocated if null and allowed).
    character(len=*), intent(in), optional :: null_value

    substitute returned for a null element.
    logical, intent(in), optional :: allow_null

    .true. => suppress abort, return empty string for a null.

  • private subroutine parquet_string_column_get_i64(self, i, res, null_value, allow_null)

    int64 specific of get: writes element i into res. A null element error stops by default; pass null_value to substitute a string, or allow_null=.true. to suppress the abort and return an empty string (detect null via is_null). When both are given, null_value takes precedence. A subroutine (not a function) so this never returns character(len=:), allocatable as a function result -- see "Build and compiler notes" in CLAUDE.md for why.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(in) :: self

    the column.
    integer(kind=int64), intent(in) :: i

    1-based element index.
    character(len=:), intent(out), allocatable :: res

    element i (unallocated if null and allowed).
    character(len=*), intent(in), optional :: null_value

    substitute returned for a null element.
    logical, intent(in), optional :: allow_null

    .true. => suppress abort, return empty string for a null.

public interface parquet_string_column_copy_to

Typed form of %copy_to, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_copy_to_i32(self, i, dest, allow_null)

    int32 specific of copy_to; see the copy_to generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(in) :: self

    the column.
    integer(kind=int32), intent(in) :: i

    1-based element index.
    character(len=*), intent(out) :: dest

    receives the element, blank-padded.
    logical, intent(in), optional :: allow_null

    .true. => a null yields blanks.

  • private subroutine parquet_string_column_copy_to_i64(self, i, dest, allow_null)

    int64 specific of copy_to: copies element i's bytes into dest, blank-padding the rest.

    The allocation-free counterpart of %get, for the very common shape where the caller already has somewhere fixed-width to put the value — a character(len=N) array being filled row by row, or a scratch buffer reused across a loop. %get must allocate, because it returns a string sized to the element; this cannot and does not.

    It follows Fortran's own assignment semantics exactly, so it is a drop-in for call c%get(i, s); dest = s: shorter values are blank-padded, and a value longer than dest is truncated rather than aborting — which is what dest = s would have done. A caller that must not truncate sizes dest from %length(i) or %max_length() first, both of which allocate nothing either.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(in) :: self

    the column.
    integer(kind=int64), intent(in) :: i

    1-based element index.
    character(len=*), intent(out) :: dest

    receives the element, blank-padded.
    logical, intent(in), optional :: allow_null

    .true. => a null yields blanks.

public interface parquet_string_column_is_null

Typed form of %is_null, generic over an int32 or int64 index/count argument.

  • private function parquet_string_column_is_null_i32(self, i) result(res)

    int32 specific of is_null; see the is_null generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(in) :: self

    the column.
    integer(kind=int32), intent(in) :: i

    1-based element index.

    Return Value logical

  • private function parquet_string_column_is_null_i64(self, i) result(res)

    int64 specific of is_null: whether element i is null (always safe -- the primary null guard).

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(in) :: self

    the column.
    integer(kind=int64), intent(in) :: i

    1-based element index.

    Return Value logical

public interface parquet_string_column_append_from

Typed form of %append_from, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_append_from_i32(self, src, i)

    int32 specific of append_from; see the append_from generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the destination column.
    type(parquet_string_column), intent(in) :: src

    the source column.
    integer(kind=int32), intent(in) :: i

    1-based source element index.

  • private subroutine parquet_string_column_append_from_i64(self, src, i)

    int64 specific of append_from: appends element i of src to the end of self, null state included.

    The allocation-free counterpart of call src%get(i, s); call dst%append_string(s), which is the shape every "copy the rows I want into a new column" loop reaches for and which costs one heap round trip per row for bytes that are already contiguous in src. See feature_risks.md Risk-60.

    It never trims, matching %append_string's own default: the bytes are copied verbatim.

    src must not be the same object as self. Fortran forbids argument-associating one object with both an intent(inout) and an intent(in) dummy of the same call once either is defined (F2018 15.5.2.13), and neither gfortran nor ifx diagnoses it. Appending a column to itself is %append_column(other)'s job, on a copy.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the destination column.
    type(parquet_string_column), intent(in) :: src

    the source column.
    integer(kind=int64), intent(in) :: i

    1-based source element index.

public interface parquet_string_column_set

Typed form of %set, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_set_i32(self, i, str, strip, trim)

    int32 specific of set; see the set generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int32), intent(in) :: i

    1-based element index.
    character(len=*), intent(in) :: str

    the replacement string.
    logical, intent(in), optional :: strip

    remove leading and trailing blanks.
    logical, intent(in), optional :: trim

    remove trailing blanks only.

  • private subroutine parquet_string_column_set_i64(self, i, str, strip, trim)

    int64 specific of set: replaces the content of element i (clearing its null status). Same-length replacement is O(length); a different length shifts the payload tail (O(N)). Same strip/trim options as append_string.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int64), intent(in) :: i

    1-based element index.
    character(len=*), intent(in) :: str

    the replacement string.
    logical, intent(in), optional :: strip

    remove leading and trailing blanks.
    logical, intent(in), optional :: trim

    remove trailing blanks only.

public interface parquet_string_column_set_null

Typed form of %set_null, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_set_null_i32(self, i)

    int32 specific of set_null; see the set_null generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int32), intent(in) :: i

    1-based element index.

  • private subroutine parquet_string_column_set_null_i64(self, i)

    int64 specific of set_null: sets element i to null, discarding any existing content (shrinks its payload span to zero width, shifting the tail left by the same amount set_i64 would for a same-index replacement with a shorter string). Idempotent: calling this on an already-null element leaves it null without double-counting null_count().

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int64), intent(in) :: i

    1-based element index.

public interface parquet_string_column_reindex

Typed form of %reindex, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_reindex_i32(self, perm)

    int32 specific of reindex; see the reindex generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int32), intent(in) :: perm(:)

    1-based permutation of 1..size().

  • private subroutine parquet_string_column_reindex_i64(self, perm)

    int64 specific of reindex: reorders every element so that element k of the result is the element that was at perm(k) before the call, rebuilding the payload, offsets and validity in one O(nchars) pass. perm must be a true permutation of 1..size(); it is fully validated first (length, range, no duplicates), so a bad permutation aborts before any buffer is touched and the column is left unchanged. The null count is preserved by construction. Invalidates every outstanding handle into the column.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int64), intent(in) :: perm(:)

    1-based permutation of 1..size().

public interface parquet_string_column_reindex_trusted

Typed form of %reindex_trusted, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_reindex_trusted_i32(self, perm)

    int32 specific of reindex_trusted; see the reindex_trusted generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int32), intent(in) :: perm(:)

    1-based permutation of 1..size().

  • private subroutine parquet_string_column_reindex_trusted_i64(self, perm)

    int64 specific of reindex_trusted: reindex without the O(n) range/duplicate scan, for a permutation the caller has already established is one. The O(1) length check still runs.

    Public only because Fortran has no narrower visibility, and reached from exactly two places: parquet_column%reindex_trusted (which is how parquet_table%sort_by avoids re-validating one permutation once per column) and pf_permute(..., assume_valid=.true.). A caller who passes a non-permutation gets silently duplicated and dropped elements, so this is internal plumbing rather than an alternative to %reindex.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int64), intent(in) :: perm(:)

    1-based permutation of 1..size().

public interface parquet_string_column_gather

Typed form of %gather, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_gather_i32(self, idx)

    int32 specific of gather; see the gather generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int32), intent(in) :: idx(:)

    1-based source index per destination element.

  • private subroutine parquet_string_column_gather_i64(self, idx)

    int64 specific of gather: rebuilds the column so that element k is the element that was at idx(k), for an index list of ANY length.

    This is the subset-and-reorder primitive reindex and delete_by_mask do not provide between them: reindex demands a permutation of the whole column, and delete_by_mask keeps the existing order. idx may name any element in 1..size(), in any order, and may name one more than once -- it is a gather, not a permutation, so the result may be shorter than, as long as, or longer than the column it replaces.

    Only the range is checked. Refusing repeats would need a seen-set sized by the SOURCE element count on every call, which is exactly the cost this primitive exists to avoid; a caller that needs distinctness (parquet_table%top_n does) checks it once for itself.

    Rebuilds into fresh buffers rather than compacting in place, because a reordering write cursor can overtake its own read cursor -- which is why delete_by_mask, whose output order is the input order, may compact in place and this may not. Two further differences from reindex_apply, both consequences of the length being free to change: the payload is sized to the SELECTED characters rather than to nchars, and n_null is recounted rather than carried over.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int64), intent(in) :: idx(:)

    1-based source index per destination element.

public interface parquet_string_column_append_nulls

Typed form of %append_nulls, generic over an int32 or int64 index/count argument.

  • private subroutine parquet_string_column_append_nulls_i32(self, n)

    int32 specific of append_nulls; see the append_nulls generic.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int32), intent(in) :: n

    number of null elements to append.

  • private subroutine parquet_string_column_append_nulls_i64(self, n)

    int64 specific of append_nulls: appends n null (zero-width, invalid) elements in one bulk operation -- the counterpart of calling append_null n times, but with a single capacity growth instead of one per element. n == 0 is a no-op; a negative n aborts.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_string_column), intent(inout) :: self

    the column.
    integer(kind=int64), intent(in) :: n

    number of null elements to append.

Derived Types

type, public ::  parquet_string_column

An owning, Arrow-LargeUtf8-compatible variable-length string column.

Read more…

Type-Bound Procedures

procedure, public :: clear

Reset to empty and release all owned buffers.
generic, public :: reserve => reserve_i32, reserve_i64

Grow capacity for at least n_rows/n_characters.
procedure, public :: shrink_to_fit

Reallocate buffers down to the current size.
procedure, public :: capacity

Current row capacity.
procedure, public :: character_capacity

Current character-buffer capacity (bytes).
procedure, public :: size => col_size

Number of elements stored.
procedure, public :: character_size

Total characters stored.
procedure, public :: empty => col_empty

.true. when no rows are stored.
procedure, public :: null_count

Number of null elements.
procedure, public :: has_validity

Whether the validity bitmap exists yet.
procedure, public :: reserve_validity

Materialize the validity bitmap up front.
procedure, public :: memory_usage

Total bytes of allocated buffers.
procedure, public :: validate

Verify class invariants.
generic, public :: length => length_i32, length_i64

Length of element i (no allocation).
generic, public :: get => get_i32, get_i64

Writes element i into an allocatable string.
generic, public :: copy_to => copy_to_i32, copy_to_i64

Copies element i into a fixed-length slot.
generic, public :: view => view_i32, view_i64

Zero-copy handle to element i.
procedure, public :: view_all

One handle per element, in order.
generic, public :: view_slice => view_slice_i32, view_slice_i64

One handle per row of [first, last].
generic, public :: is_null => is_null_i32, is_null_i64

Whether element i is null.
generic, public :: is_empty => is_empty_i32, is_empty_i64

Whether element i has zero length.
procedure, public :: append_string

Append a string to the end.
procedure, public :: append_null

Append a null element to the end.
procedure, public :: append_column

Append all elements from another column.
procedure, public :: append_values

Bulk-append a character array, trimming each element.
generic, public :: append_from => append_from_i32, append_from_i64

Append one element of another column.
generic, public :: build_from => build_from_handles, build_from_character

Clears self, then fills it -- from an array of handles, or from a character array (trailing blanks trimmed, one optional null mask). Both replace the whole column.
generic, public :: set => set_i32, set_i64

Replace the content of element i.
generic, public :: set_null => set_null_i32, set_null_i64

Sets element i to null (discards any content).
generic, public :: erase => erase_i32, erase_i64

Remove element i (shifts later elements down).
generic, public :: reindex => reindex_i32, reindex_i64

Reorder every element by a permutation.
generic, public :: reindex_trusted => reindex_trusted_i32, reindex_trusted_i64

INTERNAL -- reindex without the duplicate/range scan. Public only because Fortran offers no narrower visibility -- see reindex_trusted_i64.
procedure, public :: delete_by_mask

Keep only the elements whose mask entry is .true.
generic, public :: gather => gather_i32, gather_i64

Keep the listed elements, in the listed order.
generic, public :: append_nulls => append_nulls_i32, append_nulls_i64

Append n null elements in bulk.
procedure, public :: strip_all

Strip both ends of every non-null element.
procedure, public :: trim_all

Trailing-trim every non-null element.
procedure, public :: find

Index of first/last element equal to str.
generic, public :: contains => contains_i32, contains_i64

Whether element i contains a substring.
generic, public :: startswith => startswith_i32, startswith_i64

Whether element i begins with prefix.
generic, public :: endswith => endswith_i32, endswith_i64

Whether element i ends with suffix.
generic, public :: equals => equals_i32, equals_i64

Whether element i equals str.
procedure, public :: compare

Orders element i against element j.
procedure, public :: to_character

Materialize the whole column as a char array.
procedure, public :: clone

Independent deep copy.
generic, public :: slice => slice_i32, slice_i64

Independent, owning copy of rows [first, last].
procedure, public :: move_from

Transfer all buffers from another column.
procedure, public :: swap

Exchange contents with another column.
procedure, public :: print => col_print

Human-readable representation.
procedure, public :: summary

Writes a compact one-line overview string.
procedure, public :: statistics

Detailed metrics (optional out-args).
procedure, public :: raw_buffers

Export c_loc pointers to the internal buffers.
procedure, public :: copy_buffers

Copy the offsets and packed payload into caller arrays.
procedure, public :: append_buffers

Bulk-append one row group from C buffers.

type, public ::  parquet_string

A lightweight, non-owning handle to one element of a parquet_string_column.

Read more…

Type-Bound Procedures

procedure, public :: length => psv_length

Length of the referenced string.
procedure, public :: is_empty => psv_is_empty

Whether the referenced string has zero length.
procedure, public :: is_null => psv_is_null

Whether the referenced element is null.
procedure, public :: set_null => psv_set_null

Sets the referenced column element to null.
procedure, public :: to_string => psv_to_string

Writes the referenced string into an argument.
procedure, public :: equals => psv_equals

Exact comparison against str.
procedure, public :: contains => psv_contains

Substring search for str.
procedure, public :: startswith => psv_startswith

Prefix test.
procedure, public :: endswith => psv_endswith

Suffix test.
procedure, public :: print => psv_print

Human-readable representation.

Functions

public function parquet_string_threads() result(n)

How many threads one parquet_string_column bulk operation would use here, right now.

Read more…

Arguments

None

Return Value integer

public function parquet_debug_string_bulk_threads(col) result(n)

What a bulk operation over col would actually resolve to, floor and all. Test-only.

Read more…

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(in) :: col

the column an operation would walk.

Return Value integer

public function parquet_string_column_capacity(self)

Returns the current row capacity.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(in) :: self

the column.

Return Value integer(kind=int64)

public function parquet_string_column_character_capacity(self)

Returns the current character-buffer capacity in bytes.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(in) :: self

the column.

Return Value integer(kind=int64)

public function parquet_string_column_size(self)

Returns the number of elements stored.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(in) :: self

the column.

Return Value integer(kind=int64)

public function parquet_string_column_character_size(self)

Returns the total number of characters stored across all elements.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(in) :: self

the column.

Return Value integer(kind=int64)

public function parquet_string_column_null_count(self)

Returns the number of null elements.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(in) :: self

the column.

Return Value integer(kind=int64)

public function parquet_string_column_has_validity(self)

Whether the validity bitmap has been materialized yet.

Read more…

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(in) :: self

the column.

Return Value logical


Subroutines

public subroutine parquet_debug_set_string_min_bytes(n)

Overrides the payload floor below which a bulk operation stays serial. Test-only; <= 0 restores the real STRING_MIN_BYTES.

Read more…

Arguments

Type IntentOptional Attributes Name
integer(kind=int64), intent(in) :: n

new floor in bytes, or <= 0 to restore the real one.

public subroutine parquet_debug_set_string_max_auto_threads(n)

Overrides the ceiling on the AUTOMATIC thread count. Test-only; <= 0 restores the real STRING_MAX_AUTO_THREADS. Public for the same reason its sibling above is.

Arguments

Type IntentOptional Attributes Name
integer, intent(in) :: n

new ceiling, or <= 0 to restore the real one.

public subroutine parquet_debug_string_row_ranges(n, nt, lo, hi)

Exposes thread_row_ranges for testing. Test-only; no library code calls it.

Read more…

Arguments

Type IntentOptional Attributes Name
integer(kind=int64), intent(in) :: n

total rows.
integer, intent(in) :: nt

number of ranges.
integer(kind=int64), intent(out), allocatable :: lo(:)

first row of each range.
integer(kind=int64), intent(out), allocatable :: hi(:)

last row of each range.

public subroutine parquet_string_column_clear(self)

Resets the column to an empty state and releases all owned memory (capacity becomes 0). Invalidates every outstanding handle into this column.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(inout) :: self

the column.

public subroutine parquet_string_column_shrink_to_fit(self)

Reallocates the buffers down to exactly the current size (frees unused capacity). Invalidates every outstanding handle into this column.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(inout) :: self

the column.

public subroutine parquet_string_column_reserve_validity(self)

Materializes the validity bitmap now, with every element still valid.

Read more…

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(inout) :: self

the column.

public subroutine parquet_string_column_append_null(self)

Appends a null element to the end of the column (a zero-width, invalid slot).

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(inout) :: self

the column.

public subroutine parquet_string_column_append_column(self, other)

Appends all elements (payload and nulls) from another column. Bulk-copies the payload and offsets; never re-trims. other is left unchanged.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(inout) :: self

the destination column.
type(parquet_string_column), intent(in) :: other

the source column.

public subroutine parquet_string_column_append_values(self, values, is_null)

Bulk-appends a character array, trimming each element's trailing blanks -- the appending counterpart of build_from's character form, sharing its packing walk and existing for the same reason: parquet_column%append_values on a string column used one %append_string call per element. See build_from_character for why the byte-view copy is what makes this fast and why len_trim must stay on the element view.

Read more…

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(inout) :: self

the column, appended to.
character(len=*), intent(in), contiguous :: values(:)

elements to append; blanks trimmed.
logical, intent(in), optional :: is_null(:)

.true. => append that element as null.

public subroutine parquet_string_column_delete_by_mask(self, keep)

Keeps only the elements whose keep entry is .true., in order, compacting payload, offsets and validity in one O(nchars) in-place pass. keep must have exactly size() entries. Dropping every element leaves a valid empty column. Invalidates every outstanding handle into the column. Note this is the bulk counterpart of erase: deleting m elements one at a time costs O(m*nchars), this costs O(nchars) once.

Arguments

Type IntentOptional Attributes Name
type(parquet_string_column), intent(inout) :: self

the column.
logical, intent(in) :: keep(:)

.true. for every element to retain.