parquet_tables Module

A whole parquet file as one in-memory table: parquet_table.

parquet_table sits on top of the parquet reader/writer rather than replacing it. Opening one reads the file's SCHEMA and nothing else; each column's values are read into the type-erased parquet_column store (parquet_columns) the first time something asks for them, and the Arrow-side buffers are freed as soon as that copy exists, so the table owns the sole Fortran copy of each column it holds. From there a column is reached either by a zero-copy typed pointer (%col, exact kind) or by a widening copy (%get), a table can be built from scratch in memory (parquet_new_table + %add_column), and the whole thing is written back out through an ordinary parquet_schema (parquet_write_table).

Five things are worth knowing before using it:

  • %get is the friendly path; %col is the fast one. %get copies the column into an allocatable array of the caller's own kind, widening int32 -> int64 and float32 -> float64 on the way, so a caller who just wants the numbers never has to ask what type the file used. %col hands back a pointer straight into the store -- zero copy, writable -- but the pointer kind must match the stored kind EXACTLY, so it is for code that already knows the type (or has asked %kind).
  • Reads happen on first touch. %nrows/%kind/%width/%column_names answer from the schema and read nothing; a value access reads that column, whole, across the table's row scope. %residency reports what is held, %prefetch/%materialize_all read ahead of time, and %reload goes back to the file.
  • A table can cover part of a file. parquet_open_table(t, file, row_lo, row_hi) reads only the row groups covering that range, which is how a file bigger than memory is worked through and how a parallel program gives each thread its own share.
  • Assignment is blocked. The column store lives behind a pointer, so b = a would leave two tables sharing (and later double-freeing) one store. b = a is a hard error rather than a silent corruption; copying a table comes with %clone in a later milestone.
  • Concurrency is enforced, not merely documented. Reading an already-resident column is free -- no lock, no atomic, any number of threads -- and %append into a shared table is serialised by the table's own lock, so a parallel producer needs no !$omp critical. Every other change to a table another thread may be using is a hard error naming what to do instead: a lazy first touch, any structural change, nulling a column whose validity storage does not exist yet, and any write to a string column (whose rows share one packed store). A table a thread opened ITSELF inside the region is thread-private and exempt from all of them. See parquet_tables_parallel.f90 and doc/pages/operating/thread-safety.md.

Depends on parquet_columns (the value store) and parquet_core (the reader/writer it drives). Re-exported from parquet_settings so that a use parquet_tables program can report which Arrow/Parquet C++ it is linked against without a second import. The library's OWN version is not re-exported here -- that is parquet_get_version, in the leaf module parquet_version, carried only by the parquet facade. TEST-ONLY debug hook; deliberately NOT in README.md's API overview. See its own doc-comment for why it has to be public at all.



Variables

Type Visibility Attributes Name Initial
character(len=*), public, parameter :: PARQUET_ROW_INDEX = "parquet_row_index"

The name of the automatic column holding each row's PHYSICAL row number in the source parquet file. Declared as a constant so a program can name it without hard-coding the string, and so a collision test has something to compare against.

integer, public, parameter :: RES_EMPTY = 0

no values held (never read, or an unsupported type).

integer, public, parameter :: RES_PARTIAL = 1

reserved: some row groups resident (a later milestone).

integer, public, parameter :: RES_FULL = 2

the whole column, across the table's row scope, is held.


Interfaces

public interface parquet_open_table

Opens a file-backed table, over the whole file or over one contiguous row slice.

Given row_lo/row_hi (1-based, inclusive, in either integer kind) the table covers only those rows, and reads only the row groups covering them -- this is how a file bigger than memory is worked through, and how a parallel program gives each thread its own share (parquet_table_row_group_bounds reports where the natural boundaries are). Row indices everywhere else, %row(i) included, are then relative to the slice, not to the file.

A slice accepts the same read-time transform the whole-file form does, with ONE exception: there is no sort argument on the slice forms at all. A sort reorders rows across the whole file, so "the 1000th row" would no longer name anything a slice could be cut along -- and omitting the argument makes that a compile error rather than a runtime one. (A maml= whose extra: sort: list is non-empty is the same rejection, necessarily at runtime.)

A filtered or sampled slice does not have row_hi - row_lo + 1 rows. %nrows() is the number of rows of [row_lo, row_hi] that survive the transform, and every row index the table takes or reports counts those survivors -- row 1 is the first surviving row, not file row row_lo. Without a filter and without sample_fraction= nothing changes: the slice is trimmed out of the covering row groups in memory, exactly as it always was.

  • private interface open_table_full()

    Arguments

    None
  • private interface open_table_slice_i32()

    Arguments

    None
  • private interface open_table_slice_i64()

    Arguments

    None

public interface parquet_slice_range

Builds a start:stop:step slice. stop defaults to the table's last row (resolved when the slice is USED, not when it is built, so one slice object can outlive a row count), step to 1. A negative step counts down; a zero step is an error.

  • private interface slice_range_i32()

    Arguments

    None
  • private interface slice_range_i64()

    Arguments

    None

public interface parquet_slice_list

Builds a slice from an explicit list of 1-based row indices, in the order given -- repeats and non-monotone order are both allowed, since this is a gather, not a range.

  • private interface slice_list_i32()

    Arguments

    None
  • private interface slice_list_i64()

    Arguments

    None

interface

  • public module subroutine parquet_new_table(table)

    Prepares an empty in-memory table with no columns and no rows. The first %add_column fixes the row count; every later one must match it.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_table), intent(out) :: table

    the table to initialize.

interface

  • public module subroutine parquet_table_row_group_bounds(filename, bounds)

    Reports each row group of filename as the inclusive 1-based row range it covers: bounds(1, rg) is its first row and bounds(2, rg) its last. Together they partition 1..nrows exactly.

    Standalone on purpose: this is the PLANNING call, made before any table exists, so that each thread can work out which slice to open. It opens and closes a reader internally, which costs only a footer read -- no column data is touched.

    Arguments

    Type IntentOptional Attributes Name
    character(len=*), intent(in) :: filename

    parquet file to inspect.

    integer(kind=int64), intent(out), allocatable :: bounds(:,:)

    (2, num_row_groups).

interface

  • public module subroutine parquet_debug_table_set_inflight(table, appending, reading)

    TEST-ONLY -- forces this table's "an append is in flight"/"a read is in flight" counters, so the two concurrency aborts can be provoked from ONE thread, deterministically.

    This is a debug hook, not API. It exists because the guards it drives (table_check_no_append, and %append's own reader check) can otherwise only be triggered by two threads overlapping on demand, and a timing-dependent test is worse than no test -- it fails on a busy machine and gets disabled. Unlike the C++ parquet_debug_* hooks, which a test reaches through its own local bind(C) interface, a Fortran-side hook has no such escape hatch: these counters live on parquet_table_cache, whose components are private to this module, so forcing them requires a public procedure here. That cost was accepted deliberately (feature_risks.md Risk-6); it is excluded from README.md's API overview and no library code calls it.

    Both arguments are optional and independent: appending=.true. makes every READ on this table abort, reading=.true. makes every %append abort. Pass .false. to clear.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_table), intent(in) :: table

    the table whose counters to force.

    logical, intent(in), optional :: appending

    .true.: pretend an append is in flight.

    logical, intent(in), optional :: reading

    .true.: pretend a read is in flight.

interface

  • public module subroutine parquet_debug_colread_block_rows(lo, hi, width, mid_lo, mid_hi)

    Exposes the parallel single-column read's validity-block alignment arithmetic for testing.

    This is a debug hook, not API, public for the same reason parquet_debug_table_set_inflight is: the procedure it forwards to lives in a submodule and the property it computes cannot be observed from outside.

    What it computes is which rows of lo..hi occupy WHOLE validity-bitmap blocks, and it is the one thing standing between the parallel column read and a silent wrong answer -- two threads pasting adjacent row groups share a bitmap block unless their ranges are trimmed to this. That race is a few instructions wide, so an end-to-end test cannot be relied on to catch a mistake in it; this makes the rule itself assertable, exactly as parquet_debug_string_row_ranges does for parquet_string_column's byte-aligned split. mid_lo > mid_hi reports that no whole block exists in the range.

    Arguments

    Type IntentOptional Attributes Name
    integer(kind=int64), intent(in) :: lo

    first row of the range.

    integer(kind=int64), intent(in) :: hi

    last row of the range.

    integer(kind=int64), intent(in) :: width

    elements per row.

    integer(kind=int64), intent(out) :: mid_lo

    first row occupying a whole block.

    integer(kind=int64), intent(out) :: mid_hi

    last such row; < mid_lo when there is none.

interface

  • public module subroutine parquet_write_table(table, filename, schema, row_mask, copy_metadata, metadata_keys, write_maml, qc, compression, compression_level, chunk_size, use_threads, overwrite, release)

    Writes table to filename using schema to choose and name the output columns: every enabled schema field is looked up in the table BY ITS INTERNAL NAME, and written under its own output name (a col_map: rename is honoured automatically). A schema field with no matching table column is an error; a table column the schema does not name is simply not written. row_mask writes a row subset without changing the table.

    schema is optional. Without one the write is schema-less: every column that is currently RESIDENT is written, in slot order, under its own internal name -- a quick path for a small or temporary table that reads nothing and needs no schema built for it. A table with nothing resident writes a valid empty file. The automatic parquet_row_index column is never written by a schema-less write, even when it is resident; name it in a schema to write it. Without a schema there is no col_map: and no qc:, so output names are the internal names and writer-side qc is off.

    A schema built with %init/%add_field and never parsed is parsed here, so calling parquet_parse_maml first is optional. That is why schema is intent(inout): the caller's schema is parsed on return.

    copy_metadata=.true. carries every key/value metadata entry of the table's SOURCE FILE into the output; metadata_keys= carries only the listed keys (and error stops on one the source does not have). The two are mutually exclusive. A key the schema itself declares wins and is not overwritten -- the schema is the explicit statement. Both work after the table has detached, since the metadata was snapshotted at open, and neither adds anything to the caller's own schema.

    write_maml, qc, compression, compression_level, chunk_size, use_threads and overwrite are pass-throughs to parquet_open_writer with its own defaults and no reinterpretation, so a table write and the equivalent hand-written open stay the same calls. chunk_size is default-kind integer on purpose: a row group cannot hold more than int32 rows, so there is no int64 form to add.

    release (default .true.) leaves the table in the residency state it started in: a column this write had to materialize is evicted again once it has been written, while a column the caller had already read is left alone. Releasing frees storage a %col pointer could alias, so the generation counter advances when at least one column was actually released -- possibly a false alarm (nothing you can hold a pointer to is ever released), never a missed one.

    Arguments

    Type IntentOptional Attributes Name
    class(parquet_table), intent(in) :: table

    the table to write (any extending type too).

    character(len=*), intent(in) :: filename

    output parquet file.

    type(parquet_schema), intent(inout), optional :: schema

    output schema; absent = schema-less write.

    logical, intent(in), optional :: row_mask(:)

    per-row write mask.

    logical, intent(in), optional :: copy_metadata

    .true.: carry every source-file metadata entry.

    character(len=*), intent(in), optional :: metadata_keys(:)

    carry only these source-file keys.

    logical, intent(in), optional :: write_maml

    also save a sidecar .maml next to filename.

    logical, intent(in), optional :: qc

    run the schema's qc: checks on write; defaults to on.

    character(len=*), intent(in), optional :: compression

    Arrow compression codec name (e.g. "snappy").

    integer, intent(in), optional :: compression_level

    codec-specific compression level.

    integer, intent(in), optional :: chunk_size

    Parquet row-group size, in rows.

    logical, intent(in), optional :: use_threads

    use Arrow's multi-threaded writer.

    logical, intent(in), optional :: overwrite

    allow truncating an existing file; default .true.

    logical, intent(in), optional :: release

    evict columns this write materialized; default .true.

interface

  • public module subroutine parquet_debug_table_drop_name_index(table, had_index)

    TEST-ONLY -- deallocates this table's name index, so the next lookup has to take cache_find's linear-scan fallback.

    This is a debug hook, not API, public for the same reason parquet_debug_table_set_inflight is: the index lives on parquet_table_cache, whose components are private to this module, so nothing outside can reach it. It is excluded from README.md's API overview and no library code calls it.

    It exists because that fallback is a SAFETY NET with no route to it through the public API: every mutation maintains the index eagerly, so a correct library never reaches the scan, and it was measured executing zero times across the whole suite and every error scenario. What it protects against is a future mutation that forgets to maintain the index -- which would otherwise turn into wrong-column answers rather than a slower lookup. Untested, the net could rot away and nothing would say so.

    had_index is not optional on purpose: dropping the index is invisible from outside, so a test that did not check it would pass just as happily against a hook that did nothing. It reports whether an index was there to drop, which makes the before/after states assertable -- .true. on the first call, .false. on a second one.

    Arguments

    Type IntentOptional Attributes Name
    type(parquet_table), intent(in) :: table

    the table whose index to drop.

    logical, intent(out) :: had_index

    .true. when an index was present.


Derived Types

type, public ::  parquet_slice

Which rows to pick out of a column: 1:, 1:10, 1:10:2 or an explicit list.

Read more…

type, public ::  parquet_table

A whole table: a column store plus the row scope and provenance describing it. Declared by the caller (type(parquet_table) :: t), filled by parquet_open_table or parquet_new_table, and freed automatically when it goes out of scope.

Finalizations Procedures

final :: table_finalize

Frees the store; never fails, never validates.

Type-Bound Procedures

procedure, public :: nrows => table_nrows

Number of rows every column holds.

procedure, public :: ncols => table_ncols

Number of columns the table has.

procedure, public :: column_names => table_column_names

Copy out every column name, in order.

procedure, public :: column_index => table_column_index

A column's 1-based position, 0 when absent.

procedure, public :: column_name => table_column_name

Copy out the name at a 1-based position.

generic, public :: column => column_by_name, column_by_index

A resolved handle on one column, by name or by 1-based position. Resolves once -- the lookup, the lazy first touch and the kind -- so a per-element loop over that column stops paying for a name lookup on every access. See parquet_table_col.

generic, public :: has_nulls => has_nulls_name, has_nulls_at

Whether a column holds (or may hold) nulls -- named, or by 1-based position.

generic, public :: get_valid_mask => table_get_valid_mask, table_get_valid_mask_elem

Copy out a column's validity as a plain logical array. A rank-1 mask gives one entry per row (on a *_VEC column: "any element of the row is null"); a rank-2 mask gives the true (width, nrows) per-element state.

procedure, public :: generation => table_generation

Counter bumped by every structural change.

procedure, public :: has_column => table_has_column

Whether a column of this name exists.

generic, public :: missing_columns => missing_columns_string, missing_columns_array

Which of these columns the table does NOT have, as a packed array (zero-size when it has them all). The non-aborting half of %require_columns.

generic, public :: require_columns => require_columns_string, require_columns_array

Aborts unless the table has every one of these columns, naming EVERY missing one -- not just the first, which is what a hand-written loop reports.

generic, public :: kind => kind_name, kind_at

A column's PK_* kind discriminator -- named, or by 1-based position.

generic, public :: width => width_name, width_at

A column's values-per-row (1 if scalar) -- named, or by 1-based position.

generic, public :: unit => unit_name, unit_at

Copy out a column's unit string -- named, or by 1-based position.

generic, public :: residency => residency_name, residency_at

A column's RES_* residency state -- named, or by 1-based position.

generic, public :: is_null => is_null_i32, is_null_i64, is_null_e32, is_null_e64, is_null_at_i32, is_null_at_i64, is_null_at_e32, is_null_at_e64

Whether row i of a column is null, or -- given e as well -- element e of it.

Read more…
procedure, public :: is_detached => table_is_detached

Whether the table has left its file behind.

generic, public :: is_supported => is_supported_name, is_supported_at

Whether a column's type can be read -- named, or by 1-based position.

procedure, public :: filename => table_filename

Copy out the file this table came from.

procedure, public :: get_file_metadata => table_get_file_metadata

One key from the file's metadata.

generic, public :: prefetch => prefetch_string, prefetch_array

Reads the named column(s) now, instead of on first touch. Required before a parallel region: a first touch inside one is a hard error, since it would mutate shared state.

generic, public :: materialize => prefetch_string, prefetch_array

The same call as %prefetch, under the name that pairs with %materialize_all. Reaching for the definitive-sounding %materialize_all when only a few columns are wanted reads the whole file, silently; %materialize(names) is the one to find first.

procedure, public :: materialize_all => table_materialize_every

Read every column not yet read.

procedure, public :: reload => table_reload

Re-read one column; force= to discard local edits.

procedure, public :: evict_column => table_evict_column

Drop a column's VALUES; force= if it holds local edits.

procedure, public :: set_user_populated => table_set_user_populated

Claim a column's values as the caller's own, or unclaim.

procedure, public :: is_user_populated => table_is_user_populated

Whether a column is claimed as holding the caller's values.

procedure, public :: validate_qc => table_validate_qc

Check every qc-declaring column, holding none.

procedure, public :: print_stat => table_print_stat

Print what the table holds, to stdout.

procedure, public :: nrows_unfiltered => table_nrows_unfiltered

Rows before filter=/sample_fraction=.

procedure, public :: row_group_extent => table_row_group_extent

Rows in the row groups this table covers.

procedure, public :: row_group_bounds => table_row_group_bounds

Row-group row ranges, this table's rows or the file's.

generic, public :: row => row_at_i32, row_at_i64

A handle on one row, for code that works a row at a time rather than a column at a time. The index is 1-based within THIS table -- in the slice regime, row 1 is the slice's first row, not the file's.

generic, public :: get_slice => get_slice_i32, get_slice_i64, get_slice_f32, get_slice_f64, get_slice_bool, get_slice_date, get_slice_time, get_slice_ts, get_slice_i32v, get_slice_i64v, get_slice_f32v, get_slice_f64v, get_slice_boolv, get_slice_datev, get_slice_timev, get_slice_tsv, get_slice_str, get_slice_chr, get_slice_chrv

Copies the rows a parquet_slice selects into a freshly allocated array of the caller's own kind, widening on the way exactly as %get does.

Read more…
generic, public :: set_slice => set_slice_i32, set_slice_i64, set_slice_f32, set_slice_f64, set_slice_bool, set_slice_date, set_slice_time, set_slice_ts, set_slice_i32v, set_slice_i64v, set_slice_f32v, set_slice_f64v, set_slice_boolv, set_slice_datev, set_slice_timev, set_slice_tsv, set_slice_chr, set_slice_chrv

Writes values into the rows a parquet_slice selects -- %get_slice's counterpart. The kind must match the column's exactly, and the array must have one value per selected row.

Read more…
generic, public :: col => col_ptr_i32, col_ptr_i64, col_ptr_f32, col_ptr_f64, col_ptr_bool, col_ptr_date, col_ptr_time, col_ptr_ts, col_ptr_i32v, col_ptr_i64v, col_ptr_f32v, col_ptr_f64v, col_ptr_boolv, col_ptr_datev, col_ptr_timev, col_ptr_tsv, col_ptr_strcol

Points p at a column's storage: zero copy, writable, and the pointer kind must match the stored kind exactly (ask %kind first if you do not know it). A parquet_string_column pointer aliases a PK_STRING column's packed store: read it and edit its values in place, but do NOT change its length or element count through the pointer -- the column's own row count would no longer describe it.

Read more…
generic, public :: get => get_arr_i32, get_arr_i64, get_arr_f32, get_arr_f64, get_arr_bool, get_arr_date, get_arr_time, get_arr_ts, get_arr_i32v, get_arr_i64v, get_arr_f32v, get_arr_f64v, get_arr_boolv, get_arr_datev, get_arr_timev, get_arr_tsv, get_arr_str, get_arr_chr, get_arr_chrv

Copies a column into a freshly allocated array of the caller's own kind.

generic, public :: set => set_arr_i32, set_arr_i64, set_arr_f32, set_arr_f64, set_arr_bool, set_arr_date, set_arr_time, set_arr_ts, set_arr_i32v, set_arr_i64v, set_arr_f32v, set_arr_f64v, set_arr_boolv, set_arr_datev, set_arr_timev, set_arr_tsv, set_arr_chr, set_arr_chrv, set_arr_strcol
generic, public :: add_column => add_column_i32, add_column_i64, add_column_f32, add_column_f64, add_column_bool, add_column_date, add_column_time, add_column_ts, add_column_i32v, add_column_i64v, add_column_f32v, add_column_f64v, add_column_boolv, add_column_datev, add_column_timev, add_column_tsv, add_column_chr, add_column_chrv, add_column_strcol, add_column_col
generic, public :: get_element => get_element_i32_i32, get_element_i32_i64, get_element_i64_i32, get_element_i64_i64, get_element_f32_i32, get_element_f32_i64, get_element_f64_i32, get_element_f64_i64, get_element_bool_i32, get_element_bool_i64, get_element_date_i32, get_element_date_i64, get_element_time_i32, get_element_time_i64, get_element_ts_i32, get_element_ts_i64, get_element_i32v_i32, get_element_i32v_i64, get_element_i64v_i32, get_element_i64v_i64, get_element_f32v_i32, get_element_f32v_i64, get_element_f64v_i32, get_element_f64v_i64, get_element_boolv_i32, get_element_boolv_i64, get_element_datev_i32, get_element_datev_i64, get_element_timev_i32, get_element_timev_i64, get_element_tsv_i32, get_element_tsv_i64, get_element_chr_i32, get_element_chr_i64, get_element_chrv_i32, get_element_chrv_i64

Reads one row's value out of a column, widening into the caller's variable exactly as %get does -- the one-call form of r = t%row(i) then r%get(name, v). On a *_VEC column the value is that row's whole vector.

generic, public :: set_element => set_element_i32_i32, set_element_i32_i64, set_element_i64_i32, set_element_i64_i64, set_element_f32_i32, set_element_f32_i64, set_element_f64_i32, set_element_f64_i64, set_element_bool_i32, set_element_bool_i64, set_element_date_i32, set_element_date_i64, set_element_time_i32, set_element_time_i64, set_element_ts_i32, set_element_ts_i64, set_element_i32v_i32, set_element_i32v_i64, set_element_i64v_i32, set_element_i64v_i64, set_element_f32v_i32, set_element_f32v_i64, set_element_f64v_i32, set_element_f64v_i64, set_element_boolv_i32, set_element_boolv_i64, set_element_datev_i32, set_element_datev_i64, set_element_timev_i32, set_element_timev_i64, set_element_tsv_i32, set_element_tsv_i64, set_element_chr_i32, set_element_chr_i64, set_element_chrv_i32, set_element_chrv_i64

Writes one row's value in place. The kind must match the column's exactly (as %set does), and writing a value CLEARS that row's null -- use %set_null to put one back. On a *_VEC column the value is that row's whole vector.

generic, public :: set_null => set_null_i32, set_null_i64, set_null_e32, set_null_e64, set_null_mask, set_null_mask_elem

Marks null: row i of a column, element e of row i, or every entry a logical mask marks .false..

Read more…
generic, public :: clear_null => clear_null_i32, clear_null_i64, clear_null_e32, clear_null_e64

Marks row i -- or, given e, element e of it -- valid without saying what its value is. Only useful when a value is already there or is about to be written; %set_element clears the null itself.

procedure, public :: compact_validity => table_compact_validity

Drop a null bitmap that no longer has nulls.

procedure, public :: ensure_validity => table_ensure_validity

Allocate validity storage up front, for concurrent nulling.

procedure, public :: drop_column => table_drop_column

Remove a column; force= for a predefined one.

procedure, public :: rename_column => table_rename_column

Change the name a column is looked up by.

procedure, public :: copy_column => table_copy_column

Add a copy of a column, optionally of another kind.

procedure, public :: cast => table_cast

Convert a column to another kind, in place.

procedure, public :: filter_rows => table_filter_rows

Keep only the rows a mask selects.

generic, public :: sort_by => table_sort_by, table_sort_by_string

Reorders rows by one or more key columns. Detaching.

generic, public :: top_n => table_top_n, table_top_n_string

Keeps only the n best rows, in key order. Detaching.

generic, public :: argsort_by => table_argsort_by_i32, table_argsort_by_i64, table_argsort_by_string_i32, table_argsort_by_string_i64

The row order the keys imply, without reordering anything. Unlike %sort_by the table stays attached, so this is how to read rows in an order while keeping the file.

generic, public :: argsort_partial => table_argsort_partial_i32, table_argsort_partial_i64, table_argsort_partial_string_i32, table_argsort_partial_string_i64

The n best rows in order, by selection rather than a full sort. Also non-mutating.

generic, public :: is_sorted_by => table_is_sorted_by, table_is_sorted_by_string

Whether the rows are already in that order.

generic, public :: delete_rows => table_delete_rows_i32, table_delete_rows_i64

Removes the listed rows. A thin convenience over %filter_rows, and like it, detaching.

generic, public :: truncate => table_truncate_i32, table_truncate_i64

Keeps only the first n rows. Detaching, like every row-structural change.

generic, public :: append => table_append_table, table_append_row

Appends rows: a whole table's worth, or one row. Detaching, like every row-structural change. The bulk idiom is %clone_structure -> fill -> %append(batch).

generic, public :: append_null_rows => table_append_null_rows_i32, table_append_null_rows_i64

Appends n all-null rows, to be filled in afterwards. Detaching.

procedure, public :: compact => table_compact

Release capacity appends left behind.

generic, public :: reserve => table_reserve_i32, table_reserve_i64

Makes room for n rows in every resident column, so the appends that follow do not reallocate. %compact's counterpart; neither changes the row set, so neither detaches.

procedure, public :: reserve_columns => table_reserve_columns

Makes room for n COLUMNS, so that the %add_column calls that follow relocate nothing and leave an outstanding %col pointer valid. See its own doc-comment for the guarantee.

procedure, public :: column_capacity => table_column_capacity

Column slots allocated, or spare.

procedure, public :: clone => table_clone

Independent deep copy of this table.

procedure, public :: clone_structure => table_clone_structure

Empty table with the same columns.

procedure, public :: clone_extra => table_clone_extra

Hook -- copies an EXTENDING type's own components.

procedure, public :: bind_predefined => table_bind_predefined

Binds a generated type's predefined columns.

generic, public :: assignment(=) => assign_guard

Blocks intrinsic assignment: the store lives behind a pointer, so a default b = a would leave two tables sharing one store and double-freeing it.

Read more…

type, public ::  parquet_table_row

One row of a table, as a lightweight handle: r = t%row(i).

Read more…

Type-Bound Procedures

generic, public :: get => row_get_i32, row_get_i64, row_get_f32, row_get_f64, row_get_bool, row_get_str, row_get_date, row_get_time, row_get_ts, row_get_i32v, row_get_i64v, row_get_f32v, row_get_f64v, row_get_boolv, row_get_strv, row_get_datev, row_get_timev, row_get_tsv, row_get_col_i32, row_get_col_i64, row_get_col_f32, row_get_col_f64, row_get_col_bool, row_get_col_str, row_get_col_date, row_get_col_time, row_get_col_ts, row_get_col_i32v, row_get_col_i64v, row_get_col_f32v, row_get_col_f64v, row_get_col_boolv, row_get_col_strv, row_get_col_datev, row_get_col_timev, row_get_col_tsv

Copies this row's value for a column into the caller's own variable, widening int32 -> int64 and float32 -> float64 exactly as the table's own %get does. The column may be named by a string or by a parquet_table_col handle; the handle form does no name lookup, which is what a loop over rows would otherwise repeat.

generic, public :: set => row_set_i32, row_set_i64, row_set_f32, row_set_f64, row_set_bool, row_set_str, row_set_date, row_set_time, row_set_ts, row_set_i32v, row_set_i64v, row_set_f32v, row_set_f64v, row_set_boolv, row_set_strv, row_set_datev, row_set_timev, row_set_tsv, row_set_col_i32, row_set_col_i64, row_set_col_f32, row_set_col_f64, row_set_col_bool, row_set_col_str, row_set_col_date, row_set_col_time, row_set_col_ts, row_set_col_i32v, row_set_col_i64v, row_set_col_f32v, row_set_col_f64v, row_set_col_boolv, row_set_col_strv, row_set_col_datev, row_set_col_timev, row_set_col_tsv

Writes this row's value for a column. The kind must match the column's exactly (a write never widens), and writing a value CLEARS that row's null. The TABLE is updated -- a handle is a view of it, not a copy. The column may be named by a string or by a parquet_table_col handle.

generic, public :: ref => row_ref_i32, row_ref_i64, row_ref_f32, row_ref_f64, row_ref_bool, row_ref_date, row_ref_time, row_ref_ts, row_ref_i32v, row_ref_i64v, row_ref_f32v, row_ref_f64v, row_ref_boolv, row_ref_datev, row_ref_timev, row_ref_tsv

Points p at this row's storage: zero copy, writable, exact kind. A scalar column gives a scalar pointer, a vector column a pointer to that row's whole vector. The two string kinds have no %ref -- a packed variable-length store has no fixed slot to point at -- and the pointer dies with any structural change, exactly as the table's own %col pointers do.

generic, public :: is_null => row_is_null, row_is_null_elem

Whether this row is null in a column, or -- given e -- element e of it.

procedure, public :: index => row_index

This row's 1-based index within the table.

procedure, public :: is_valid => row_is_valid

Whether the handle is attached AND still current.

type, public ::  parquet_table_col

A resolved handle on ONE column of a parquet_table: the slot, its kind and the table's row scope, captured once so a per-element loop stops resolving a name on every access.

Read more…

Type-Bound Procedures

generic, public :: get => col_get_i32_i32, col_get_i32_i64, col_get_i64_i32, col_get_i64_i64, col_get_f32_i32, col_get_f32_i64, col_get_f64_i32, col_get_f64_i64, col_get_bool_i32, col_get_bool_i64, col_get_str_i32, col_get_str_i64, col_get_date_i32, col_get_date_i64, col_get_time_i32, col_get_time_i64, col_get_ts_i32, col_get_ts_i64, col_get_i32v_i32, col_get_i32v_i64, col_get_i64v_i32, col_get_i64v_i64, col_get_f32v_i32, col_get_f32v_i64, col_get_f64v_i32, col_get_f64v_i64, col_get_boolv_i32, col_get_boolv_i64, col_get_strv_i32, col_get_strv_i64, col_get_datev_i32, col_get_datev_i64, col_get_timev_i32, col_get_timev_i64, col_get_tsv_i32, col_get_tsv_i64, col_get_i32v_e32, col_get_i32v_e64, col_get_i64v_e32, col_get_i64v_e64, col_get_f32v_e32, col_get_f32v_e64, col_get_f64v_e32, col_get_f64v_e64, col_get_boolv_e32, col_get_boolv_e64, col_get_strv_e32, col_get_strv_e64, col_get_datev_e32, col_get_datev_e64, col_get_timev_e32, col_get_timev_e64, col_get_tsv_e32, col_get_tsv_e64

Copies one row's value into the caller's variable, widening exactly as the table's own %get_element does. No name, no lookup -- the handle already knows the slot. Given e as well, copies ONE ELEMENT of that row without materialising the rest, which the name form cannot do at all.

generic, public :: set => col_set_i32_i32, col_set_i32_i64, col_set_i64_i32, col_set_i64_i64, col_set_f32_i32, col_set_f32_i64, col_set_f64_i32, col_set_f64_i64, col_set_bool_i32, col_set_bool_i64, col_set_str_i32, col_set_str_i64, col_set_date_i32, col_set_date_i64, col_set_time_i32, col_set_time_i64, col_set_ts_i32, col_set_ts_i64, col_set_i32v_i32, col_set_i32v_i64, col_set_i64v_i32, col_set_i64v_i64, col_set_f32v_i32, col_set_f32v_i64, col_set_f64v_i32, col_set_f64v_i64, col_set_boolv_i32, col_set_boolv_i64, col_set_strv_i32, col_set_strv_i64, col_set_datev_i32, col_set_datev_i64, col_set_timev_i32, col_set_timev_i64, col_set_tsv_i32, col_set_tsv_i64, col_set_i32v_e32, col_set_i32v_e64, col_set_i64v_e32, col_set_i64v_e64, col_set_f32v_e32, col_set_f32v_e64, col_set_f64v_e32, col_set_f64v_e64, col_set_boolv_e32, col_set_boolv_e64, col_set_strv_e32, col_set_strv_e64, col_set_datev_e32, col_set_datev_e64, col_set_timev_e32, col_set_timev_e64, col_set_tsv_e32, col_set_tsv_e64

Writes one row's value. The kind must match the column's exactly (a write never widens), and writing a value CLEARS that row's null. The TABLE is updated -- a handle is a view of it, not a copy. Given e as well, writes ONE ELEMENT and clears that element's null rather than the whole row's.

generic, public :: is_null => col_is_null_i32, col_is_null_i64, col_is_null_e32, col_is_null_e64

Whether row i of this column is null, or -- given e as well -- element e of it. On a *_VEC column the row form answers "ANY element of the row is null".

generic, public :: set_null => col_set_null_i32, col_set_null_i64, col_set_null_e32, col_set_null_e64

Marks row i null, or -- given e -- element e of it. Naming only a row marks every element of it, exactly as the table's own %set_null does.

generic, public :: clear_null => col_clear_null_i32, col_clear_null_i64, col_clear_null_e32, col_clear_null_e64

Clears row i's null, or -- given e -- element e of it. The stored VALUE is whatever was there; clearing a null does not write one.

generic, public :: ref => col_ref_i32, col_ref_i64, col_ref_f32, col_ref_f64, col_ref_bool, col_ref_date, col_ref_time, col_ref_ts, col_ref_i32v, col_ref_i64v, col_ref_f32v, col_ref_f64v, col_ref_boolv, col_ref_datev, col_ref_timev, col_ref_tsv, col_ref_strcol

Points p at this column's live storage -- the %col pointer, without the name lookup. Same rules: the kind must match exactly (a pointer never widens), a write through p changes the table, and REORDERING one column through its pointer breaks the table's row alignment with nothing to report it.

procedure, public :: is_valid => col_is_valid

Whether the handle is attached AND still current.

procedure, public :: index => col_index

This column's 1-based position in the table.

procedure, public :: kind => col_kind

This column's PK_* kind.

procedure, public :: name => col_name

This column's name.

procedure, public :: width => col_width

This column's values per row (1 for a scalar kind).

procedure, public :: unit => col_unit

This column's unit string, or "".

procedure, public :: residency => col_residency

Whether this column is RES_EMPTY/RES_PARTIAL/RES_FULL.

procedure, public :: set_user_populated => col_set_user_populated

Claim this column's values as the caller's own, or unclaim.

procedure, public :: is_user_populated => col_is_user_populated

Whether this column is claimed as holding the caller's values.