Core reader/writer implementation for Apache Parquet files: schema construction/validation from MAML, writers/readers, row filtering, quality-control (qc:) enforcement, and flat key-value table metadata. Submodules parquet_write.f90/parquet_read.f90/parquet_metadata.f90 hold the bodies of the module procedures declared in the interface block below.
This is an internal implementation module -- do not use parquet_core
directly. Everything public here is re-exported by the parquet facade
module (parquet.f90), which is the one and only supported entry point and
the surface the library's semantic-versioning promise covers. parquet_core
has to stay accessible because the sibling modules (parquet_tables,
...) use it, but its name and contents may change in any release.
Public only so the sibling parquet_tables module can reach them -- src/parquet.f90
makes both private again, so neither is part of the use parquet surface.
parquet_split_name_list is the library's one name-list tokenizer, and
parquet_parse_sort_key its one sort-key direction grammar; the table layer's
string key lists spell direction exactly as a read-time parquet_sortkey does
because they run the same parser rather than a second copy of it.
| Type | Visibility | Attributes | Name | Initial | |||
|---|---|---|---|---|---|---|---|
| integer, | public, | parameter | :: | parquet_size_auto | = | -1 |
col_size/array_size sentinel for a MAML col_size: auto/array_size: auto declaration: resolved later, either by schema%set_col_size/%set_array_size before parquet_open_writer, or automatically from the actual data's own shape at the first parquet_write_column/parquet_write_column_chunk call able to supply it. col_size needs a matrix-form write (never the flat/1-D form, which needs col_size already known to interpret its own row count); array_size is resolved by either shape, from the caller's declared character length -- so a scalar string column's array_size: auto resolves on an ordinary 1-D write. See parquet_resolve_or_check_col_size/_array_size in parquet_write.f90. |
| character(len=*), | public, | parameter | :: | parquet_sample_algorithm | = | "sample:bernoulli-u<p/philox/v1" |
The frozen contract identifier for Reading `"sample:bernoulli-u |
| integer(kind=int64), | public, | parameter | :: | parquet_sample_label | = | 4994076164431785653_int64 |
Label separating the row sample's stream from every other family drawn from the same seed. Not decoration -- feature_risks.md Risk-123. Without it, The value is arbitrary beyond having to differ from every other label and from 0, and is public so a caller can reproduce the selection outside the library. |
Overrides the default structure constructor so a schema can be built in one expression (my_maml = parquet_schema(table="my_table")) as an alternative to call my_maml%init(table="my_table"); both call parquet_schema_new/schema_init under the hood. table is the only required argument (the MAML table: key); survey, dataset, version, date, author, description, license, and maml_version are all optional and set the correspondingly-named MAML header key when given. Returns the newly initialized schema.
Writes one column's values to an open parquet_writer (writer), under
the given column name (name), dispatched by the actual/declared
type/kind of values (scalar values(:), one value per row; or
matrix/vector values(:,:), (element, row) values). is_valid (optional)
marks per-element nulls (matching values' rank); a .false. entry for a
protected column (see parquet_column_type%is_protected) error stops.
A scalar "string" column can also be passed as a type(parquet_string_column) (see
module parquet_strings, doc/pages/types/string-columns.md) instead of a padded
character(len=...) array -- this specific carries its own per-element null status (see
%append_null/%is_null), so is_valid is not accepted for it; every null already in the
column is written as a Parquet Null directly. Scalar (1-D) columns only -- there is no
vector/matrix parquet_string_column specific, unlike every other type family above.
Starts a new row group of nrows rows on writer -- see parquet_write_column_chunk
below. Every column already known to writer (whether schema-declared or already
chunk-written for an earlier row group) must then receive exactly one
parquet_write_column_chunk call for this row group (or already be a whole column from an
earlier parquet_write_column call) before parquet_finish_row_group. Error stops if a row
group is already open (i.e. called again without an intervening parquet_finish_row_group)
rather than silently abandoning the still-open one.
Writes one column's values for the currently-open row group (see parquet_new_row_group)
to an open parquet_writer (writer), under the given column name (name) -- the streaming
counterpart to parquet_write_column, for a large column you build and write one row group
at a time instead of as one complete array. Dispatched by the actual/declared type/kind of
values, same as parquet_write_column (scalar values(:) or matrix/vector values(:,:)),
and every column's chunk for the currently-open row group must have exactly
parquet_new_row_group's own nrows rows. A schema-enforced column's values are converted to
its declared data_type exactly as parquet_write_column converts them (int32/int64/float32/
float64 in any combination; a float-to-integer conversion error stops on a non-integral or
out-of-range value, per chunk), so the same values are accepted whichever path writes them --
see parquet_append_as_schema_chunk_int32. A kind the declared type is not compatible with at
all (a logical chunk into an int32 column, say) is still an error stop. A column written
once via parquet_write_column can never also be written via parquet_write_column_chunk
(or vice versa), and every column must appear in the first row group written for this
writer, since a Parquet file's schema is fixed from that point on.
A scalar "string" column can also be passed as a type(parquet_string_column) holding
exactly this row group's rows -- see parquet_write_column's own doc comment above for the
shared parquet_string_column notes (no is_valid, scalar only).
Sets a whole-file row filter ("mask") on writer: mask(i) = .false. drops row i
entirely from every column written from this point on (no trace at all -- no offset, no
validity bit; this is not the same as writing a Null). Must be called after
parquet_open_writer and before the writer's first parquet_write_column or
parquet_new_row_group call, whichever comes first.
For a writer using only parquet_write_column (no row groups): every subsequent
parquet_write_column call's values (or the row dimension of values(:,:)) must have
length size(mask) exactly, and the resulting column has count(mask) rows.
For a writer using parquet_new_row_group/parquet_write_column_chunk: each
parquet_new_row_group(writer, nrows) call automatically claims the next nrows positions
of mask (in file order) as that row group's window, and every column's
parquet_write_column_chunk call for that row group must supply values of length exactly
nrows; the row group's actual written row count is count() of its window's mask slice
(which may be anywhere from 0 to nrows). mask must be fully consumed by the writer's
row groups by the time it closes (error stop at parquet_close_writer otherwise). See
doc/pages/io/writing.md's "Filtering rows with a mask" for a worked example.
Mutually exclusive with parquet_write_chunk_row_mask on the same writer -- once either has
been used, calling the other is an error stop. Callable at most once per writer -- a
second call is an error stop rather than silently replacing the first mask.
Sets the row filter ("mask") for the currently-open row group only, for a writer using
exclusively parquet_write_column_chunk (no parquet_write_column calls anywhere in its
lifetime). Must be called once per row group, after that row group's parquet_new_row_group
and before its first parquet_write_column_chunk call; size(mask) must equal that row
group's own nrows (from parquet_new_row_group) exactly, and the row group's actual
written row count is count(mask). A second call for the same still-open row group is an
error stop rather than silently replacing the first mask.
If used for a writer's first row group, it must be used for every row group of that
writer (all-or-nothing per writer, not per row group) -- error stop otherwise. It is
unavailable (error stop) once any column has been written whole via parquet_write_column
anywhere in the writer's lifetime; that combination can only use parquet_write_row_mask
instead. Mutually exclusive with parquet_write_row_mask on the same writer.
Returns writer_or_reader's row-group size ("chunk_size", matching parquet_open_writer's
own chunk_size argument name -- Arrow's own WriteTable convenience function uses this same
term for the concept). For a parquet_writer: the caller's own explicit chunk_size, validated
against every declared vector column, or -- if chunk_size was never set -- an estimate
computed from the schema's declared types/col_size. Usable at any point after
parquet_open_writer, including before any column has been written, and it does not change
afterwards: parquet_new_row_group's own nrows is what sizes each streamed row group and
never feeds back into this value, and a writer that only uses parquet_write_column has its
row groups sized at parquet_close_writer from the finished table's real byte size, which a
schema-only estimate here cannot anticipate. So on the write side this is a suggestion for a
chunked-write loop, not a promise about the file's final layout. For a parquet_reader: the
row group's size at row_group (1-based; omitted defaults to the first row group),
reflecting the file's actual, already-written-and-fixed layout -- row groups are not
guaranteed uniform, so it is only ever a suggestion for a chunked-read loop there too.
Returns reader's row-group count in num_row_groups, dispatched by its
integer(int32)/integer(int64) kind (the int32 specific also error stops if the actual
count overflows int32 -- vanishingly unlikely in practice, but kept for consistency with
parquet_get_nrows's own int32/int64 overload). Reflects the file's physical layout;
unaffected by any filter= given to parquet_open_reader. See "Streaming/chunked reads" in
doc/pages/io/reading.md for the chunked-read loop this and parquet_get_chunk_size/
parquet_read_column_chunk are meant to be used together for.
Measures the uniform element-count-per-row (width) of a plain Parquet LIST/LARGE_LIST
column, over the 1-based inclusive row-group range row_group_lo..row_group_hi, without
ever materializing the whole column. Both bounds are accepted as integer(int32) or
integer(int64); row_group_lo <= 0 means "every row group in the file".
Such a column may hold a different number of elements in every row, so unlike a
FIXED_SIZE_LIST (this library's own vector layout) its width is a property of the data
rather than the schema -- see parquet_column_width_needs_data. width comes back as 1 when
no single width above 1 covers every row, which is also the answer for a genuinely scalar
column or a FIXED_SIZE_LIST of width 1, and as 0 for a column with no rows at all --
matching what parquet_get_col_size reports for an empty list column.
proven chooses how much work to do, and the difference matters:
.false. -- decide from the file footer alone, reading no column data at all. Per row
group, the mean elements per row is compared against its neighbours; a non-integral or
disagreeing mean proves no uniform width exists. A surviving answer is a CANDIDATE only:
rows of length 3, 1, 3, 1 average to exactly 2. Use this when a wrong answer is safe
because something downstream will reject it..true. -- screen as above, then confirm by reading the covered row groups one at a time,
stopping at the first row that disagrees. Peak memory stays at one row group, so this is
safe on a column far larger than memory, but it does read data.Whether name contains any Null over the 1-based inclusive row-group range
row_group_lo..row_group_hi, answered from the file's own statistics. Both bounds are
accepted as integer(int32) or integer(int64); row_group_lo <= 0 means "every row group".
Reads no column data: Parquet records a null count per column chunk in the footer. That is what makes this worth calling before a read -- a caller that knows a column is Null-free can skip building a validity mask for it entirely.
Answers .true. when the column has at least one Null or when the file cannot say.
Statistics are optional in the Parquet format, so a chunk without them (or without a null
count) reads as "might have Nulls"; a dotted struct-field path is declined outright, because a
struct leaf's validity is combined with every ancestor struct's on read and the leaf's own null
count therefore does not describe the result. Treat .false. as a guarantee and .true. as
"assume the worst" -- that is the direction which keeps a caller that skips a mask on the
strength of this answer from ever meeting an unexpected Null.
Parses a MAML into a parquet_schema (its %maml, %cinfo and %metadata).
The file form takes a filename (.maml file path) and loads it from
disk first; the object form takes only schema, whose %maml has
already been populated (e.g. built in memory). Both forms fill in
schema's %cinfo and %metadata in place.
Runs the full set of MAML validity checks (a table: key, at least one
field, a valid data_type on every field, ...) against either an
already-loaded MAML passed as maml (a parquet_maml_file, e.g. via
parquet_load_maml_file or built in memory) or a MAML filename passed
as maml (a character(len=*) .maml file path, loaded from disk first,
then checked identically).
Reads one column, named name, from an open parquet_reader (reader)
into values, dispatched by its actual/declared type/kind and rank
(scalar 1-D values(:), one value per row -- the column_1d specifics;
or a full 2-D values(:,:), (element, row) values -- the array_full
specifics, for a vector column). null_value (optional) fills missing
entries; is_valid (optional) reports which elements were actually
present (same rank as values).
A scalar "string" column can also be read straight into a type(parquet_string_column)
(see module parquet_strings) instead of a padded character(len=...) array -- values is
cleared then filled with the whole column; every Null lands as %append_null(), so
neither null_value nor is_valid is accepted for this specific -- check %is_null(i)
afterward instead. Scalar (1-D) columns only.
Reads one row of a vector (array) column named name from an open
parquet_reader (reader): values receives that row's full element
vector, selected by the 1-based row_index. Dispatched by values'
actual/declared type/kind, and separately by row_index's own kind
(integer(int32) or integer(int64) -- the latter needed only to address
a row beyond huge(1_int32), 2,147,483,647, in a file that large).
null_value (optional) fills missing entries; is_valid (optional)
reports which elements were actually present. See
parquet_read_array_element_mode for the complementary "one element
across all rows" access pattern. Reads only the one row group row_index falls in (not the
whole column), unless a row filter is active (parquet_open_reader(..., filter=)/
parquet_reader_set_filter), in which case row_index addresses the filtered result and the
whole (filtered) column is read -- mapping a filtered row index back to its physical row
group needs a per-row-group survivor count that this path does not yet use.
Reads one element position of a vector (array) column named name
from an open parquet_reader (reader) across all rows: values
receives that element (selected by the 1-based elem_index) from
every row. Dispatched by values' actual/declared type/kind.
null_value (optional) fills missing entries; is_valid (optional)
reports which rows were actually present. See
parquet_read_array_row_mode for the complementary "one row" access pattern. Unlike
parquet_read_array_row_mode (which only needs one row group), this access pattern
inherently touches every row, so every row group contributes -- none can be skipped.
Instead, unless a row filter is active, this streams the file row group by row group
(never materializing the whole column's flattened element count in a single internal
call), so a vector column whose total element count (rows times per-row width) would
otherwise exceed 2,147,483,647 can still be read this way. If a row filter is active
(parquet_open_reader(..., filter=)/parquet_reader_set_filter), the whole (filtered) column
is read instead, for the same reason parquet_read_array_row_mode falls back: elem_index
then addresses the filtered result.
Reads one row group's worth of one column, named name, from an open parquet_reader
(reader) into values -- the row-group-chunked counterpart to parquet_read_column, for a
large column you read one row group at a time instead of materializing the whole column.
row_group (1-based) selects which row group; reads are stateless/random-access (unlike
the write side's parquet_new_row_group/parquet_finish_row_group pairing, there is no
"currently open" row group to track -- call with any row_group, in any order, as many
times as you like). Use parquet_get_num_row_groups to learn how many row groups a file
has, and parquet_get_chunk_size(reader, ..., row_group=) to learn a specific row group's
own row count before allocating values. Dispatched by values' actual/declared
type/kind and rank (scalar values(:) or vector values(:,:)) exactly like
parquet_read_column, and separately by row_group's own kind (integer(int32) or
integer(int64) -- the latter needed only to address a row group beyond
huge(1_int32) in a file that large). null_value (optional) fills missing entries;
is_valid (optional) reports which elements were actually present.
Two restrictions not shared with parquet_read_column: disallowed (error stop) on a reader with an active sort (a permutation destroys row-group locality, so there is no coherent "row group N of the sorted output" to serve -- see check_reader_no_sort; a filter= or sample_fraction= mask only ever REMOVES rows, so a chunked read works under one and hands back that row group's surviving rows); and, when the reader was opened with qc=.true., every chunk read runs the usual qc: min/max/miss checks scoped to just that one row group's own data (not the whole column) -- a hard-mode (qc_soft=.false.) violation error stops naming the offending row group; a soft-mode (qc_soft=.true.) violation warns at most once per column, same throttling as every other read path. See "Streaming/chunked reads" in doc/pages/io/reading.md.
A scalar "string" column can also be read into a type(parquet_string_column) holding
just row group row_group's rows -- values is cleared then filled, same no-null_value/
no-is_valid convention as parquet_read_column's own parquet_string_column specific above.
Returns reader's post-filter row count in nrows, dispatched by
its integer(int32)/integer(int64) kind (the int32 specific also error
stops if the actual row count overflows int32). check_positive
(optional, default .false.): if .true., error stops instead of
returning 0 rows.
Opens filename for reading into reader. use_threads (optional):
use Arrow's multi-threaded reader. filter (optional): a parquet_filter
row filter to apply. sample_fraction (optional, real(real64)): keeps
each row independently with probability sample_fraction (Bernoulli
sampling, not an exact row count) -- omitted, or >= 1.0, reads every
row (the current/default behavior); must not be negative or NaN
(error stops); exactly 0.0 deterministically yields zero rows. Shares
its underlying mask with filter= (see "Row filtering with
parquet_filter" in doc/pages/io/filter-sort-sample.md): a filter, if
also given, is applied
on top of the downsample, and sample_fraction < 1.0 alone (even with
no filter=) carries the same consequences filter= already has --
chunked reads (parquet_read_column_chunk) are disallowed, and array
row/element-mode reads fall back to a whole-column read. sample_seed
(optional, integer(int32)): omitted or <= 0 draws a fresh seed from
entropy (a different sample each call); > 0 makes the draw
reproducible. The seed actually used (caller-supplied or
entropy-drawn) is always reported by parquet_close_reader(...,
print_stat=.true.), so a non-deterministic run's seed can be read
back afterward and reused. schema (optional): a parquet_schema/qc-maml
to validate columns against. qc (optional): enable qc: min/max/miss
enforcement (needs schema). qc_soft (optional): qc violations warn
instead of error-stopping. prefetch (optional, default .false.): when
.true., every column in the file is read and cached right away, after
any filter/sample has been applied, instead of each column being read
lazily on first request -- equivalent to calling
parquet_prefetch_columns for every column immediately after opening;
materializes the whole file in memory up front, see
doc/pages/operating/performance.md.
nrows= is generic over integer(int32)/integer(int64)
(parquet_open_reader_nrows_int32/_int64: fills nrows with the
post-filter/post-sample row count, error-stopping if it overflows the
requested kind), plus the original nrows-less form (parquet_open_reader_base)
for when nrows isn't wanted at all. This mirrors parquet_get_nrows's
own int32/int64 overload above; nrows is required (not optional) in
the two typed specifics -- an optional dummy that may be absent can't
be the sole thing distinguishing two specific procedures in a generic
interface (a call omitting it would be ambiguous), so
parquet_open_reader_base carries the nrows-absent case as a separate
specific instead. sample_fraction/sample_seed are each single-kind
(real64/int32) rather than dual-kind like nrows, specifically to avoid
that same ambiguity: a second independently-optional dual-kind
argument on this generic would force a 3x3 cross product of specifics
(absent/real32/real64 sample_fraction times the three nrows shapes) --
not worth it for a fraction/seed argument, which isn't the
row-count/size/index category that convention exists for.
Reads back one key's value from the flat key-value table metadata a
parquet_writer wrote via add_metadata (see parquet_reader%metadata,
populated once by parquet_open_reader). value's declared type/kind
selects the specific procedure, so it also selects which stored
representation is expected -- the stored string (always written by
add_metadata as plain text, see parquet_metadata.f90) is parsed back
into that type. Any key is allowed, including the writer's own
reserved/internal keys (e.g. "DATE", "column.
If key is missing: returns default if given (printing a WARNING
unless warn=.false.), else error stops. If key is present but its
stored text cannot be converted to the requested type (including a
stored integer that overflows a 32-bit target): always prints a
WARNING, then falls back to default if given, else error stops.
default must be the same type/kind as value; warn defaults to
.true. and only affects the "key missing, default used" case.
Returns the total element count of a vector (array) column named
name, read from an open parquet_reader (reader), across every row
(i.e. nrows * col_size) in total_elements, dispatched by its
integer(int32)/integer(int64) kind. Reads no column data for a scalar or FIXED_SIZE_LIST
column (nrows and col_size are both already known from the file footer/schema), so this is
safe to call even on a column whose total element count itself exceeds int32 -- unlike an
earlier implementation, which materialized the whole column just to answer this query and
could hit Arrow's own int32 list-index ceiling on a large enough column (see CLAUDE.md's
"Guarding a hard Arrow int32-only ceiling"). A plain LIST/LARGE_LIST column has no
schema-level width, so it is measured one row group at a time by the same helper
parquet_get_col_size uses -- data is read, but never more than one row group at once.
Reads and caches the named column(s) of an open parquet_reader
(reader) right away rather than lazily on first request.
parquet_prefetch_columns accepts names as either an array of column
names (each element sharing one declared length -- pad shorter names
with blanks) or a single scalar string listing the names separated by
commas and/or semicolons ("ra;dec,mag"). The scalar form avoids the
fixed-length array pitfall where a too-short declared length silently
truncates a name.
Applies a filter to an already-open reader (see the specifics' own doc-comments for the full contract). Two forms: whole-file, and scoped to an inclusive 1-based row-group range.
The scoped form is the memory-bounded one, and the difference is in how the mask is BUILT, not merely in which rows survive. Whole-file, every filter column is read in one batched, thread-parallel pass and left decoded in the reader's cache -- fastest, and the right default. Scoped, the expression is evaluated one row group at a time and each chunk is released before the next is read, so peak memory is one row group's worth of the filter columns rather than the whole file; rows outside the range never match, and nothing is left cached, so a filter column read afterwards is read again. Use it when the file is larger than memory, or when only part of it is of interest -- typically alongside chunked reads over the same row groups.
It is the PRESENCE of the row-group arguments that selects between those two engines, not
their value: row_group_lo = 0 means "all row groups" on the memory-bounded engine (and
row_group_hi is then ignored), which is how a whole file larger than memory is filtered
without first asking parquet_get_num_row_groups how many there are. A non-positive lower
bound reads as "all row groups" here exactly as it already does in
parquet_measure_list_width and parquet_column_has_nulls.
A third form takes a physical ROW range as well (four numeric arguments rather than two): only rows row_lo..row_hi, 1-based and inclusive, may match. It exists because a row-group range can only ever start and end on a row-group boundary, so a caller working over an arbitrary row range -- a parquet_table slice, typically -- would otherwise get back the whole covering row groups' survivors and have no way to trim them, the mask being the only thing that knows which rows those are. With this form the filter answers for exactly the requested rows, and parquet_get_nrows afterwards is that range's own surviving count. The filter may hold no rules at all in this form, which installs the range by itself. The row range must lie inside the rows its row groups span, or the call fails with error stop rather than quietly handing back the intersection of the two -- which for a disjoint pair is empty, and an empty result is indistinguishable from a filter that matched nothing.
Loads filename's raw lines from disk into a parquet_maml_file,
running the full parquet_validate_maml checks (table:, at least
one field, valid data_type, ...); not parsed into a schema yet
(see parquet_parse_maml).
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| character(len=*), | intent(in) | :: | filename |
.maml file path. |
validated raw MAML.
Loads a qc-maml's raw lines from disk WITHOUT running the full parquet_validate_maml checks parquet_load_maml_file always applies (table:, at least one field, valid data_type, ...) -- a qc-maml only needs name + qc: min:/max:/miss: per field, and doesn't need data_type at all. Its own (lighter) validation happens later, in parquet_parse_qc_maml, when parquet_open_reader(..., schema=) uses it. Returns a parquet_schema with only %maml populated (the raw qc-maml lines); %cinfo/%metadata stay empty, since a qc-maml is never parsed.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| character(len=*), | intent(in) | :: | filename |
qc-maml file path. |
schema with only %maml populated (raw qc-maml lines).
Validates and parses a qc-maml's fields: entries into rules, one
entry per field that has at least a name (data_type/unit/ucd/etc.
are irrelevant here and never required) -- error stops on a
duplicate field name, an unknown top-level section/sub-key, or an
unrecognized qc: miss: value (anything other than Null/NA,
case-insensitive, or empty). Table-level metadata is ignored
entirely. See parquet_qc_rule's own doc comment for min_text/max_text.
The columns a qc-MAML declares an actual qc: block for, as a blank-padded array.
Answers "what does this qc actually constrain?" without exposing the rule objects
themselves: a field that merely NAMES a column, with no qc: key, declares nothing and
is left out -- the same distinction parquet_apply_qc makes before it reaches the reader.
names comes back zero-size when nothing is constrained. Sized to the longest name and
blank-padded, so trim(names(i)) is the name to pass on.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_schema), | intent(in) | :: | schema |
a qc schema (composed or loaded). |
||
| character(len=:), | intent(out), | allocatable | :: | names(:) |
columns with a qc: block. |
Validates user_maml against base_maml (the full base schema):
populates user_maml%missing_columns/%col_map, error stops on any
field user_maml declares that base_maml doesn't.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_maml_file), | intent(in) | :: | base_maml |
the full base schema to validate against. |
||
| type(parquet_maml_file), | intent(inout) | :: | user_maml |
user-supplied MAML being validated. |
Creates filename and opens writer for writing. Without schema,
the writer is schema-less: columns are inferred from the first
parquet_write_column call for each name, with no field metadata/QC.
With schema, every column/type/QC rule is fixed up front and
enforced on every write. By default (overwrite=.true.) an existing
file at filename is silently truncated; pass overwrite=.false.
to instead fail immediately with error stop if filename already
exists, rather than clobbering it.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_writer), | intent(out) | :: | writer |
writer to open. |
||
| character(len=*), | intent(in) | :: | filename |
output .parquet path. |
||
| type(parquet_schema), | intent(in), | optional | :: | schema |
schema to enforce; schema-less writer if absent. |
|
| logical, | intent(in), | optional | :: | write_maml |
also save a sidecar .maml next to filename (needs schema). |
|
| logical, | intent(in), | optional | :: | qc |
enable qc: min/max/miss WARNING checks on write; defaults to present(schema) (on whenever a schema is given), pass .false. to opt out; no-op without a schema. |
|
| character(len=*), | intent(in), | optional | :: | compression |
Arrow compression codec name (e.g. "snappy", "zstd"). |
|
| integer, | intent(in), | optional | :: | compression_level |
codec-specific compression level. |
|
| integer, | intent(in), | optional | :: | chunk_size |
Parquet row-group size. |
|
| logical, | intent(in), | optional | :: | use_threads |
use Arrow's multi-threaded writer. |
|
| logical, | intent(in), | optional | :: | overwrite |
allow truncating an existing file at filename; default .true. |
Ends the currently-open row group -- see parquet_new_row_group above. Error stops if
any column known to writer has no data for this row group (either a chunk just
written, or an already-whole column with enough rows left to slice). On the very
first call for writer, also locks the file's schema (from every column established
by then) and opens it for writing -- no column can be introduced after this point.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_writer), | intent(inout) | :: | writer |
open writer, with a row group open. |
Flushes and closes writer; error stops if SOME (but not all) of a schema-enforced
writer's declared/enabled columns were written -- naming the column that was missed.
A writer that had NOTHING written to it is not an error: every enabled column is written with zero rows and a WARNING says so, so an analysis that legitimately produced no rows still yields a valid file carrying the whole schema (see parquet_write_empty_columns_if_none_written). Writing a zero-length array to every column by hand produces the same file and no warning. Two cases are excluded and keep the abort: a row mask was set (rows were expected), and a row group was opened.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_writer), | intent(inout) | :: | writer |
writer to close. |
Whether name's vector width can only be determined by reading its data.
.true. for exactly one case: a plain Parquet LIST/LARGE_LIST column, whose rows may
each hold a different number of elements, so no width exists in the schema to read.
.false. for a scalar column (width 1 by construction) and for a FIXED_SIZE_LIST -- the
layout this library always writes, and the one any Arrow-based writer preserves -- whose
width is a schema constant. Answered from the schema; reads nothing.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=*), | intent(in) | :: | name |
column name. |
.true. only for a plain LIST/LARGE_LIST column.
Returns .true. if column name (a top-level or dotted struct-leaf path, same
convention as every other column-name argument) exists in reader's schema,
optionally restricted to a set of allowed data types via types.
types asks "can I read this column as one of these?", not "is its physical type
literally one of these?" Each token is compared against the column's target -- the
Fortran kind this library reads it into, which is the same narrowest-lossless mapping
parquet_get_column_type reports (see its doc-comment below for the full table). So
types="int32" matches an int8 or uint16 column, and types="int64" matches a uint32
one, because those are the kinds those columns are read into.
types is a comma-separated list of tokens: any of valid_query_data_types's nine single
types ("int32"/"int64"/"float32"/"float64"/"string"/"boolean"/"date"/"time"/"timestamp"),
and/or the group aliases "int" (any integer column), "float" (any column readable into a
float -- which, since every numeric physical type converts to float64, means every
numeric column, integers and decimals included), and "temporal" (date, time, or
timestamp). Comparison is case-insensitive. Omit types to check existence regardless
of type. error stops if types contains an unrecognized token (checked before the
existence check, so a malformed filter is reported even for a column that doesn't
exist).
An alias is therefore NOT the union of its member tokens, and that is deliberate.
types="float" matches an int32 column (an integer is readable as a float) while
types="float64" does not (that column's target kind is int32). The two ask different
questions -- "can I read this as a float at all?" against "is float64 the right
declaration?" -- and both are useful.
A column this library cannot read at all (a MAP, say) has the target "unknown" and
matches no token, but is still found by a plain (no types) existence check.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=*), | intent(in) | :: | name |
column name (dotted struct-leaf path allowed). |
||
| character(len=*), | intent(in), | optional | :: | types |
comma-separated type tokens/group aliases. |
.true. if the column exists and (if types given) matches one of its tokens.
Applies sort_by to an already-open reader, exactly as
parquet_open_reader(..., sort_by=) would have: every column read
from here on -- and every column already decoded -- comes back in
key order. Provided for the same reasons parquet_reader_set_filter
is: a caller that does not own the parquet_open_reader call, or one
that can only choose its keys after inspecting the file's schema.
Refuses, with error stop, in three states: when the reader already has a sort (add every key to one parquet_sortkey instead), when any column has already been decoded on this reader (data already handed back could not then be aligned with anything read afterwards), and when a chunked read has already been done (its rows were handed back in physical row-group order, which no permutation can reconcile). The last two are separate checks because a chunked read caches nothing, so the decoded-columns predicate cannot see it. A reader opened with filter= or sample_fraction= is fine: the sort orders the surviving rows -- that is the supported order, and the reverse (set_filter under an active sort) is refused by set_filter itself.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(inout) | :: | reader |
open, unsorted reader with no column decoded yet. |
||
| type(parquet_sortkey), | intent(in) | :: | sort_by |
sort keys, parsed and validated here. |
Gives reader the read-time transform source has already worked out, instead of making
it work the same thing out again from the same file.
What this is for. The recommended way to read one file from several threads is to give
each thread its own parquet_reader (see the Thread safety guide). When that file is read
with a filter= or a sort_by=, every one of those readers would otherwise re-decode the
filter's key columns and rebuild the whole sort permutation -- work that is identical in
every reader and can cost more than the parallelism saves. This hands it over instead.
The cost is two atomic refcount increments, not a data copy: a filter mask and a sort permutation are immutable Arrow arrays, so the readers share them. Everything else transferred is proportional to the file's row-group count.
source may be adopted from by several threads at once, provided it is idle -- it is
only read. What it must NOT be is in use by another thread at the same moment, which is
refused rather than raced.
Aborts unless: the two readers are open on files with the same row-group and row counts;
reader has no filter, sample or sort of its own; and no column has been read on reader
yet (one already read was read unmasked, and could not be lined up with an adopted mask).
A source carrying no transform at all is a no-op, so a caller need not ask first.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(inout) | :: | reader |
open reader with no transform of its own yet. |
||
| type(parquet_reader), | intent(in) | :: | source |
open, idle reader whose transform is adopted. |
Closes reader, freeing the underlying C++ handle. check_complete (optional,
default .false.): verify every column read via parquet_read_column_chunk had every
one of the file's row groups read by now (row-mode/whole-column reads are excluded --
only chunk-read columns are tracked at all). check_hard (optional, default .true.
when check_complete is .true.): an incomplete column error stops (naming the column
and its missing row group(s)) when .true., or prints a WARNING and continues when
.false. -- mirrors parquet_open_reader's qc/qc_soft pairing.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(inout) | :: | reader |
reader to close. |
||
| logical, | intent(in), | optional | :: | print_stat |
print Arrow read-statistics to stdout on close. |
|
| logical, | intent(in), | optional | :: | check_complete |
verify every chunk-read column's row groups were all read. |
|
| logical, | intent(in), | optional | :: | check_hard |
error stop (.true., default) vs WARNING (.false.) on incompleteness. |
Returns name's declared col_size (vector-column element count;
1 for a scalar column) in col_size. Reads no column data for a scalar column or a
FIXED_SIZE_LIST one (whose width is a schema-level constant), so this is safe even on a
column whose total element count (nrows * col_size) itself exceeds int32. A plain
LIST/LARGE_LIST column -- which this library never writes, but another producer may --
has no schema-level width at all, so it is screened from the footer and then proven one
row group at a time: that does read data, but never holds more than one row group.
See parquet_get_column_total_elements, which answers via the same helper.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=*), | intent(in) | :: | name |
column name. |
||
| integer, | intent(out) | :: | col_size |
that column's declared element count. |
Returns the longest actual string length among name's values in
max_string_length, so a caller can size a fixed-length
character(len=...) buffer before reading a string column.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=*), | intent(in) | :: | name |
string column name. |
||
| integer, | intent(out) | :: | max_string_length |
longest value length actually present in the column. |
Returns column name's stored time unit and (for a timestamp) timezone. unit
(optional) receives one of the parquet_unit_* selectors; timezone (optional,
allocatable) receives the IANA timezone string ("" for a timezone-naive timestamp or
a time column). Aborts if name is not a time/timestamp column. Distinct from
parquet_get_metadata, which serves user-defined key/value metadata rather than this
schema-level property.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=*), | intent(in) | :: | name |
time/timestamp column name. |
||
| integer, | intent(out), | optional | :: | unit |
stored unit (a parquet_unit_* selector). |
|
| character(len=:), | intent(out), | optional, | allocatable | :: | timezone |
IANA tz, or "" if naive. |
Returns the Fortran type existing column name is READ INTO, in type_name: one of
valid_query_data_types's nine tokens ("int32"/"int64"/"float32"/"float64"/"string"/
"boolean"/"date"/"time"/"timestamp"), or "unknown" for a column this library cannot
read at all. A vector (FIXED_SIZE_LIST) column reports its element type, e.g. an int32
vector column reports "int32" (see parquet_get_col_size for its element count).
The question it answers is "what do I declare?", so the answer is the narrowest lossless Fortran kind for the column's physical type, not the physical type's own name -- all four numeric targets accept the same 15 physical types, so which one a read actually uses is chosen by the caller's declaration rather than by the file:
The two lossy rows return the conventional target rather than "unknown" on purpose: a caller asking what to declare is better served by the kind the library will actually use than by being told a readable column is unreadable.
error stops only if name doesn't exist -- that is a caller mistake, and
parquet_column_exists is the query for it. An unreadable type is an answer ("unknown"),
not an error.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=*), | intent(in) | :: | name |
existing column name (dotted struct-leaf path allowed). |
||
| character(len=:), | intent(out), | allocatable | :: | type_name |
resolved canonical type token. |
Reports whether existing column name is declared NULLABLE in reader's file schema,
in is_nullable. A schema-only query -- it reads no column data, and says nothing about
whether the column actually contains any Null (parquet_column_has_nulls answers that,
from the footer's own null counts).
The two are genuinely different questions. A nullable column may hold no Null at all, which is the normal state of a column written through the streaming API with an all-.true. is_valid mask; a non-nullable one cannot hold a Null even in principle, because the file's own schema forbids it.
For a VECTOR column this reports the ELEMENT (child) field's flag, not the outer list field's -- the outer one is non-nullable by construction here, since a row's vector is never itself missing, so reporting it would answer a constant. A dotted struct path reports the leaf's own flag, the same rule parquet_get_column_type follows.
error stops only if name doesn't exist. A column this library cannot READ still has
a perfectly meaningful nullability flag, so an unreadable type is an answer here too,
exactly as it is for parquet_get_column_type.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=*), | intent(in) | :: | name |
existing column name (dotted struct-leaf path allowed). |
||
| logical, | intent(out) | :: | is_nullable |
.true. if the stored field is declared nullable. |
Returns every column reader's file contains, in schema order, as the same
(possibly dotted) names every other column-name argument in this module accepts:
a nested STRUCT field contributes one entry per leaf beneath it ("addr.city"), never
its own bare name, and every other field contributes one entry under its own name.
names comes back allocated to exactly the column count, each element trimmed to the
longest name present (blank-padded), so trim(names(i)) is the name to pass on.
A zero-column file yields a zero-size names.
LIST/MAP columns (and any leaf beneath a struct that is itself a LIST/MAP) ARE listed,
even though no read entry point supports them: the purpose here is to report what the
file actually holds. Pass a listed name to parquet_column_exists (no types) or
parquet_get_column_type to find out whether it can be read.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=:), | intent(out), | allocatable | :: | names(:) |
one entry per column, schema order. |
The 1-based PHYSICAL file row index of each row this reader currently returns, in the order it returns them.
Without a filter=/sample_fraction=/sort_by= this is simply 1, 2, 3, ..., and
with one it is the only way to find out which file rows survived and in what order --
that lives in the reader's own mask and permutation and is not otherwise visible.
rows comes back with one entry per row the reader reports (parquet_get_nrows).
This is what parquet_table's automatic parquet_row_index column is built on.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| integer(kind=int64), | intent(out), | allocatable | :: | rows(:) |
one physical row index per returned row. |
Every key/value metadata entry the file carries, in the order it is stored.
parquet_get_metadata answers for a key you already know; this reports what is there at all, which is what a caller copying metadata from one file to another needs (it is how parquet_table snapshots a file's metadata at open, so that %get_file_metadata keeps working after the table has detached from its file, and how parquet_write_table's copy_metadata= carries it to an output file).
keys and values are index-aligned and each is allocated to its own longest entry,
blank-padded, so trim(keys(i)) is the key to pass on -- the same convention
parquet_get_column_names uses. Both come back zero-size for a file with no metadata.
Reads nothing: the answer comes from the copy parquet_open_reader already made of the
file's footer metadata.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=:), | intent(out), | allocatable | :: | keys(:) |
one entry per metadata item, stored order. |
|
| character(len=:), | intent(out), | allocatable | :: | values(:) |
that item's value, same order. |
Frees column name's decoded Arrow buffers inside reader, after the caller has
copied the values it wanted into its own Fortran storage. Purely a memory/time trade:
a later read of the same column transparently re-reads and re-decodes it, so this can
never change a result. Composes with an active filter=/sample_fraction= (both are
re-applied to every freshly decoded column).
name may be a dotted struct-leaf path, but the reader caches a struct as ONE array,
so releasing any leaf frees the whole struct -- when walking several leaves of one
struct, release only after the last of them, or each will re-read the struct.
A name that doesn't exist, or a column that was never read, is a silent no-op.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_reader), | intent(in) | :: | reader |
open reader. |
||
| character(len=*), | intent(in) | :: | name |
column to free (dotted struct-leaf path allowed). |
Parses one parquet_sortkey%add key ("ra asc", "-dec", "main.inner.age") into its column name and direction. Purely syntactic: reports a parse failure via ok/errmsg -- never aborts, so the caller can attach the reader's file context to the message -- and leaves every schema-dependent check (column exists, type is orderable) to the C++ side.
Declared here rather than in parquet_read.f90 (and public) because parquet_tables parses the same grammar for its own string key lists -- t%sort_by("ra,-dec"). One parser, so the read-time and in-memory spellings of a direction cannot drift apart.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| character(len=*), | intent(in) | :: | key |
raw key text from one %add call. |
||
| character(len=:), | intent(out), | allocatable | :: | name |
column name (possibly a dotted struct path). |
|
| logical, | intent(out) | :: | descending |
.true. for a descending key. |
||
| logical, | intent(out) | :: | ok |
.true. if the key parsed. |
||
| character(len=:), | intent(out), | allocatable | :: | errmsg |
parse-failure message; "" when ok. |
Merges a MAML-declared and a code-declared read-time QC into the single schema
parquet_open_reader(..., schema=) takes, applying the per-column override rule: for any
column whose MAML fields: entry carries a qc: key AT ALL -- even an empty one, which
already means "no Nulls here" -- the MAML's declaration wins in full and the code's entry
for that column is dropped entirely, rather than merged bound by bound. A column the MAML
says nothing about (or names without a qc: key) takes the code's entry instead.
composed carries ONLY qc-bearing field entries: a MAML entry with no qc: key is not
copied across, because nothing but qc is read from this schema (parquet_open_reader
passes it to parquet_apply_qc and nowhere else). That is also what keeps the two sources
from colliding over a column the MAML merely names.
ncolumns is how many columns composed ends up declaring qc for. Pass composed to
parquet_open_reader only when it is nonzero -- a schema with no rules still switches qc
on C++-side for no benefit.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_schema), | intent(in), | optional | :: | schema |
the MAML's own schema, if there is one. |
|
| type(parquet_read_qc), | intent(in), | optional | :: | qc |
code-declared qc, already in FILE column names. |
|
| type(parquet_schema), | intent(out) | :: | composed |
the merged qc schema. |
||
| integer, | intent(out) | :: | ncolumns |
columns |
One schema field's write-time metadata and QC bounds, parsed from a MAML fields: entry (or built via schema%add_field); one array element per field, held in parquet_column_info%col(:).
| Type | Visibility | Attributes | Name | Initial | |||
|---|---|---|---|---|---|---|---|
| logical, | public | :: | is_set | = | .false. |
true if this column is currently enabled to be written; toggled by set_column_available/set_column_unavailable. |
|
| logical, | public | :: | is_deactivated | = | .false. |
true for columns merged in from a base MAML that the user's MAML excluded; protects is_set from being changed by set_column_available/set_column_unavailable (bulk or by name). |
|
| logical, | public | :: | is_protected | = | .false. |
true if this column's name is listed under extra: protected_cols: in whichever MAML built this cinfo; parquet_write_column error stops if an is_valid mask with any .false. entry is passed for such a column. |
|
| logical, | public | :: | has_qc_min | = | .false. |
true if a qc: min: bound was declared for this field. |
|
| logical, | public | :: | has_qc_max | = | .false. |
true if a qc: max: bound was declared for this field. |
|
| character(len=2), | public | :: | qc_min_op | = | ">=" |
one of ">", ">="; default when qc: min: has no operator prefix. |
|
| character(len=2), | public | :: | qc_max_op | = | "<=" |
one of "<", "<="; default when qc: max: has no operator prefix. |
|
| character(len=:), | public, | allocatable | :: | qc_min_raw |
The qc: min: bound text, operator prefix already stripped/ trimmed; numeric for int32/int64/float32/float64, literal for string. |
||
| character(len=:), | public, | allocatable | :: | qc_max_raw |
qc: max: bound text, same convention as qc_min_raw. |
||
| character(len=:), | public, | allocatable | :: | qc_miss_raw |
The qc: miss: value exactly as declared (unquoted, trimmed), kept ONLY so parquet_validate_field_rules can name the offending text when it is not one of the three legal forms; qc_allow_null below is what every enforcement path actually reads. Left unallocated when the field declares no qc: miss: at all, which is what distinguishes "not declared" from "declared empty" here -- qc_allow_null alone cannot, since both leave it .true./.false. by value only. |
||
| logical, | public | :: | qc_allow_null | = | .true. |
Whether Nulls are an expected part of this field's output, in which case parquet_write_column/parquet_open_reader's qc: enforcement never warns/errors about them. .true. for a declared qc: miss: Null/NA AND -- this is what the default carries -- for a field that declares no qc: miss: at all, since an undeclared miss: means the author said nothing about Nulls and the library does not invent a restriction. Only an EXPLICIT, EMPTY qc: miss: sets this .false., which is how a schema asks for Null validation; finding a Null then triggers a qc: violation. The .true. default is load-bearing and is the only inverted default in this type: the "no miss: declared" case works by nothing ever assigning this, so flipping it back to .false. would silently turn Null validation on for every column of every schema. May be set on any data_type. |
|
| character(len=:), | public, | allocatable | :: | name |
The name of the field [required]; always the internal/ canonical name, i.e. what parquet_write_column/set_column_available/etc. use -- never affected by a col_map: rename (see output_name). |
||
| character(len=:), | public, | allocatable | :: | unit |
The unit of measurement for the field. |
||
| character(len=:), | public, | allocatable | :: | info |
A short description of the field. |
||
| character(len=:), | public, | allocatable | :: | ucd |
Unified Content Descriptor for IVOA (can have many). |
||
| character(len=:), | public, | allocatable | :: | data_type |
The data type of the field [required]. For a temporal column this is the base token only ("date"/"time"/"timestamp"); the unit/utc suffix is parsed out into time_unit/is_utc below. |
||
| integer, | public | :: | time_unit | = | 0 |
For a time/timestamp field, its stored unit as a parquet_unit_* selector (0 = not a temporal column, or unset). A bare time/timestamp token resolves to parquet_unit_micros. |
|
| logical, | public | :: | is_utc | = | .false. |
For a timestamp field, .true. if declared UTC-adjusted (timestamp[...,utc]). |
|
| integer, | public | :: | array_size | = | 1 |
Maximum length of character strings; parquet_size_auto if declared array_size: auto (not yet resolved -- see schema%set_array_size). |
|
| integer, | public | :: | col_size | = | 1 |
The number of elements in the vector column; parquet_size_auto if declared col_size: auto (not yet resolved -- see schema%set_col_size). |
|
| character(len=:), | public, | allocatable | :: | output_name |
The name actually written to the parquet file/VOTable
header. Equal to |
Column-level schema state for one MAML: an array of parquet_column_type, one per declared field, plus lookups/toggles over it. Embedded in parquet_schema%cinfo; parquet_schema's own type-bound procedures are flat passthroughs to the ones here.
| Type | Visibility | Attributes | Name | Initial | |||
|---|---|---|---|---|---|---|---|
| type(parquet_column_type), | public, | allocatable | :: | col(:) |
One entry per declared field, in MAML source order. |
| procedure, public :: get_column_index | 1-based index of a column by name; error stops if not found. |
| procedure, public :: is_column_set | Whether a column is currently enabled to be written; error stops if not found. |
| procedure, public :: get_num_fields | Total number of declared fields. |
| procedure, public :: get_field_name | Field name at a given 1-based MAML source position. |
| procedure, public :: get_field_by_name | Full field definition by name; error stops if not found. |
| procedure, public :: get_field_by_index | Full field definition by 1-based MAML source position. |
| generic, public :: get_field => get_field_by_name, get_field_by_index | Reads back a field's full, add_field-equivalent definition, by name or by 1-based source position. |
| procedure, public :: set_column_unavailable => set_unavailable | Disables a column, or every column if no name is given. |
| procedure, public :: set_column_available => set_available | Enables a column, or every column if no name is given. |
| procedure, public :: set_col_size | Resolves a column's col_size (only if currently "auto" unless force=.true.). |
| procedure, public :: set_protected | Marks/unmarks a column Null-protected (extra: protected_cols:). |
| procedure, public :: set_array_size | Resolves a string column's array_size (only if currently "auto" unless force=.true.). |
Flat key-value table metadata: one array of parquet_metadata_entry plus the add_metadata family of type-bound procedures that append to it (one specific per supported type/kind, dispatched via the add_metadata generic).
| Type | Visibility | Attributes | Name | Initial | |||
|---|---|---|---|---|---|---|---|
| type(parquet_metadata_entry), | public, | allocatable | :: | items(:) |
One entry per add_metadata call (or MAML keyarray: item). |
||
| character(len=:), | public, | allocatable | :: | source_maml_lines(:) |
Verbatim source MAML lines, populated by parquet_read_maml; used by parquet_open_writer(..., write_maml=.true.) to save a sidecar .maml file next to the .parquet output. add_metadata calls made after parquet_read_maml append a new keyarray: entry to these lines (see parquet_append_keyarray_line), so the sidecar reflects them; it is NOT kept in sync with which columns end up enabled/written, though. |
||
| integer, | public | :: | n_base_items | = | 0 |
Count of %items present immediately after the most recent parquet_parse_maml (table:/survey:/... header keys plus any real MAML keyarray: entries) -- the "base" metadata %clear_metadata preserves, discarding only entries appended by %add_metadata calls made after that parse. 0 for a never-parsed table_metadata. |
| procedure, public :: clear_metadata => metadata_clear_metadata | Discards %add_metadata entries added after the most recent parse, keeping the base (header keys + keyarray:) entries. |
| procedure, public :: add_metadata_int32 | int32 specific. |
| procedure, public :: add_metadata_int64 | int64 specific. |
| procedure, public :: add_metadata_float32 | float32 specific. |
| procedure, public :: add_metadata_float64 | float64 specific. |
| procedure, public :: add_metadata_logical | logical specific. |
| procedure, public :: add_metadata_string | string specific. |
| procedure, public :: add_metadata_int32_array | int32 array specific. |
| procedure, public :: add_metadata_int64_array | int64 array specific. |
| procedure, public :: add_metadata_float32_array | float32 array specific. |
| procedure, public :: add_metadata_float64_array | float64 array specific. |
| procedure, public :: add_metadata_logical_array | logical array specific. |
| procedure, public :: add_metadata_string_array | string array specific. |
| generic, public :: add_metadata => add_metadata_int32, add_metadata_int64, add_metadata_float32, add_metadata_float64, add_metadata_logical, add_metadata_string, add_metadata_int32_array, add_metadata_int64_array, add_metadata_float32_array, add_metadata_float64_array, add_metadata_logical_array, add_metadata_string_array | Appends one flat key-value metadata entry; dispatched by value's type/kind/rank. |
Bundles a parsed MAML source together with the column schema (cinfo) and table metadata (metadata) that parquet_parse_maml derives from it, so a single variable carries everything parquet_open_writer needs. %cinfo and %metadata are public: their fields (col(:), items(:)) and own type-bound procedures stay directly reachable, and the procedures below are flat convenience passthroughs (schema%set_column_available("id") instead of schema%cinfo%set_column_available("id")).
| Type | Visibility | Attributes | Name | Initial | |||
|---|---|---|---|---|---|---|---|
| type(parquet_maml_file), | public | :: | maml |
Raw MAML source plus (once parsed) missing-columns/col_map. |
|||
| type(parquet_column_info), | public | :: | cinfo |
Per-field schema/QC state, derived from %maml by parquet_parse_maml. |
|||
| type(parquet_table_metadata), | public | :: | metadata |
Flat key-value table metadata, derived from %maml's keyarray:. |
Overrides the default structure constructor so a schema can be built in one expression (my_maml = parquet_schema(table="my_table")) as an alternative to call my_maml%init(table="my_table"); both call parquet_schema_new/schema_init under the hood. table is the only required argument (the MAML table: key); survey, dataset, version, date, author, description, license, and maml_version are all optional and set the correspondingly-named MAML header key when given. Returns the newly initialized schema.
| private interface parquet_schema_new () |
| procedure, public :: init => schema_init | Initializes a from-scratch schema (table: key + optional metadata). |
| procedure, public :: is_init => schema_is_init | Whether this schema is ready to use (via %init or a MAML parse). |
| procedure, public :: is_parsed => schema_is_parsed | Whether %cinfo has actually been populated by parquet_parse_maml. |
| procedure, public :: clear => schema_clear | Resets the entire schema back to its pristine, just-declared (never-initialized) state. |
| procedure, public :: add_field => schema_add_field | Appends one fields: entry to a from-scratch schema. |
| procedure, public :: add_field_from => schema_add_field_from | Copies one field's definition from another (already-parsed) schema and appends it here via %add_field. |
| procedure, public :: set_column_available | Enables a column, or every column if no name is given. |
| procedure, public :: set_column_unavailable | Disables a column, or every column if no name is given. |
| procedure, public :: set_col_size => schema_set_col_size | Resolves a column's col_size before parquet_open_writer. |
| procedure, public :: set_protected => schema_set_protected | Marks/unmarks a column Null-protected. |
| procedure, public :: set_array_size => schema_set_array_size | Resolves a string column's array_size before parquet_open_writer. |
| procedure, public :: get_column_index => schema_get_column_index | 1-based index of a column by name; error stops if not found. |
| procedure, public :: is_column_set => schema_is_column_set | Whether a column is currently enabled to be written; error stops if not found. |
| procedure, public :: get_num_fields => schema_get_num_fields | Total number of declared fields. |
| procedure, public :: get_field_name => schema_get_field_name | Field name at a given 1-based MAML source position. |
| procedure, public :: get_field_by_name => schema_get_field_by_name | Full field definition by name; error stops if not found. |
| procedure, public :: get_field_by_index => schema_get_field_by_index | Full field definition by 1-based MAML source position. |
| generic, public :: get_field => get_field_by_name, get_field_by_index | Reads back a field's full, add_field-equivalent definition, by name or by 1-based source position; forwards to %cinfo%get_field. |
| procedure, public :: print_schema_info => schema_print_schema_info | Writes a "Table name:" line plus an aligned name/unit/type/len/ucd/info listing of enabled (is_set) columns to a unit/file. |
| procedure, public :: add_col_qc => schema_add_col_qc | Appends one qc: field entry from a compact string. |
| procedure, public :: set_col_qc => schema_set_col_qc | In-place form of %add_col_qc; parses the name into its argument. |
| procedure, public :: schema_add_metadata_int32 | int32 specific. |
| procedure, public :: schema_add_metadata_int64 | int64 specific. |
| procedure, public :: schema_add_metadata_float32 | float32 specific. |
| procedure, public :: schema_add_metadata_float64 | float64 specific. |
| procedure, public :: schema_add_metadata_logical | logical specific. |
| procedure, public :: schema_add_metadata_string | string specific. |
| procedure, public :: schema_add_metadata_int32_array | int32 array specific. |
| procedure, public :: schema_add_metadata_int64_array | int64 array specific. |
| procedure, public :: schema_add_metadata_float32_array | float32 array specific. |
| procedure, public :: schema_add_metadata_float64_array | float64 array specific. |
| procedure, public :: schema_add_metadata_logical_array | logical array specific. |
| procedure, public :: schema_add_metadata_string_array | string array specific. |
| generic, public :: add_metadata => schema_add_metadata_int32, schema_add_metadata_int64, schema_add_metadata_float32, schema_add_metadata_float64, schema_add_metadata_logical, schema_add_metadata_string, schema_add_metadata_int32_array, schema_add_metadata_int64_array, schema_add_metadata_float32_array, schema_add_metadata_float64_array, schema_add_metadata_logical_array, schema_add_metadata_string_array | Appends one flat key-value metadata entry; dispatched by value's type/kind/rank. |
| procedure, public :: clear_metadata => schema_clear_metadata | Forwards to %metadata%clear_metadata. |
parquet_writer/parquet_reader own a handle to a C++-side Arrow/Parquet object with no automatic Fortran cleanup. Always prefer an explicit parquet_close_writer/parquet_close_reader call; the FINAL procedures below are only a safety net for a handle that's still open when its variable goes out of scope or is overwritten (e.g. an early RETURN between open and close), not a substitute for closing normally -- for a writer specifically, the safety net skips parquet_close_writer's completeness checks (so an incomplete write never crashes an implicit finalizer), meaning the resulting file is not guaranteed valid/complete unless parquet_close_writer was actually called.
| final :: writer_finalize | Safety-net close if the writer is still open when it goes out of scope. |
Opaque handle for an open parquet file being read; see parquet_writer's doc comment above for the shared handle-ownership/no-copy rules.
| final :: reader_finalize | Safety-net close if the reader is still open when it goes out of scope. |
A row filter for parquet_open_reader/parquet_reader_set_filter: each %add
call contributes one boolean expression over the file's columns, and
several %add calls are AND-combined, i.e. (expr1) and (expr2). One
expression is either a single clause, "
| Type | Visibility | Attributes | Name | Initial | |||
|---|---|---|---|---|---|---|---|
| character(len=:), | public, | allocatable | :: | rules(:) |
Raw, unvalidated rule text, one entry per %add call. Deferred-length: every entry shares the length of the longest rule added so far (%add grows it as needed), so a parenthesised multi-clause expression is not constrained by a fixed component width. |
||
| integer, | public | :: | n | = | 0 |
Number of rules actually in use. |
| procedure, public :: add => parquet_filter_add | Appends one AND-combined filter expression. |
| procedure, public :: remap_column_names => parquet_filter_remap_column_names | Renames the columns every rule refers to, in place: |
A read-time sort specification: an ordered list of sort KEYS, each naming one column and the direction to order it by. Passed as parquet_open_reader(..., sort_by=), or applied to an already-open reader with parquet_reader_set_sort. Every column read afterwards comes back in that order.
| Type | Visibility | Attributes | Name | Initial | |||
|---|---|---|---|---|---|---|---|
| character(len=:), | public, | allocatable | :: | keys(:) |
Raw, unvalidated key text, one entry per %add call. Deferred-length, the same way parquet_filter%rules is: every entry shares the length of the longest key added so far. |
||
| logical, | public, | allocatable | :: | nulls_first(:) |
Per key: .true. to place that key's null rows before its values instead of after. Same extent as keys(1:n). |
||
| integer, | public | :: | n | = | 0 |
Number of keys actually in use. |
| procedure, public :: add => parquet_sortkey_add | Appends one sort key, applied after those already added. |
| procedure, public :: remap_column_names => parquet_sortkey_remap_column_names | Renames the column every key orders by, in place: |
Read-time quality control declared in CODE rather than in a MAML file: one column per %add call, in exactly the compact "col, min, max, miss" string parquet_schema%add_col_qc already takes, so there is one read-time-QC grammar in this library rather than two. See "Building a qc-maml in code" in doc/pages/schema/quality-control.md for the field syntax -- the operator prefixes (>, >=, <, <=) and the Null/NA/empty miss: convention are inherited from %add_col_qc verbatim.
| Type | Visibility | Attributes | Name | Initial | |||
|---|---|---|---|---|---|---|---|
| character(len=:), | public, | allocatable | :: | entries(:) |
Raw, unvalidated entry text, one per %add call. Deferred-length: every entry shares the length of the longest added so far, the same way parquet_filter%rules does. |
||
| integer, | public | :: | n | = | 0 |
Number of entries actually in use. |
| procedure, public :: add => parquet_read_qc_add | Appends one "col, min, max, miss" declaration. |
| procedure, public :: remap_column_names => parquet_read_qc_remap_column_names | Renames the column each entry declares, in place: |
Splits a scalar string of column names into the packed array every name-taking array form expects. Separators are commas and semicolons, interchangeably; each token is trimmed of surrounding blanks, and an empty token is dropped rather than being an error (so "a,,b" is two names and "" is none).
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| character(len=*), | intent(in) | :: | text |
names separated by commas and/or semicolons. |
||
| character(len=:), | intent(out), | allocatable | :: | names(:) |
one entry per non-empty token. |