parquet_writer/parquet_reader own a handle to a C++-side Arrow/Parquet object with no automatic Fortran cleanup. Always prefer an explicit parquet_close_writer/parquet_close_reader call; the FINAL procedures below are only a safety net for a handle that's still open when its variable goes out of scope or is overwritten (e.g. an early RETURN between open and close), not a substitute for closing normally -- for a writer specifically, the safety net skips parquet_close_writer's completeness checks (so an incomplete write never crashes an implicit finalizer), meaning the resulting file is not guaranteed valid/complete unless parquet_close_writer was actually called.
Do not copy a parquet_writer/parquet_reader (w2 = w1, passing one as
a function result, etc.): the handle is a plain c_ptr, so a copy
aliases the same underlying C++ object without any reference counting.
Whichever copy is finalized/closed first frees it out from under the
other, which would then double-free/use-after-free when it is itself
later closed or finalized. Always use a single named writer/reader,
passed by reference (as every procedure in this module already does).
Fully opaque from outside this module: every component is an
implementation detail (the raw C handle, schema bookkeeping, write
tracking) that parquet_write_column/parquet_open_writer/etc. manage
internally. Never referenced directly by any test or consumer -- only
parquet_core.f90's own submodules (parquet_write, parquet_read,
parquet_metadata) need access, which private here still allows.
Safety-net close if the writer is still open when it goes out of scope.
FINAL procedure: safety-net close for a writer whose variable goes out of scope (or is overwritten) still open; see parquet_writer's own doc comment for why this is not a substitute for %close.
| Type | Intent | Optional | Attributes | Name | ||
|---|---|---|---|---|---|---|
| type(parquet_writer), | intent(inout) | :: | this |
writer being finalized. |
type parquet_writer private type(c_ptr) :: handle = c_null_ptr !! Opaque C++ Arrow/Parquet writer handle; c_null_ptr until opened. type(parquet_column_type), allocatable :: all_columns(:) !! Every column the schema declares, enabled or not. type(parquet_column_type), allocatable :: enabled_columns(:) !! Subset of all_columns currently enabled (is_set). integer, allocatable :: write_counts(:) !! Per-enabled-column count of parquet_write_column calls so far. !> Per-all_columns longest element (in characters) any parquet_string_column write has !> actually written to this column; 0 for a column no such write touched. Unallocated for a !> schema-less writer. !> !> The compact path neither reads nor enforces a declared array_size -- it stores each !> element's own bytes, and a reader takes each length from the data -- so a caller may !> legitimately write elements longer than the schema declares. What must NOT happen is the !> file then advertising the declaration: parquet_close_writer reconciles the two !> (parquet_reconcile_string_sizes) so the sidecar .maml and the file's own !> column.<name>.array_size describe what was written. Accepting loose input is a choice; !> emitting wrong metadata is not. integer, allocatable :: observed_string_len(:) logical :: is_schema_enforced = .false. !! true when opened with a schema (vs. a schema-less writer). logical :: qc = .false. !! defaults to present(schema) (i.e. on whenever a schema is given), overridable !! via parquet_open_writer(..., qc=); when true, parquet_write_column checks each column's qc: min/max !! (if declared) against its valid (is_valid) elements, and -- only for a column declaring an explicit, !! empty qc: miss: -- its Null elements, printing a WARNING (never an error) on violation. !! No-op without a schema. integer(c_long_long) :: expected_nrows = -1 !! set by the first parquet_write_column call; every later !! call must supply this same row count (see parquet_check_row_count), since !! Arrow/Parquet requires every column in a table to have equal length. character(len=256), allocatable :: written_names(:) !! Schema-less writer only (no cinfo, so !! enabled_columns/write_counts below don't exist): every name !! parquet_write_column has already written, so a repeat can still be !! caught -- see parquet_mark_column_written. character(len=:), allocatable :: maml_name !! Set from schema%maml%name by parquet_open_writer !! when a schema is given (unallocated for a schema-less writer, or if !! the schema's own %maml%name was never set); solely so !! parquet_write_column's "column not defined"/"type mismatch" errors !! can name which maml the schema came from -- see writer_maml_suffix. !> Set by parquet_open_writer from its own `filename` argument, solely !> so parquet_close_writer's missing-write error can name the output !> file; not used for anything else. character(len=:), allocatable :: filename !> true when parquet_open_writer(..., write_maml=.true.) was requested -- the sidecar !> .maml itself is written by parquet_close_writer (not parquet_open_writer), once every !> column's col_size/array_size is guaranteed resolved (no longer "auto"), so the sidecar !> always reflects what was actually written to the .parquet file. logical :: write_maml_requested = .false. !> Pruned (disabled fields removed) working copy of schema%metadata%source_maml_lines, !> saved by parquet_open_writer when write_maml_requested; parquet_close_writer rewrites !> its col_size:/array_size: values to their final resolved state and writes it out. character(len=:), allocatable :: sidecar_lines(:) !> True between parquet_new_row_group and its matching parquet_finish_row_group -- !> mirrors the C++-side flag of the same purpose; kept here too so a !> parquet_write_column_chunk call can validate its own row count against !> current_row_group_nrows (below) without a round trip into C++. logical :: in_row_group = .false. !> The `nrows` most recently passed to parquet_new_row_group, valid only while !> in_row_group is .true. -- every parquet_write_column_chunk call for the open row !> group must supply exactly this many rows. integer(c_long_long) :: current_row_group_nrows = 0 !> Row-filtering ("mask") state -- see parquet_write_row_mask/parquet_write_chunk_row_mask !> and doc/pages/io/writing.md's "Filtering rows with a mask". The two masking schemes are !> mutually exclusive per writer; only the fields relevant to whichever scheme (if any) is !> actually used are ever populated. logical, allocatable :: file_mask(:) !! Whole-file mask set by parquet_write_row_mask; unallocated if unused. integer(c_long_long) :: mask_cursor = 0 !! Cursor of file_mask positions already claimed by row groups !! (parquet_write_row_mask + row groups scheme only); see parquet_new_row_group_impl. logical :: mask_used_with_row_groups = .false. !! true once file_mask windowing has been applied by at !! least one parquet_new_row_group call -- drives the mask-fully-consumed check at !! parquet_close_writer (irrelevant/unused for a pure whole-column masked writer). logical, allocatable :: chunk_mask(:) !! Mask applicable to the currently-open row group: an identity !! mask by default, a file_mask window (shared-mask scheme), or an explicit !! parquet_write_chunk_row_mask call (per-row-group scheme) -- reset every parquet_new_row_group. logical :: chunk_mask_set_this_group = .false. !! true once parquet_write_chunk_row_mask has been called !! for the currently-open row group; reset by parquet_new_row_group. integer :: chunk_mask_scheme = 0 !! 0 = undecided (no row group's first chunk write yet), 1 = per-row-group !! masking (parquet_write_chunk_row_mask) used for every row group, 2 = never used -- fixed permanently !! at the first parquet_write_column_chunk call of the writer's first row group. logical :: row_group_first_write_done = .false. !! true once any parquet_write_column_chunk call has !! happened for the currently-open row group; reset by parquet_new_row_group. Rejects a !! parquet_write_chunk_row_mask call made too late (after that row group's first chunk write). logical :: cpp_row_group_open = .false. !! true once the underlying C++ row group has actually been !! opened (parquet_writer_new_row_group) for the currently-open Fortran-level row group; reset by !! parquet_new_row_group. The C++ open must be told the row group's post-mask *kept* row count, which !! isn't known at parquet_new_row_group time for the per-row-group mask scheme (its mask arrives in a !! separate, later call) -- so the open is deferred until the kept count is actually known: immediately !! if the shared whole-file mask windowing applies, else at whichever comes first of !! parquet_write_chunk_row_mask or this row group's first parquet_write_column_chunk call. logical :: row_group_is_empty = .false. !! true once the currently-open row group's post-mask kept !! count is known to be exactly 0 -- a zero-row row group has no underlying C++ row group at all (Arrow's !! own NewRowGroup requires a positive row count), so every parquet_write_column_chunk call for it must !! skip its C++ append entirely (nothing to append into) and parquet_finish_row_group must skip both the !! open and the finish. Reset by parquet_new_row_group. logical :: any_whole_column_write = .false. !! true once any parquet_write_column call has actually !! proceeded (not skipped as disabled) -- parquet_write_chunk_row_mask is unavailable once this is !! true (only the whole-file parquet_write_row_mask scheme can mask a writer with whole-column writes). logical :: write_started = .false. !! true once the writer's first parquet_write_column or !! parquet_new_row_group call happens -- parquet_write_row_mask must be called before this. contains final :: writer_finalize !! Safety-net close if the writer is still open when it goes out of scope. end type parquet_writer