parquet_tables_maml Submodule

Read-in (Role-B) MAML support for parquet_table: parsing the extra: remap: block that relabels a file's columns for reading, and building the descriptor slots from it.

A read-in MAML describes the PHYSICAL file and travels with it, so everything it names is a file column name. remap: is the one place the two vocabularies meet: its keys are the table-facing (internal) names a program will use, its values the file columns they read. Everything downstream then keeps the split the descriptor already has -- %name is the lookup key, %file_name is what the reader is asked for -- which is why remapping needs no change to classification, materialization or residency at all.

Three rules this file implements, all of them easy to get subtly wrong:

  • An internal name may equal a physical column name it does not refer to. A file often carries columns a program does not want, and one of them happening to share a name with an internal name must not make that name unusable. The physical column is then simply unreachable -- a deliberate, silent shadow, not an error.
  • Two internal names may read the SAME physical column. They become two ordinary, independent columns from that point on: each materializes on its own first touch (reading the physical column again) and each may be mutated without the other seeing it.
  • File order is preserved. Slots are emitted by walking the physical columns, not the remap list, so a struct's leaves stay adjacent -- which the batch release policy in materialize_marked depends on -- and duplicate slots land next to each other, where one decode serves both.

The parser is deliberately independent of the write-side parquet_parse_col_map, which is validated against a base schema a generic table has no equivalent of and serves the opposite direction. It scans lines already loaded from disk (via parquet_load_qc_maml_file), so the shared reader's line-length cap and CRLF handling apply without this file re-reading anything.


Uses