parquet_open_table Interface

public interface parquet_open_table

Opens a file-backed table, over the whole file or over one contiguous row slice.

Given row_lo/row_hi (1-based, inclusive, in either integer kind) the table covers only those rows, and reads only the row groups covering them -- this is how a file bigger than memory is worked through, and how a parallel program gives each thread its own share (parquet_table_row_group_bounds reports where the natural boundaries are). Row indices everywhere else, %row(i) included, are then relative to the slice, not to the file.

A slice accepts the same read-time transform the whole-file form does, with ONE exception: there is no sort argument on the slice forms at all. A sort reorders rows across the whole file, so "the 1000th row" would no longer name anything a slice could be cut along -- and omitting the argument makes that a compile error rather than a runtime one. (A maml= whose extra: sort: list is non-empty is the same rejection, necessarily at runtime.)

A filtered or sampled slice does not have row_hi - row_lo + 1 rows. %nrows() is the number of rows of [row_lo, row_hi] that survive the transform, and every row index the table takes or reports counts those survivors -- row 1 is the first surviving row, not file row row_lo. Without a filter and without sample_fraction= nothing changes: the slice is trimmed out of the covering row groups in memory, exactly as it always was.


Module Procedures

private interface open_table_full()

Arguments

None

private interface open_table_slice_i32()

Arguments

None

private interface open_table_slice_i64()

Arguments

None