parquet_reader_set_filter Interface

public interface parquet_reader_set_filter

Applies a filter to an already-open reader (see the specifics' own doc-comments for the full contract). Two forms: whole-file, and scoped to an inclusive 1-based row-group range.

The scoped form is the memory-bounded one, and the difference is in how the mask is BUILT, not merely in which rows survive. Whole-file, every filter column is read in one batched, thread-parallel pass and left decoded in the reader's cache -- fastest, and the right default. Scoped, the expression is evaluated one row group at a time and each chunk is released before the next is read, so peak memory is one row group's worth of the filter columns rather than the whole file; rows outside the range never match, and nothing is left cached, so a filter column read afterwards is read again. Use it when the file is larger than memory, or when only part of it is of interest -- typically alongside chunked reads over the same row groups.

It is the PRESENCE of the row-group arguments that selects between those two engines, not their value: row_group_lo = 0 means "all row groups" on the memory-bounded engine (and row_group_hi is then ignored), which is how a whole file larger than memory is filtered without first asking parquet_get_num_row_groups how many there are. A non-positive lower bound reads as "all row groups" here exactly as it already does in parquet_measure_list_width and parquet_column_has_nulls.

A third form takes a physical ROW range as well (four numeric arguments rather than two): only rows row_lo..row_hi, 1-based and inclusive, may match. It exists because a row-group range can only ever start and end on a row-group boundary, so a caller working over an arbitrary row range -- a parquet_table slice, typically -- would otherwise get back the whole covering row groups' survivors and have no way to trim them, the mask being the only thing that knows which rows those are. With this form the filter answers for exactly the requested rows, and parquet_get_nrows afterwards is that range's own surviving count. The filter may hold no rules at all in this form, which installs the range by itself. The row range must lie inside the rows its row groups span, or the call fails with error stop rather than quietly handing back the intersection of the two -- which for a disjoint pair is empty, and an empty result is indistinguishable from a filter that matched nothing.


Module Procedures

private interface parquet_reader_set_filter_base()

Arguments

None

private interface parquet_reader_set_filter_scoped_int32()

Arguments

None

private interface parquet_reader_set_filter_scoped_int64()

Arguments

None

private interface parquet_reader_set_filter_rows_int32()

Arguments

None

private interface parquet_reader_set_filter_rows_int64()

Arguments

None