Readers

A reader yields a file as Arrow record batches holding the model columns present in the file. Text formats yield raw strings, so the conversion policy sees exactly what the file contains. Typed formats yield their stored types. A reader decides nothing about missing values: an empty field is an empty string, and the null policy is applied afterwards.

open_reader picks the reader from the file suffix: .parquet is read as Parquet, any other suffix as delimited text. load_file opens the reader with it, taking quote_mode and chunksize as arguments, and load_file(reader=...) passes a configured reader explicitly.

Reader	Input
DelimitedTextReader	Delimited text with a tab or comma delimiter and detected encoding
ParquetReader	Parquet files, in batches of chunksize rows

---

FileReader

A new format is a new FileReader subclass that implements batches() and sets columns and empty.

Bases: ABC

A file read as Arrow record batches of the model's columns.

A reader yields batches holding only the model columns present in the file, named as the model names them. Text formats yield raw strings, so the null policy sees exactly what the file contains. Typed formats yield typed columns. A reader decides nothing about missing values.

Parameters:

Name	Type	Description	Default
path	Path	File to read.	required
model_columns	Mapping[str, ColumnElement[Any]]	Model columns by name; only these are read.	required

Attributes:

Name	Type	Description
columns	list[str]	Model columns present in the file, in file order.
file_columns	list[str]	Every column of the file under its normalised name, in file order.
empty		True when the file has no content at all.
skipped	list[tuple[int | None, str]]	Lines the reader left out as malformed, each as the number of the line in the file (None when unknown) and the reason, filled in while the batches are read.

batches() abstractmethod

Yield the file's rows as non-empty record batches.

normalise_headers(names)

Return header names lowercased, stripped of quotes and padding and of a _hash suffix.

Raises:

Type	Description
IngestError	Two columns have the same normalised name.

---

DelimitedTextReader

Reads raw strings in 4 MB blocks, so memory does not grow with the row count. Header names are lowercased and stripped of quotes, padding and a trailing _hash, and file columns that are not model columns are ignored with a warning. Parquet column names are normalised the same way.

quote_mode selects how double quotes are read:

Mode	Behaviour
csv	Double quotes quote fields
literal	Double quotes are data
by_delimiter (default)	literal for tab-delimited files, csv otherwise

Bases: FileReader

A delimited text file read as raw strings in bounded blocks.

Every value is kept as read, so an empty field is an empty string and nothing is null. Date columns are dictionary-encoded. Header names are normalised as described by normalise_headers. File columns that are not model columns are ignored.

Parameters:

Name	Type	Description	Default
path	Path	File to read.	required
model_columns	Mapping[str, ColumnElement[Any]]	Model columns by name; only these are read.	required
quote_mode	str | None	"csv" reads double quotes as field quoting, "literal" keeps them as data, "by_delimiter" chooses "literal" for tab delimited files and "csv" otherwise. None uses DEFAULT_QUOTE_MODE.	None
block_size	int	Bytes per block; memory use follows this, not the row count.	4 << 20
skip_malformed	bool	Leave out a line with the wrong number of fields and record it in skipped, instead of failing the read.	False

Attributes:

Name	Type	Description
DEFAULT_QUOTE_MODE		Quote mode used when none is given.

batches()

Yield the file's rows in blocks of block_size bytes.

---

ParquetReader

Bases: FileReader

A Parquet file read with its stored types.

Column names are normalised as described by normalise_headers.

Parameters:

Name	Type	Description	Default
path	Path	File to read.	required
model_columns	Mapping[str, ColumnElement[Any]]	Model columns by name; only these are read.	required
chunksize	int | None	Rows per batch.	None

batches()

Yield the stored rows in batches of chunksize rows.

---

open_reader

Open a file with the reader for its format.

A .parquet suffix selects Parquet; every other suffix is read as delimited text.

Parameters:

Name	Type	Description	Default
path	Path	File to read.	required
model_columns	Mapping[str, ColumnElement[Any]]	Model columns by name; only these are read.	required
quote_mode	str | None	Quote handling for delimited text. None uses the reader's default.	None
chunksize	int | None	Rows per batch for formats that batch by rows.	None
skip_malformed	bool	Leave out lines with the wrong number of fields and record them in the reader's skipped list. Delimited text only.	False

Returns:

Type	Description
FileReader	The opened reader.

---

Detection

The delimited reader detects the encoding with chardet (ASCII is normalised to UTF-8, and only a small prefix of the file is inspected), the delimiter from the first line (tabs win over commas when more frequent).

Resolve a requested quote_mode to a concrete "csv" or "literal".

Parameters:

Name	Type	Description	Default
quote_mode	str	"csv" reads double quotes as RFC 4180 field quoting. "literal" keeps them as data. "by_delimiter" chooses "literal" for tab delimited files, where a field never embeds the delimiter, and "csv" otherwise.	required
delimiter	str	Delimiter of the file.	required

Returns:

Type	Description
str	"csv" or "literal".

Raises:

Type	Description
ValueError	quote_mode is not one of the above.