Skip to content

Loader Context and Report

This page documents the shared data structures used by the loaders subsystem.

LoaderContext

LoaderContext is an immutable coordination object passed through all loader operations.

It encapsulates all state required to load a file without relying on globals or implicit configuration.

Fields

Field Description
tableclass Target ORM table class
session Active SQLAlchemy session
path Path to the input file
backend The session's database backend, resolved once for the load
normalise Whether to interpret values under the policy; when false, values are written as read
dedupe Whether the staging table keeps only the first row for each primary key
policy ConversionPolicy: how values are interpreted and what happens to unreadable ones
rejects_dir Directory that rejected rows are written to
report LoadReport row accounting

Immutable context passed through a load.

The context holds all state required to load a file into a staging table, without relying on global variables.

Attributes:

Name Type Description
tableclass type[FileTableProtocol]

ORM table class being loaded.
session Session

Active SQLAlchemy session.
path Path

Path to the source data file.
backend DatabaseBackend

Backend of the session's database, resolved once for the load.
normalise bool

Whether to interpret values under the policy.
dedupe bool

Whether the staging table keeps only the first row for each primary key.
policy ConversionPolicy

How values are interpreted and what happens to unreadable ones.
rejects_dir Path | None

Directory that rejected rows are written to. Defaults to a rejects directory inside the source file's directory.
report LoadReport

Row accounting, filled in as the load runs.

LoadReport

LoadReport accounts for every row of a load: rows read, loaded, rejected and removed as duplicates. DedupeResult holds the key columns and the rows removed. A load that cannot account for every row raises.

Row accounting for one load.

Attributes:

Name Type Description
rows_read int

Data rows read from the file.
rows_loaded int

Rows written to the staging table.
rows_rejected int

Rows dropped because a value was unreadable, a required value was missing or a line was malformed, with ErrorPolicy.on_error set to "reject_row".
dedupe DedupeResult | None

Result of deduplicating the staging table, if it was requested.
rejects_path Path | None

File the rejected rows were written to, if any.

accounted property

Whether every row read was loaded, rejected or dropped as a duplicate.

rows_deduplicated property

Rows removed as duplicates, zero when deduplication did not run.

What deduplicating a staging table removed.

Attributes:

Name Type Description
key_columns tuple[str, ...]

Primary key columns that identify a duplicate.
rows_before int

Rows in the staging table before deduplication.
rows_removed int

Rows removed because an earlier row had the same key.

rows_after property

Rows left in the staging table.

Design notes

  • Unreadable values and missing required values raise by default, so no row disappears without a trace
  • With ErrorPolicy(on_error="reject_row") each rejected row is written to a rejects file with its raw text and the reason