core¶
Foundational services with no clinical-domain assumptions. A concept resolver behaves
the same whether it is mapping tumour morphology or procedures; a patient timeline is
the same object whatever populates it. Domain-specific concept sets, thresholds, and
grading rules belong in analytics, not here.
Concept resolution¶
Turns a declarative description of which concepts belong in a lookup into a runtime resolver that maps free text and source codes to OMOP concept IDs.
from omop_alchemy.toolkit.core.concepts import make_concept_resolver
resolver = make_concept_resolver(
session,
name="condition lookup",
domain_id="Condition",
)
concept_id = resolver.lookup("Adenocarcinoma of lung")
Resolve free-text terms and source codes to OMOP concept IDs.
Source data rarely arrives with concept IDs attached. This module turns a declarative description of which concepts are eligible into a runtime resolver that maps incoming text to those concepts, applying the same normalisation on both sides so that matching is predictable.
Three pieces make up the workflow:
LookupSpec
Declares which concepts belong in a lookup — by vocabulary, domain,
concept class, or explicit ancestry — and which text fields are
indexed.
LookupIndex
The materialised table of normalised text keys to concept IDs that a
spec produces against the vocabulary tables.
ConceptResolver
Wraps an index and resolves terms at runtime, applying the same
normalisation used to build the keys.
make_concept_resolver bundles all three for the common case::
from omop_alchemy.toolkit.core.concepts import make_concept_resolver
resolver = make_concept_resolver(
session,
name="condition lookup",
domain_id="Condition",
)
concept_id = resolver.lookup("Adenocarcinoma of lung")
Normalisation is composable. compose_normalizers chains individual
rules — normalize_default for whitespace and casing, strip_uicc
and make_stage for staging text, site_to_NOS for site
generalisation — so that a resolver's matching behaviour is stated
explicitly rather than implied.
Building an index queries the vocabulary tables, so resolvers are worth
reusing. ConceptResolverRegistry constructs each resolver on first
access and caches it for the registry's lifetime.
Governed concept sets are the other half of this module. Where a resolver maps text to concepts, a concept group answers whether a concept ID belongs to a governed clinical set::
from omop_alchemy.toolkit.core.concepts import (
ConceptGroupSpec,
resolve_concept_group,
)
RT = ConceptGroupSpec(name="radiotherapy", unit=...) # governed anchors
group = resolve_concept_group(session, RT)
procedure.procedure_concept_id in group # Python, O(1)
group.expression(Procedure_Occurrence.procedure_concept_id) # SQL
Both access paths derive from one spec, so they cannot disagree. Expansions are
cached per vocabulary, not per engine, so recreating an engine against the same
database does not re-run the closure queries — see :mod:.identity for how an
engine declares which vocabulary it reads, and note that a caller building its
own engines must register that itself.
Nothing here touches a database at import time: specs are declarative and registries build on first request.
CacheStats
dataclass
¶
CacheStats(
entries: int = 0,
cached_bytes: int = 0,
evictions: int = 0,
rebuilds_after_evict: int = 0,
)
Observability for a bounded registry.
rebuilds_after_evict is the load-bearing number. Evicting an entry
that is never asked for again is exactly what a bound is for; evicting one
that is then rebuilt is thrashing, and it is otherwise invisible because it
presents as ordinary slowness rather than as a cache problem. While it
stays at zero the bound is correct.
ConceptGroupRegistry ¶
ConceptGroupRegistry(
engine: Engine,
*,
max_bytes: int = DEFAULT_MAX_CACHE_BYTES,
)
Bases: _LazyBoundedRegistry[ResolvedConceptGroup]
Lazy registry for governed concept groups, scoped to one vocabulary.
Obtain one through :func:concept_group_registry rather than constructing
it directly, so registries are shared per vocabulary identity instead of
per engine.
register_spec ¶
register_spec(spec: ConceptGroupSpec) -> None
Register spec under its governed name, if not already present.
ConceptGroupSpec
dataclass
¶
ConceptGroupSpec(
name: str,
unit: Any,
include_descendants: bool = True,
require_standard: bool = False,
include_classification: bool = True,
)
A governed omop-semantics semantic unit plus how to expand it.
Declarative and side-effect free — constructing one performs no I/O and touches no database.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
Stable identifier for this group, used as the cache key. Use the governed semantic-unit name so the key derives from the governed identity rather than a locally invented label. |
required |
unit
|
Any
|
The omop-semantics |
required |
include_descendants
|
bool
|
Expand |
True
|
require_standard
|
bool
|
Restrict expansion to concepts carrying a standardness flag. Defaults to
False, matching the historical oncology behaviour of reading
|
False
|
include_classification
|
bool
|
Widens |
True
|
exact_ids ¶
exact_ids() -> tuple[int, ...]
Governed members matched directly, without descendant expansion.
excluded_parent_ids ¶
excluded_parent_ids() -> tuple[int, ...]
Governed anchors whose descendants are subtracted.
expression_for ¶
expression_for(
column: SQLColumnExpression[Any],
) -> sa.ColumnElement[bool]
SQL membership for this group, as a subquery over concept_ancestor.
Available on the spec because the SQL form needs no session: the
traversal is performed by the database when the query runs. That is
what lets a hybrid_property expose the same governed set at class
level, where no session exists.
Deliberately a subquery rather than a literal IN list built from a
resolved group: a closure of tens of thousands of IDs degrades query
plans and can exceed driver parameter limits.
ConceptResolver ¶
ConceptResolver(
index: LookupIndex,
*,
normalizer: Normaliser | None = None,
corrections: list[Callable[[str], str]] | None = None,
)
Runtime resolver for mapping free-text terms to OMOP concept IDs.
A ConceptResolver wraps a pre-built LookupIndex and applies runtime normalisation and optional correction passes to resolve arbitrary input strings to concept IDs. It is intentionally lightweight and stateless: all semantic scope and vocabulary constraints are encoded upstream in the LookupSpec and LookupIndex.
Resolution proceeds in ordered stages:
1. Apply the primary normaliser to the input term and attempt a direct lookup.
2. If no hit is found, apply each correction function in turn, re-normalise,
and retry the lookup.
3. If no match is found, return the configured unknown concept ID.
This design allows simple, explicit handling of common data quality issues (e.g. formatting differences, legacy codes, mild normalisation errors) without introducing fuzzy matching, probabilistic scoring, or hidden inference logic.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
index
|
LookupIndex
|
Pre-built LookupIndex providing the normalised key → concept_id mapping. |
required |
normalizer
|
Normaliser | None
|
Optional normalisation function applied to input terms at lookup time.
Defaults to |
None
|
corrections
|
list[Callable[[str], str]] | None
|
Optional ordered list of correction functions applied to the raw input term prior to normalisation and lookup. Each correction is tried in sequence until a match is found. |
None
|
Notes
- ConceptResolver performs no database access and no dynamic expansion of vocabularies; it operates over the materialised LookupIndex.
- Resolution is deterministic and transparent: there is no fuzzy matching, ranking, or probabilistic inference.
- Correction functions are applied conservatively and in-order; later corrections do not override earlier successful matches.
lookup_exactbypasses correction passes and performs a single normalised lookup, which is useful for validation and debugging.
Examples:
>>> resolver = ConceptResolver(index)
>>> resolver.lookup("Stage III")
123456
>>> resolver.lookup("stage-3")
123456
all_concepts
cached
property
¶
all_concepts: set[int]
Every concept ID reachable through this resolver's index.
Cached: the index is fixed at construction, and callers legitimately union several resolvers' sets, which previously rebuilt each one per access. Returned by reference, so treat it as read-only.
estimated_bytes ¶
estimated_bytes() -> int
Approximate retained size, for cache accounting.
A name/code/synonym index costs several times more per concept than a bare ID set, which is why the cache bound is measured in bytes rather than entry counts. Measured at ~350 bytes per concept for OMOP-length names and codes with one synonym each.
ConceptResolverRegistry ¶
ConceptResolverRegistry(
engine: Engine,
*,
max_bytes: int = DEFAULT_MAX_CACHE_BYTES,
)
Bases: _LazyBoundedRegistry[ConceptResolver]
Lazy registry for ConceptResolvers.
Resolvers are constructed on first access and cached for the lifetime of this registry instance. The registry is scoped to a SQLAlchemy Engine, ensuring vocab lookups are built once per database.
LookupIndex
dataclass
¶
LookupIndex(
name: str, unknown: int | None, mapping: dict[str, int]
)
Materialised lookup table from normalised text keys to OMOP concept IDs.
A LookupIndex is the runtime artifact produced by a LookupSpec and a ConceptSource. It represents a flat, precomputed mapping from one or more normalised string representations (e.g. concept names, codes, synonyms) to OMOP concept IDs.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Human-readable identifier for the lookup. |
unknown |
int | None
|
Concept ID to return when a lookup fails, or None if failures should propagate as null. |
mapping |
dict[str, int]
|
Dictionary mapping normalised string keys to OMOP concept IDs. Keys are expected to already be normalised at build time. |
Notes
The mapping may contain multiple textual representations pointing to the same concept ID (e.g. name + code + synonym).
LookupSpec
dataclass
¶
LookupSpec(
name: str,
unknown: int | None = 0,
domain_id: str | None = None,
concept_class_id: list[str] | None = None,
vocabulary_id: list[str] | None = None,
require_standard: bool = True,
include_classification: bool = True,
require_active: bool = False,
code_filter: str | None = None,
parents: list[int] | None = None,
include_non_standard_descendants: bool = False,
include_synonyms: bool = False,
normalizer: Normaliser = normalize_default,
include: tuple[str, ...] = (
"concept_name",
"concept_code",
),
)
Declarative specification for constructing a vocabulary lookup index.
A LookupSpec defines what concepts should be included in a lookup and which textual representations should be indexed for resolution.
The spec is consumed by a OMOPConceptSource, which materialises a LookupIndex by querying an OMOP vocabulary source and extracting the requested fields.
This separation allows lookup semantics (domain, vocabulary, hierarchy, standardness, synonyms, normalisation) to be expressed explicitly and versioned independently of runtime resolution logic.
Attributes:
| Name | Type | Description |
|---|---|---|
name |
str
|
Stable identifier for this lookup specification. |
unknown |
int | None
|
Concept ID to return for unmatched terms. Set to None to preserve nulls, or to a sentinel concept ID to force closed-world behaviour. |
domain_id |
str | None
|
Optional OMOP domain filter |
concept_class_id |
list[str] | None
|
Optional list of OMOP concept_class_id values to restrict the lookup |
vocabulary_id |
list[str] | None
|
Optional list of OMOP vocabulary_id values to restrict the lookup |
require_standard |
bool
|
If True, restricts the lookup to concepts carrying a standardness flag.
Named to match |
include_classification |
bool
|
Widens |
require_active |
bool
|
If True, excludes concepts with an |
code_filter |
str | None
|
Optional substring filter applied to concept_code (ILIKE-based). Useful for coarse scoping (e.g. AJCC-only codes). |
parents |
list[int] | None
|
Optional list of ancestor concept IDs from which to expand the lookup via the Concept_Ancestor table. |
include_non_standard_descendants |
bool
|
If True, includes non-standard concepts when expanding from parents. Has no effect if parents is None. |
include_synonyms |
bool
|
If True, include Concept_Synonym entries in the lookup keys. |
normalizer |
Normaliser
|
Function applied to all indexed strings at build time. This should match (or be compatible with) the normalisation used at resolution time by ConceptResolver. |
include |
tuple[str, ...]
|
Tuple of ConceptRow attribute names to index as keys (e.g. ("concept_name", "concept_code")). This controls which textual fields become resolvable inputs. |
Notes
- LookupSpec encodes semantic intent; LookupIndex encodes runtime state.
- Specs are designed to be stable, inspectable configuration objects that can be versioned and reviewed as part of phenotype or ETL definitions.
- Normalisation and correction policies are intentionally split between build-time (this spec) and runtime (ConceptResolver) to make lookup behaviour explicit and testable.
OMOPConceptSource ¶
Concrete ConceptSource backed by OMOP CDM vocabulary tables.
It is a thin, explicit adapter between SQLAlchemy + OMOP CDM and higher-level vocabulary indexing logic.
Used exclusively to builds a query based on provided parameters (adds
filter for each non-None parameter, and joins to Concept_Ancestor
if parents are specified).
fetch_concepts
staticmethod
¶
fetch_concepts(
session: Session,
*,
domain_id: str | None = None,
concept_class_id: Iterable[str] | None = None,
vocabulary_id: Iterable[str] | None = None,
require_standard: bool = True,
include_classification: bool = True,
require_active: bool = False,
code_filter: str | None = None,
parents: Iterable[int] | None = None,
include_non_standard_descendants: bool = False,
) -> list[ConceptRow]
Fetch concepts matching the provided constraints.
This method supports two primary modes: 1. Flat filtering by domain / class / vocabulary 2. Hierarchical expansion from parent concept(s)
Domain, vocabulary, standardness and validity are delegated to
:class:~omop_alchemy.cdm.query.ConceptFilter so this layer shares one
implementation of the OMOP flag rules with the rest of the package. In
particular the flag comparisons are normalised, so a blank or
whitespace-padded standard_concept is classified the same way here as
it is by Concept.is_standard.
Recognition is permissive by default, because this layer resolves text to a concept, not a concept to a mapping target. Both defaults follow from that:
- include_classification is
True— classification concepts are legitimate things to recognise, even though they are not valid mapping targets. - require_active is
False— a deprecated concept that matches the text is a useful hit, because the caller can followMaps to/Concept replaced byto a valid successor. Filtering it out at recognition time discards the term entirely, with nothing to resolve forward from.
Pass include_classification=False / require_active=True when the
index feeds concept selection rather than recognition, where the strict
OMOP mapping-target rules apply.
fetch_synonyms
staticmethod
¶
fetch_synonyms(
session: Session,
*,
concept_ids: Iterable[int] | None = None,
) -> list[tuple[int, str]]
Return (concept_id, synonym) pairs for concept synonyms.
The join to concept scopes the result to synonyms whose concept
actually exists, and concept_ids narrows it further to a caller-supplied
set. Both are applied in SQL: without them this streams every synonym row
in the vocabulary back to Python to be discarded, which on a full Athena
load is millions of rows for a lookup covering a handful of domains.
ResolvedConceptGroup
dataclass
¶
ResolvedConceptGroup(
spec: ConceptGroupSpec, ids: frozenset[int]
)
A governed group expanded against one vocabulary.
Membership is
(descendants(parents) - descendants(excluded parents)) | exact_ids.
Exact members are explicit inclusions and are never subtracted by the
exclusion step, matching omop-semantics' composition contract.
estimated_bytes ¶
estimated_bytes() -> int
Approximate retained size, for cache accounting.
Measured at ~80 bytes per ID for a frozenset of OMOP-magnitude
concept IDs: a distinct PyLong each, plus hash-table overhead at a
0.6 load factor.
expression ¶
expression(
column: SQLColumnExpression[Any],
) -> sa.ColumnElement[bool]
SQL membership for this group — delegates to the spec.
Kept here so a caller holding a resolved group can reach either rendering without going back to the spec, but the definition lives in one place.
build_concept_group ¶
build_concept_group(
session: Session, spec: ConceptGroupSpec
) -> ResolvedConceptGroup
Expand a governed group against the vocabulary behind session.
Performs up to two queries — the include closure and, when the group
declares exclusions, the exclude closure. Callers should normally go
through the cache in registry rather than calling this directly.
clear_concept_group_cache ¶
clear_concept_group_cache() -> None
Drop every cached group expansion, across all vocabularies.
Per-vocabulary keying means this is rarely needed — moving database gives a different identity and therefore a different registry. It remains an escape hatch for a dataset reloaded in place under an unchanged identity.
clear_vocabulary_identity ¶
clear_vocabulary_identity(engine: Engine) -> None
Forget engine's registered identity, if it had one.
concept_group_cache_stats ¶
concept_group_cache_stats() -> dict[
str | int, dict[str, int]
]
Per-scope cache statistics, for monitoring the bound.
Watch rebuilds_after_evict: while it is zero the shared bound is
holding, and if it climbs the per-scope byte totals are the evidence for
raising it or splitting the budget by payload kind.
concept_group_registry ¶
concept_group_registry(
session: Session,
) -> ConceptGroupRegistry
Return the group registry for the vocabulary behind session.
Registries are keyed on vocabulary identity where one has been registered
(see :mod:.identity), so recreating an engine against the same dataset
reuses expansions. Otherwise they are keyed weakly on the engine, which is
still built-once-per-engine but is not shared across engines.
In-memory SQLite intentionally lands in the second case: two such engines are separate databases despite identical configuration, so cross-engine sharing would serve one database's concept sets for another.
make_concept_resolver ¶
make_concept_resolver(
session: Session,
*,
name: str,
unknown: int | None = 0,
domain_id: str | None = None,
concept_class_id: list[str] | None = None,
vocabulary_id: list[str] | None = None,
require_standard: bool = True,
include_classification: bool = True,
require_active: bool = False,
code_filter: str | None = None,
parents: list[int] | None = None,
include_non_standard_descendants: bool = False,
include_synonyms: bool = False,
include: tuple[str, ...] = (
"concept_name",
"concept_code",
),
build_normalizer: Normaliser = normalize_default,
runtime_normalizer: Normaliser | None = None,
corrections: list[Callable[[str], str]] | None = None,
) -> ConceptResolver
Convenience factory for constructing a ConceptResolver from declarative inputs.
This function bundles the common workflow of: - defining a LookupSpec - materialising a LookupIndex from OMOP - constructing a ConceptResolver for runtime use
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
session
|
Session
|
Active SQLAlchemy session connected to the OMOP CDM database. |
required |
name
|
str
|
Stable identifier for this lookup specification, used in logging and debugging. |
required |
unknown
|
int | None
|
Concept ID to return for unmatched terms. Set to None to preserve nulls, or to a sentinel concept ID to force closed-world behaviour. |
0
|
domain_id
|
str | None
|
Optional OMOP domain filter for the concepts to include in the lookup. |
None
|
concept_class_id
|
list[str] | None
|
Optional list of OMOP concept_class_id values to restrict the lookup. |
None
|
vocabulary_id
|
list[str] | None
|
Optional list of OMOP vocabulary_id values to restrict the lookup. |
None
|
require_standard
|
bool
|
If True, restricts the lookup to concepts carrying a standardness flag.
Named to match |
True
|
include_classification
|
bool
|
Widens |
True
|
require_active
|
bool
|
If True, excludes concepts with an |
False
|
code_filter
|
str | None
|
Optional substring filter applied to concept_code (ILIKE-based). Useful for coarse scoping (e.g. AJCC-only codes). |
None
|
parents
|
list[int] | None
|
Optional list of ancestor concept IDs from which to expand the lookup via the Concept_Ancestor table. |
None
|
include_non_standard_descendants
|
bool
|
If True, includes non-standard concepts when expanding from parents. Has no
effect if |
False
|
include_synonyms
|
bool
|
If True, include Concept_Synonym entries in the lookup keys. |
False
|
include
|
tuple[str, ...]
|
Tuple of ConceptRow attribute names to index as keys (e.g. ("concept_name", "concept_code")). This controls which textual fields become resolvable inputs. |
('concept_name', 'concept_code')
|
build_normalizer
|
Normaliser
|
Normalisation function applied to all indexed strings at build time. This should |
normalize_default
|
runtime_normalizer
|
Normaliser | None
|
Optional normalisation function applied to input terms at lookup time. Defaults to
|
None
|
corrections
|
list[Callable[[str], str]] | None
|
Optional ordered list of correction functions applied to the raw input term prior to normalisation and lookup |
None
|
register_vocabulary_identity ¶
register_vocabulary_identity(
engine: Engine, identity: str
) -> None
Declare which vocabulary dataset engine reads.
Concept-set caches keyed on this identity are shared by every engine registered under it, so recreating an engine reuses the expansion.
Pass the engine your factory returns.
ResolvedCDMDatabase.create_engine ends with
execution_options(schema_translate_map=...), which yields a derived
OptionEngine; that is the object sessions bind to, and the one lookups
will see.
Do not register an identity for an ephemeral database — notably in-memory SQLite, where two engines built from identical configuration are genuinely separate databases. Those correctly fall back to per-engine caching.
resolve_concept_group ¶
resolve_concept_group(
session: Session, spec: ConceptGroupSpec
) -> ResolvedConceptGroup
Expand spec against session's vocabulary, cached.
The cache key is spec.name within the vocabulary's registry, so it
derives from the governed semantic-unit name rather than a locally invented
label.
Patient timelines¶
Projects a person's clinical rows — conditions, measurements, drug exposures — into a single time-ordered event stream. This has its own dedicated page, since it predates the rest of the toolkit reorg: see Patient Timelines.
Unit conversion¶
Converts measurement values to canonical units. Kilograms, pounds, centimetres, and
inches mean the same thing in every clinical domain, so the conversion rules live here
rather than with any one domain's measurement logic — see
analytics.body_metrics for where those domain-specific
measurements are resolved and normalised using these rules.
Convert measurement values to canonical units.
Measurement rows carry whatever unit the source system recorded. Before values can be compared, trended, or fed into a calculation they need to agree on a unit, and that conversion must be explicit about what it will and will not accept.
BodyUnitConversionRules converts anthropometric measurements to
kilograms and centimetres, driven by the unit concept on each row rather
than by guesswork. default_body_unit_conversion_rules supplies the
standard rule set; INCH_TO_CM and LB_US_TO_KG are the underlying
factors where a caller needs them directly.
A reading whose unit concept is unrecognised is not converted and not silently passed through — callers are told the value could not be normalised, so an unconvertible unit never reaches a calculation disguised as a valid one.
BodySizeUnitConcepts
dataclass
¶
BodySizeUnitConcepts(
kg: int, lb: int, cm: int, inch: int, m2: int
)
The unit concepts anthropometric conversion recognises.
Kilograms, pounds, centimetres, inches, and square metres mean the same thing in every clinical domain, so the mapping from concept ID to unit lives here rather than with any one domain's measurement logic.
BodyUnitConversionRules
dataclass
¶
BodyUnitConversionRules(units: BodySizeUnitConcepts)
Unit normalisation for anthropometric measurements.
Conversion is driven by the unit concept recorded on each row. A value
whose unit concept is not in units converts to None rather than being
passed through, so a reading in an unrecognised unit cannot reach a
calculation disguised as a valid one.
Deployments that record body size against non-standard unit concepts can
supply their own units instead of the governed defaults.