Data lineage - the principles of connector containerization

The {openAudit} architecture relies on containerized components deployed within the Client’s environment.

Each component is specialized for a given technology and handles the collection of the metadata needed to reconstruct the data lineage: databases, ETL/ELT platforms, data visualization tools, and audit databases.

The extractors produce standardized CSV files containing the collected metadata. These files can be reviewed by the Client before being transferred to the central processing components.

Where anonymization is required, a dedicated container is inserted into the collection chain to replace strings, SQL constants, and other sensitive information with variables before the exchange files are produced.

Key point: extractors and anonymizers are decoupled from the processing engines. Each component can be deployed and updated independently.

Technical principles

• One dedicated container per technology.
• Metadata collected as close as possible to the analyzed systems.
• Optional anonymization before transfer.
• Production of standardized CSV files.
• Docker and Podman compatibility.
• On-premise, SaaS, or hybrid deployment.
• Independent updates for each component.

Extraction

The extractors connect to the analyzed technologies using dedicated service accounts.

The metadata collected relates to:

• Databases.
• ETL / ELT platforms.
• Data visualization tools.
• Audit databases.
• The scheduler.

Each extractor produces a consistent output format regardless of the technology being analyzed.

Anonymization

An anonymization container can be positioned between metadata collection and transfer.

The following elements can be replaced with technical variables:

• Strings.
• Application parameters.
• Literal values.

The goal is to preserve technical dependencies while removing sensitive information.

Key point: anonymization happens before the transferred files are generated.

Exchange files

The components produce standardized CSV files containing:

• Technical objects.
• Dependencies.
• Transformations.
• Usage information, where available.

These files can be archived, reviewed, or validated by the Client before being transferred.

Processing

The generated files are used by {openAudit}’s central components to:

• Reconstruct dependencies.
• Consolidate metadata.
• Integrate usage from audit logs.
• And produce the data lineage.

Result: a consistent collection chain, independent of the analyzed technologies and deployment modes.