Data lineage — technical principles |

Data lineage - connection protocols to the various analytics technologies

Thu, 02 Jul 2026 00:00:00 +0000

Metadata acquisition relies on the native interfaces exposed by the technologies analyzed by {openAudit}.

Depending on the platform, collection uses database protocols, REST APIs, SDKs, technical repositories, or standardized exports.

Key point: the collectors use the mechanisms officially exposed by the vendors (JDBC, ODBC, REST API, SDK, technical repositories, exports).

Technical principles

Dedicated service accounts.
Access limited to metadata.
On-premise, Cloud, and hybrid compatibility.
Use of platforms’ native interfaces.
Password encryption after first use.

Databases

Relational databases are analyzed via:

JDBC.
ODBC.
Proprietary connectors.

Collection covers:

Schemas.
Tables.
Views.
Stored procedures.
Usage statistics (when analyzed).

ETL / ELT

ETL/ELT platforms are analyzed via:

Technical repositories.
Administration APIs.
XML or JSON exports.

The collected metadata makes it possible to reconstruct:

Jobs.
Transformations.
Dependencies.
Data flows.

Data Visualization Tools

Data visualization tools are analyzed via:

SDKs.
REST APIs.
Internal repositories.

Collection covers:

Data sources.
Semantic layers.
Dashboards.
Business rules.

This information makes it possible to extend the data lineage all the way to the reporting layer.

Typical use case: build data lineage in the sources and connect it with the data lineage in the data visualization layer, all the way to the dashboard cell.

Secure environments

Collection can be securely supervised via:

VMware Horizon.
OpenOTP (two-factor authentication).
Administration bastion.
Logging and traceability of operations.

Access remains limited to the components needed for metadata collection.

For information: the same collection mechanism applies in Cloud, on-premise, or hybrid environments.

Data lineage - feeding third-party data catalogs to expose technical data lineage

Thu, 02 Jul 2026 00:00:00 +0000

{openAudit} lets you browse data lineage through its Mosaïc web portal, but the consolidated metadata can also be published to various data catalogs on the market, which often have significant limitations when it comes to data lineage. The benefit is having a single tool for optimal data governance.

The data lineage is reconstructed in the {openAudit} technical repository, then exposed to data catalogs via dedicated connectors.

This architecture makes it possible to decouple the data lineage collection and reconstruction mechanisms from the exposure layer used by governance teams.

Key point: the technical repository stays independent from the target catalog. The collection and lineage-reconstruction (and connection) mechanisms don’t depend on the data catalog in use.

Architecture

The collected metadata is consolidated in the oA.mdb columnar database after processing by the oA.Job engine.

The central repository notably contains:

Field-level data lineage.
Usage data.
Associated governance information, where applicable.

The catalogs are then fed from this single repository via exposure connectors independent of the data catalog solutions in use.

Exposure connectors

The connectors handle:

Automated publication of the lineage.
Continuous synchronization.
Restriction management.
Support for the various Workspaces.

Data lineage exposure can be complete or partial depending on the display capabilities of the target catalog, keeping in mind that the data lineage produced upstream is itself available at field level.

Result: catalogs only consume the consolidated metadata, not the original technical sources.

Supported catalogs

The available connectors currently support the following data catalogs:

DataGalaxy
Zeenea
Collibra

Usage management

When audit logs are available, observed usage is integrated into the technical repository and then exposed to the catalog.

This lets users distinguish:

“Live” flows, consumed by tools or queried.
Flows — and therefore objects — that see no consumption (often more than 80% of the total).

Typical use case: enrich the data catalog with usage information to identify objects that are candidates for rationalization or decommissioning.

Data lineage - the principles of connector containerization

Thu, 02 Jul 2026 00:00:00 +0000

The {openAudit} architecture relies on containerized components deployed within the Client’s environment.

Each component is specialized for a given technology and handles the collection of the metadata needed to reconstruct the data lineage: databases, ETL/ELT platforms, data visualization tools, and audit databases.

The extractors produce standardized CSV files containing the collected metadata. These files can be reviewed by the Client before being transferred to the central processing components.

Where anonymization is required, a dedicated container is inserted into the collection chain to replace strings, SQL constants, and other sensitive information with variables before the exchange files are produced.

Key point: extractors and anonymizers are decoupled from the processing engines. Each component can be deployed and updated independently.

Technical principles

One dedicated container per technology.
Metadata collected as close as possible to the analyzed systems.
Optional anonymization before transfer.
Production of standardized CSV files.
Docker and Podman compatibility.
On-premise, SaaS, or hybrid deployment.
Independent updates for each component.

Extraction

The extractors connect to the analyzed technologies using dedicated service accounts.

The metadata collected relates to:

Databases.
ETL / ELT platforms.
Data visualization tools.
Audit databases.
The scheduler.

Each extractor produces a consistent output format regardless of the technology being analyzed.

Anonymization

An anonymization container can be positioned between metadata collection and transfer.

The following elements can be replaced with technical variables:

Strings.
Application parameters.
Literal values.

The goal is to preserve technical dependencies while removing sensitive information.

Key point: anonymization happens before the transferred files are generated.

Exchange files

The components produce standardized CSV files containing:

Technical objects.
Dependencies.
Transformations.
Usage information, where available.

These files can be archived, reviewed, or validated by the Client before being transferred.

Processing

The generated files are used by {openAudit}’s central components to:

Reconstruct dependencies.
Consolidate metadata.
Integrate usage from audit logs.
And produce the data lineage.

Result: a consistent collection chain, independent of the analyzed technologies and deployment modes.