Data lineage - connection protocols to the various analytics technologies

Thu, 02 Jul 2026 00:00:00 +0000

Metadata acquisition relies on the native interfaces exposed by the technologies analyzed by {openAudit}.

Depending on the platform, collection uses database protocols, REST APIs, SDKs, technical repositories, or standardized exports.

Key point: the collectors use the mechanisms officially exposed by the vendors (JDBC, ODBC, REST API, SDK, technical repositories, exports).

Technical principles

Dedicated service accounts.
Access limited to metadata.
On-premise, Cloud, and hybrid compatibility.
Use of platforms’ native interfaces.
Password encryption after first use.

Databases

Relational databases are analyzed via:

JDBC.
ODBC.
Proprietary connectors.

Collection covers:

Schemas.
Tables.
Views.
Stored procedures.
Usage statistics (when analyzed).

ETL / ELT

ETL/ELT platforms are analyzed via:

Technical repositories.
Administration APIs.
XML or JSON exports.

The collected metadata makes it possible to reconstruct:

Jobs.
Transformations.
Dependencies.
Data flows.

Data Visualization Tools

Data visualization tools are analyzed via:

SDKs.
REST APIs.
Internal repositories.

Collection covers:

Data sources.
Semantic layers.
Dashboards.
Business rules.

This information makes it possible to extend the data lineage all the way to the reporting layer.

Typical use case: build data lineage in the sources and connect it with the data lineage in the data visualization layer, all the way to the dashboard cell.

Secure environments

Collection can be securely supervised via:

VMware Horizon.
OpenOTP (two-factor authentication).
Administration bastion.
Logging and traceability of operations.

Access remains limited to the components needed for metadata collection.

For information: the same collection mechanism applies in Cloud, on-premise, or hybrid environments.

Data lineage - the principles of connector containerization

Thu, 02 Jul 2026 00:00:00 +0000

The {openAudit} architecture relies on containerized components deployed within the Client’s environment.

Each component is specialized for a given technology and handles the collection of the metadata needed to reconstruct the data lineage: databases, ETL/ELT platforms, data visualization tools, and audit databases.

The extractors produce standardized CSV files containing the collected metadata. These files can be reviewed by the Client before being transferred to the central processing components.

Where anonymization is required, a dedicated container is inserted into the collection chain to replace strings, SQL constants, and other sensitive information with variables before the exchange files are produced.

Key point: extractors and anonymizers are decoupled from the processing engines. Each component can be deployed and updated independently.

Technical principles

One dedicated container per technology.
Metadata collected as close as possible to the analyzed systems.
Optional anonymization before transfer.
Production of standardized CSV files.
Docker and Podman compatibility.
On-premise, SaaS, or hybrid deployment.
Independent updates for each component.

Extraction

The extractors connect to the analyzed technologies using dedicated service accounts.

The metadata collected relates to:

Databases.
ETL / ELT platforms.
Data visualization tools.
Audit databases.
The scheduler.

Each extractor produces a consistent output format regardless of the technology being analyzed.

Anonymization

An anonymization container can be positioned between metadata collection and transfer.

The following elements can be replaced with technical variables:

Strings.
Application parameters.
Literal values.

The goal is to preserve technical dependencies while removing sensitive information.

Key point: anonymization happens before the transferred files are generated.

Exchange files

The components produce standardized CSV files containing:

Technical objects.
Dependencies.
Transformations.
Usage information, where available.

These files can be archived, reviewed, or validated by the Client before being transferred.

Processing

The generated files are used by {openAudit}’s central components to:

Reconstruct dependencies.
Consolidate metadata.
Integrate usage from audit logs.
And produce the data lineage.

Result: a consistent collection chain, independent of the analyzed technologies and deployment modes.

Security |

Data lineage - connection protocols to the various analytics technologies

Technical principles

Databases

ETL / ELT

Data Visualization Tools

Secure environments

Data lineage - the principles of connector containerization

Technical principles

Extraction

Anonymization

Exchange files

Processing