Data lineage - connection protocols to the various analytics technologies

Metadata acquisition relies on the native interfaces exposed by the technologies analyzed by {openAudit}.

Depending on the platform, collection uses database protocols, REST APIs, SDKs, technical repositories, or standardized exports.

Key point: the collectors use the mechanisms officially exposed by the vendors (JDBC, ODBC, REST API, SDK, technical repositories, exports).

Technical principles

• Dedicated service accounts.
• Access limited to metadata.
• On-premise, Cloud, and hybrid compatibility.
• Use of platforms’ native interfaces.
• Password encryption after first use.

Databases

Relational databases are analyzed via:

• JDBC.
• ODBC.
• Proprietary connectors.

Collection covers:

• Schemas.
• Tables.
• Views.
• Stored procedures.
• Usage statistics (when analyzed).

ETL / ELT

ETL/ELT platforms are analyzed via:

• Technical repositories.
• Administration APIs.
• XML or JSON exports.

The collected metadata makes it possible to reconstruct:

• Jobs.
• Transformations.
• Dependencies.
• Data flows.

Data Visualization Tools

Data visualization tools are analyzed via:

• SDKs.
• REST APIs.
• Internal repositories.

Collection covers:

• Data sources.
• Semantic layers.
• Dashboards.
• Business rules.

This information makes it possible to extend the data lineage all the way to the reporting layer.

Typical use case: build data lineage in the sources and connect it with the data lineage in the data visualization layer, all the way to the dashboard cell.

Secure environments

Collection can be securely supervised via:

• VMware Horizon.
• OpenOTP (two-factor authentication).
• Administration bastion.
• Logging and traceability of operations.

Access remains limited to the components needed for metadata collection.

For information: the same collection mechanism applies in Cloud, on-premise, or hybrid environments.