Data lineage - on-premise architecture

In the most demanding environments in terms of security (banking, insurance, etc.), {openAudit} can be deployed directly within the Client’s environment, whether on-premise or in a Cloud tenant of the Client’s choice.

The architecture relies on a dedicated VM hosting the various components used to collect, process, and expose the data lineage.

Key point: all processing, metadata, and repositories stay hosted within the Client’s environment. No external SaaS component is required.

Technical principles

Dedicated VM hosted at the Client’s site.
Collectors containerized per technology.
Dedicated service accounts.
Password encryption after first use.
Production of standardized CSV files.
Parsing and consolidation by oA.Job.
Technical repository oA.mdb (DuckDB).
Exposure via the Mosaïc web portal.
Audit logs that may contain sensitive information can be anonymized before processing.

Good to know: this data lineage can feed a third-party solution, typically a data catalog that needs to address data lineage more specifically.

Metadata collection

Metadata is collected using containers specialized by technology. These components connect to databases, ETL/ELT platforms, data visualization tools, and audit databases via dedicated service accounts. Passwords are encrypted after first use.

The collectors produce standardized CSV files containing the metadata needed to reconstruct the lineage. These files can be reviewed and validated by the Client before processing.

Where audit logs are used, they can be anonymized through a dedicated container before being integrated into the repository.

Processing

The files are then sent to the oA.Job server, responsible for parsing and consolidating the metadata. The results are stored in the oA.mdb columnar database (DuckDB), which forms the technical repository used by the Mosaïc web portal.

This repository can be dumped in the format requested by the client to run specific analyses: the repository is extremely rich, and various pre-built queries are made available to make exploring it easier.

Exposure

The data lineage can be browsed via the proprietary Mosaïc web portal, hosted on the dedicated VM.

Overall flow mapping

Automatic reconstruction of dependencies between databases, ETL/ELT processes, and usage in data visualization tools.

Typical use case: impact analysis before modifying a table or a business indicator.

Usage analysis

Cross-referencing data lineage with audit logs to identify objects that are actually used and branches that see no consumption.

Result: quick identification of unused flows and simplification opportunities.

Detailed data lineage

Navigation through the data lineage down to field level, with visualization of the transformations (drill-through).

Key point: instantly drill into the code driving the transformations to identify and fix an issue in a flow.

Data lineage in dashboards

Complete traceability between source data, business rules, and the dashboard cell. Dashboard source data is linked into the data lineage as far back as its feeds, for end-to-end impact analysis.