Data lineage - on-premise architecture

Thu, 02 Jul 2026 00:00:00 +0000

In the most demanding environments in terms of security (banking, insurance, etc.), {openAudit} can be deployed directly within the Client’s environment, whether on-premise or in a Cloud tenant of the Client’s choice.

The architecture relies on a dedicated VM hosting the various components used to collect, process, and expose the data lineage.

Key point: all processing, metadata, and repositories stay hosted within the Client’s environment. No external SaaS component is required.

Technical principles

Dedicated VM hosted at the Client’s site.
Collectors containerized per technology.
Dedicated service accounts.
Password encryption after first use.
Production of standardized CSV files.
Parsing and consolidation by oA.Job.
Technical repository oA.mdb (DuckDB).
Exposure via the Mosaïc web portal.
Audit logs that may contain sensitive information can be anonymized before processing.

Good to know: this data lineage can feed a third-party solution, typically a data catalog that needs to address data lineage more specifically.

Metadata collection

Metadata is collected using containers specialized by technology. These components connect to databases, ETL/ELT platforms, data visualization tools, and audit databases via dedicated service accounts. Passwords are encrypted after first use.

The collectors produce standardized CSV files containing the metadata needed to reconstruct the lineage. These files can be reviewed and validated by the Client before processing.

Where audit logs are used, they can be anonymized through a dedicated container before being integrated into the repository.

Processing

The files are then sent to the oA.Job server, responsible for parsing and consolidating the metadata. The results are stored in the oA.mdb columnar database (DuckDB), which forms the technical repository used by the Mosaïc web portal.

This repository can be dumped in the format requested by the client to run specific analyses: the repository is extremely rich, and various pre-built queries are made available to make exploring it easier.

Exposure

The data lineage can be browsed via the proprietary Mosaïc web portal, hosted on the dedicated VM.

Overall flow mapping

Automatic reconstruction of dependencies between databases, ETL/ELT processes, and usage in data visualization tools.

Typical use case: impact analysis before modifying a table or a business indicator.

Usage analysis

Cross-referencing data lineage with audit logs to identify objects that are actually used and branches that see no consumption.

Result: quick identification of unused flows and simplification opportunities.

Detailed data lineage

Navigation through the data lineage down to field level, with visualization of the transformations (drill-through).

Key point: instantly drill into the code driving the transformations to identify and fix an issue in a flow.

Data lineage in dashboards

Complete traceability between source data, business rules, and the dashboard cell. Dashboard source data is linked into the data lineage as far back as its feeds, for end-to-end impact analysis.

Typical use case: validating a business indicator or tracing the origin of a value displayed in a dashboard.

Data lineage - SaaS architecture

Thu, 02 Jul 2026 00:00:00 +0000

In a SaaS deployment, the collection components remain hosted within the Client’s environment, while the processing, consolidation, and exposure components are hosted on dedicated infrastructure operated by Ellipsys.

The extractors are deployed as containers specialized by technology. They collect metadata from databases, ETL/ELT platforms, data visualization tools, and audit databases using dedicated service accounts.

The collected metadata is produced as standardized CSV files. These files can be reviewed by the Client before being transferred to Ellipsys’s infrastructure.

The files are then sent to the oA.Job server, responsible for parsing and reconstructing dependencies. Metadata consolidation happens in the oA.mdb technical repository, based on DuckDB (a columnar database).

The data lineage is then exposed via the proprietary Mosaïc web portal or via third-party solutions, typically off-the-shelf data catalog products.

Where audit logs are used, they can be anonymized through a dedicated container before transfer.

Key point: the metadata needed to reconstruct the lineage is transferred to the SaaS platform in CSV format and can be reviewed beforehand.

Technical principles

Containerized extractors deployed within the Client’s environment.
Service accounts dedicated to collection.
Password encryption after first use.
Production of standardized CSV files.
Optional anonymization of audit logs.
Parsing and consolidation via oA.Job.
Source database for the data lineage: oA.mdb (DuckDB).
Exposure via the Mosaïc web portal or in data catalog solutions.

Good to know: business data is never used. Only technical metadata is collected.

Metadata collection

The collectors can analyze:

Databases.
ETL/ELT platforms.
Data visualization tools.
Audit databases.

Each technology is handled by a dedicated component.

Processing

The oA.Job server performs:

Metadata parsing.
Dependency reconstruction.
Consolidation of technical objects.
Integration of usage data from audit logs.

The results are stored in the oA.mdb repository.

Exposure

The data lineage can be viewed:

Via the Mosaïc web portal.
Via exposure of {openAudit}’s data lineage into third-party solutions.
Via the Client’s own integration mechanisms.

Key point: the collection and lineage-reconstruction mechanisms are independent from the exposure mechanisms.

Overall flow mapping

Mapping of dependencies between databases, ETL/ELT processes, and data usage.

Typical use case: impact analysis before modifying a technical object.

Usage analysis

Cross-referencing data lineage with audit logs to identify flows that are actually used.

Result: detection of unused flows, i.e. dead branches.

Detailed data lineage

Navigation through the data lineage down to field level, with visualization of intermediate transformations.

Key point: dependencies can be explored iteratively, from operational sources all the way to usage.

Data lineage in dashboards

Traceability between source data, transformations, calculation rules, and reporting objects.

Typical use case: tracing the origin of a business indicator or validating a business rule.

Usage |

Data lineage - on-premise architecture

Technical principles

Metadata collection

Processing

Exposure

Overall flow mapping

Usage analysis

Detailed data lineage

Data lineage in dashboards

Data lineage - SaaS architecture

Technical principles

Metadata collection

Processing

Exposure

Overall flow mapping

Usage analysis

Detailed data lineage

Data lineage in dashboards