Data lineage - SaaS architecture

In a SaaS deployment, the collection components remain hosted within the Client’s environment, while the processing, consolidation, and exposure components are hosted on dedicated infrastructure operated by Ellipsys.

The extractors are deployed as containers specialized by technology. They collect metadata from databases, ETL/ELT platforms, data visualization tools, and audit databases using dedicated service accounts.

The collected metadata is produced as standardized CSV files. These files can be reviewed by the Client before being transferred to Ellipsys’s infrastructure.

The files are then sent to the oA.Job server, responsible for parsing and reconstructing dependencies. Metadata consolidation happens in the oA.mdb technical repository, based on DuckDB (a columnar database).

The data lineage is then exposed via the proprietary Mosaïc web portal or via third-party solutions, typically off-the-shelf data catalog products.

Where audit logs are used, they can be anonymized through a dedicated container before transfer.

Key point: the metadata needed to reconstruct the lineage is transferred to the SaaS platform in CSV format and can be reviewed beforehand.

Technical principles

• Containerized extractors deployed within the Client’s environment.
• Service accounts dedicated to collection.
• Password encryption after first use.
• Production of standardized CSV files.
• Optional anonymization of audit logs.
• Parsing and consolidation via oA.Job.
• Source database for the data lineage: oA.mdb (DuckDB).
• Exposure via the Mosaïc web portal or in data catalog solutions.

Good to know: business data is never used. Only technical metadata is collected.

Metadata collection

The collectors can analyze:

• Databases.
• ETL/ELT platforms.
• Data visualization tools.
• Audit databases.

Each technology is handled by a dedicated component.

Processing

The oA.Job server performs:

• Metadata parsing.
• Dependency reconstruction.
• Consolidation of technical objects.
• Integration of usage data from audit logs.

The results are stored in the oA.mdb repository.

Exposure

The data lineage can be viewed:

• Via the Mosaïc web portal.
• Via exposure of {openAudit}’s data lineage into third-party solutions.
• Via the Client’s own integration mechanisms.

Key point: the collection and lineage-reconstruction mechanisms are independent from the exposure mechanisms.

Overall flow mapping

Mapping of dependencies between databases, ETL/ELT processes, and data usage.

Typical use case: impact analysis before modifying a technical object.

Usage analysis

Cross-referencing data lineage with audit logs to identify flows that are actually used.

Result: detection of unused flows, i.e. dead branches.

Detailed data lineage

Navigation through the data lineage down to field level, with visualization of intermediate transformations.

Key point: dependencies can be explored iteratively, from operational sources all the way to usage.

Data lineage in dashboards

Traceability between source data, transformations, calculation rules, and reporting objects.

Typical use case: tracing the origin of a business indicator or validating a business rule.