Data lineage and simplification | Ellipsys

Data lineage - on-premise architecture

Thu, 02 Jul 2026 00:00:00 +0000

In the most security-sensitive environments (banking, insurance…), {openAudit} can be deployed directly at the Client’s site, on-premise or in a Cloud tenant of their choice, on a dedicated VM hosting data lineage collection, processing, and exposure.

Key point: processing, metadata, and repositories stay within the Client’s environment; no external SaaS component is required.

Metadata collection

Containers specialized by technology connect to databases, ETL/ELT platforms, data visualization tools, and audit databases through dedicated service accounts; passwords are encrypted after first use. They produce standardized CSV files that the Client can check before processing.

Audit logs, which may contain sensitive information, can be anonymized by a dedicated container before being integrated.

Processing

The oA.Job server parses and consolidates the metadata, which is stored in the oA.mdb columnar database (MonetDB). This very rich repository can be exported (dump) in the format the Client needs for specific analyses; queries are provided to explore it.

Exposure

Data lineage is available through the Mosaïc web portal, hosted on the VM. It can also feed a third-party solution, typically a data catalog.

Overall flow mapping

Automatic reconstruction of dependencies between databases, ETL/ELT processes, and usage in data visualization tools.

Typical use case: impact analysis before modifying a table or a business indicator.

Usage analysis

Cross-referencing data lineage with audit logs separates the objects actually used from unconsumed branches, and reveals simplification opportunities.

Detailed data lineage

Navigation down to field level, with direct access to the transformation code (drill through) to identify and fix an issue in a flow.

Data lineage in dashboards

End-to-end traceability, from data feeds through business rules to the dashboard cell.

Typical use case: validating a business indicator or tracing the origin of a displayed value.

Frequently asked questions

Can {openAudit} be deployed entirely in our own environment, without SaaS?

Yes. {openAudit} can be deployed on-premise or in a cloud tenant chosen by the client, on a dedicated VM. Processing, metadata and repositories stay hosted at the client, and no external SaaS component is required.

Which components are installed for on-premise data lineage?

Containerized collectors for each technology, the oA.Job server for parsing and consolidation, the oA.mdb technical repository (MonetDB) and the Mosaïc web portal for browsing, all hosted on the dedicated VM.

How is sensitive information in audit logs protected?

Audit logs can be anonymized by a dedicated container before they are loaded. Collectors produce standardized CSV files that the client can review and validate before processing, and service account passwords are encrypted after first use.

How detailed is the data lineage?

Down to field level, with the transformations visible, and all the way to the dashboard cell. Cross-referenced with audit logs, it also identifies the objects actually used and the branches never consumed.

Data lineage - SaaS architecture

Thu, 02 Jul 2026 00:00:00 +0000

In a SaaS deployment, collection stays within the Client’s environment; processing, consolidation, and exposure are hosted on a dedicated infrastructure operated by Ellipsys.

Key point: only technical metadata is transferred, as CSV files that can be checked beforehand; business data is never used.

Metadata collection

Containerized extractors, one per technology, collect metadata from databases, ETL/ELT platforms, data visualization tools, and audit databases through dedicated service accounts; passwords are encrypted after first use. They produce standardized CSV files that the Client can check before transfer.

Audit logs can be anonymized by a dedicated container before being transferred.

Processing

The oA.Job server parses the metadata, reconstructs dependencies, consolidates technical objects, and integrates usage from audit logs. Results are stored in the oA.mdb repository (MonetDB, columnar database).

Exposure

Data lineage is available through the Mosaïc web portal, in third-party solutions (typically market data catalogs), or through the Client’s own integration mechanisms: collection and reconstruction are independent of exposure.

Overall flow mapping

Mapping of dependencies between databases, ETL/ELT processes, and data usage.

Typical use case: impact analysis before modifying a technical object.

Usage analysis

Cross-referencing data lineage with audit logs identifies the flows actually used, and therefore the dead branches.

Detailed data lineage

Navigation down to field level, transformation by transformation, from operational sources to usage.

Data lineage in dashboards

Traceability between source data, transformations, calculation rules, and reporting objects.

Typical use case: tracing the origin of a business indicator or validating a business rule.

Frequently asked questions

How does {openAudit} work in SaaS mode?

Collectors stay in the client’s environment, as containers specialized by technology. Processing, consolidation and exposure of the data lineage are hosted on dedicated infrastructure operated by Ellipsys.

What information is transferred to the SaaS platform?

Only technical metadata, as standardized CSV files that the client can review before transfer. Business data is never used, and audit logs can be anonymized before transfer.

Where can the data lineage be viewed in SaaS mode?

In the Mosaïc web portal, in third-party solutions such as market data catalogs, or through the client’s own integration mechanisms.

How are unused flows detected?

By cross-referencing the data lineage with audit logs: the flows actually used are identified, and unused flows (dead branches) stand out.

Data lineage - feeding third-party data catalogs to expose technical data lineage

Thu, 02 Jul 2026 00:00:00 +0000

{openAudit} exposes data lineage through its Mosaïc web portal, but the consolidated metadata can also be published to market data catalogs, which are often limited when it comes to data lineage. Governance teams can then work in a single tool.

Key point: collection, connection, and data lineage reconstruction stay in the {openAudit} technical repository, independent of the target catalog.

Architecture

After processing by the oA.Job engine, metadata is consolidated in the oA.mdb columnar database: field-level data lineage, usage, and governance information where applicable. Catalogs are fed from this single repository through dedicated exposure connectors; they only consume this consolidated metadata, never the original technical sources.

Exposure connectors

The connectors handle automated data lineage publishing, continuous synchronization, restriction management, and support for multiple Workspaces. Exposure can be full or partial depending on the catalog’s display capabilities, while the data lineage produced upstream remains available at field level.

Catalogs supported to date: DataGalaxy, Zeenea, Collibra.

Usage management

When audit logs are available, observed usage is integrated into the repository and then exposed in the catalog. Users can thus tell “live” flows, consumed by tools or queried, from unconsumed flows and objects (often more than 80% of the total).

Typical use case: identify in the data catalog the objects that are candidates for rationalization or decommissioning.

Frequently asked questions

Which data catalogs can {openAudit} feed with data lineage?

The available connectors currently feed DataGalaxy, Zeenea and Collibra.

Why feed a data catalog with {openAudit} data lineage?

Data catalogs often have significant limitations when it comes to data lineage. {openAudit} rebuilds it in its technical repository, then publishes it to the catalog through dedicated connectors: automated publication, continuous synchronization, restriction management and support for multiple Workspaces. Governance teams work in a single tool.

Is the data lineage exposed at field level in the catalog?

The data lineage is produced at field level in the {openAudit} repository. Its exposure in the catalog can be full or partial, depending on the display capabilities of the target catalog.

Can the catalog show which data is actually used?

Yes, when audit logs are available. Usage is loaded into the repository and then exposed to the catalog, which distinguishes consumed flows from unconsumed objects, often more than 80% of the total.

Data lineage - the principles of connector containerization

Thu, 02 Jul 2026 00:00:00 +0000

{openAudit} collection relies on containerized components deployed in the Client’s environment, as close as possible to the systems analyzed: one container per technology (databases, ETL/ELT platforms, data visualization tools, audit databases), compatible with Docker and Podman, for on-premise, SaaS, or hybrid deployment.

ETL / ELT

SAP Data ServicesIBM DataStageTalendSSISAb InitioStambiadbtOracle Data IntegratorInformatica PowerCenter

Databases

OracleTeradataIBM DB2SQL ServerGoogle BigQueryPostgreSQL

Languages and scripts

SQLJavaPerlCOBOL

Data visualization and reporting

SAP BusinessObjectsIBM CognosPower BITableauQlik SenseSSRSCrystal ReportsDigDashStrategySASLookerSpotfire

Key point: extractors and anonymizers are decoupled from the processing engines; each component is deployed and updated independently.

Extraction

Extractors connect to the technologies analyzed through dedicated service accounts and collect metadata from databases, ETL/ELT platforms, data visualization tools, audit databases, and the scheduler. Each extractor produces a homogeneous output format, whatever the technology.

Anonymization

When required, an anonymization container sits between collection and the production of exchange files: character strings, application parameters, and literal values (SQL constants…) are replaced with technical variables. Dependencies are preserved, sensitive information removed.

Exchange files

The components produce standardized CSV files: technical objects, dependencies, transformations and, when available, usage information. The Client can archive, check, or validate them before transfer.

Processing

The central {openAudit} components use these files to reconstruct dependencies, consolidate metadata, integrate usage from audit logs, and produce the data lineage.

Result: a homogeneous collection chain, independent of the technologies analyzed and of deployment modes.

Frequently asked questions

How does {openAudit} collect metadata?

With containerized components deployed in the client’s environment, one per technology. They connect to databases, ETL/ELT platforms, data visualization tools, audit databases and the scheduler using dedicated service accounts, and produce standardized CSV files.

Which technologies do the collectors cover?

ETL/ELT: SAP Data Services, IBM DataStage, Talend, SSIS, Ab Initio, Stambia, dbt, Oracle Data Integrator and Informatica PowerCenter. Databases: Oracle, Teradata, IBM DB2, SQL Server, Google BigQuery and PostgreSQL. Dataviz: SAP BusinessObjects, IBM Cognos, Power BI, Tableau, Qlik Sense, SSRS, Crystal Reports, DigDash, Strategy, SAS, Looker and Spotfire. Plus the SQL, Java, Perl and COBOL languages.

How does metadata anonymization work?

A dedicated container, placed between collection and transfer, replaces character strings, application parameters and literal values with technical variables before the exchange files are produced. Technical dependencies are preserved.

In which environments can the collectors be deployed?

On-premise, SaaS or hybrid, with Docker or Podman. Each component can be deployed and updated independently.

Data lineage - connection protocols to the various analytics technologies

Thu, 02 Jul 2026 00:00:00 +0000

{openAudit} acquires metadata through the native interfaces officially exposed by vendors: database protocols (JDBC, ODBC), REST APIs, SDKs, technical repositories, or standardized exports. Collection uses dedicated service accounts, limited to metadata, with passwords encrypted after first use.

Available connectors

ETL / ELT

SAP Data ServicesIBM DataStageTalendSSISAb InitioStambiadbtOracle Data IntegratorInformatica PowerCenter

Databases

OracleTeradataIBM DB2SQL ServerGoogle BigQueryPostgreSQL

Languages and scripts

SQLJavaPerlCOBOL

Data visualization and reporting

SAP BusinessObjectsIBM CognosPower BITableauQlik SenseSSRSCrystal ReportsDigDashStrategySASLookerSpotfire

Databases

Through JDBC, ODBC, or proprietary connectors: schemas, tables, views, stored procedures and, when analyzed, usage statistics.

ETL / ELT

Through technical repositories, administration APIs, or XML / JSON exports: jobs, transformations, dependencies, and data flows.

Data Visualization Tools

Through SDKs, REST APIs, or internal repositories: data sources, semantic layers, dashboards, and business rules. Source data lineage is thus extended down to the dashboard cell.

Secure environments

Collection can be supervised through VMware Horizon, OpenOTP (two-factor authentication), or an administration bastion, with logging and traceability of operations. The mechanism is the same in Cloud, on-premise, or hybrid environments.

Frequently asked questions

Which protocols does {openAudit} use to collect metadata?

The native interfaces exposed by vendors: JDBC, ODBC and proprietary connectors for databases; technical repositories, administration APIs and XML or JSON exports for ETL/ELT tools; SDKs, REST APIs and internal repositories for data visualization tools.

What access needs to be granted for a data lineage project?

Dedicated service accounts with access limited to metadata. Passwords are encrypted after first use.

What is collected from data visualization tools?

Data sources, semantic layers, dashboards and business rules. This extends the data lineage to the reporting layer, down to the dashboard cell.

Can collection run in a highly secured environment?

Yes. Collection can be supervised through VMware Horizon, OpenOTP (two-factor authentication) and an administration bastion, with logging and traceability of operations. The mechanism is the same in cloud, on-premise or hybrid environments.