Middlewares — {oa.tbx} and {oa-lake} |

ETL Migration - from legacy ETL to an open SQL ETL - {oa.tbx}

Thu, 02 Jul 2026 00:00:00 +0000

Once ETL processes have been converted to SQL by {openAudit}, they can be run inside the {oa.tbx} container.

{oa.tbx} provides an execution environment for SQL, Python, or Java processes, driven by configuration and orchestration repositories.

Key point: the migrated processes become SQL instructions that can be executed independently of the original ETL platform.

Principles

SQL is encapsulated in open containers, in ETL mode.

Containers: transformations run inside an external container, {oa.tbx}, detached from the target database. Temporary containers act as intermediate storage areas where data is loaded before transformation. Jobs are tested to verify correct behavior, and possibly the data itself, to guarantee its integrity.
Orchestration with Airflow: Airflow drives the execution of the containers, step by step. Each transformation, within a container, is defined as a job.

The scheduling of an ETL migrated to Airflow automatically picks up all of the original instructions and scheduling logic.

Execution chain

ETL jobs are converted to SQL by {openAudit}.

The generated processes are integrated into a configuration repository and a control repository used by {oa.tbx}.

The workflows are then orchestrated by tools such as Airflow or Dagster.

Execution container

{oa.tbx} runs the processes described in the workflows.

Dataflows can combine:

SQL;
Python;
Java;
others.

The engine can also rely on embedded databases such as DuckDB for certain intermediate processing steps.

Sources and targets

Input data can come from any technology accessible via SQL or JDBC.

Results can be loaded into any target system compatible with standard write mechanisms.

The architecture remains independent of source and target technologies.

Orchestration

Orchestration remains separate from the processes themselves.

Dependencies, execution parameters, schedules, and triggers are handled by the workflow layer.

Business logic stays contained within the SQL scripts and dataflows executed by {oa.tbx}.

Benefits

{oa.tbx} runs open SQL pipelines, orchestrated by Airflow and independent of proprietary Cloud platform runtimes.

Across all migrated jobs — with volumes ranging from a few thousand to more than 50 million rows — execution times are often just a few seconds, with an almost systematic OK status.

Governance and data lineage

Once ETL processes have been converted to SQL, {oa.tbx} keeps the complete trace between the original logic and the executed SQL. This data lineage is exposed at two levels of detail, suited to different needs.

Key point: a synthetic view for governance, an exhaustive detail view for maintenance.

High-level pipeline view: only source and target tables are visible; intermediate ETL stages are grouped into a single SQL block. A readable view, even for complex pipelines.

Detail on click for a transformation: clicking a stage (a Transformer, for example) displays its generated SQL equivalent, along with its inputs/outputs and its original metadata (type, stage ID). The mapping between the original ETL logic and the executed SQL is immediate.

Result: complete governance of the migrated pipeline, from the overview down to the executed SQL, regardless of the original ETL platform.

SQL as the pivot format

The SQL produced by {openAudit} becomes the primary vehicle for the transformations.

Business rules, joins, aggregations, and calculations are expressed in an open, version-controllable format, independent of legacy ETL tools.

Result: a SQL-based ETL/ELT chain, orchestrated by workflows and run in an open environment.

DataViz Migration - an upstream/downstream SQL interoperability layer - {oa-lake}

Fri, 03 Jul 2026 00:00:00 +0000

Decision-support platforms often rely on several databases, several data warehouses / data lakes, and several reporting tools.

{oa-lake} introduces an intermediate SQL layer between data sources and dataviz tools.

Business rules, business objects, dimensions, measures, and calculations are centralized in this layer and then exposed to consuming tools.

Key point: data sources and reporting tools become independent of one another.

Architecture

{oa-lake} connects simultaneously to multiple data systems: relational databases, data warehouses, lakehouses, files, or APIs.

Queries are federated by a distributed SQL engine that performs the necessary joins, calculations, and transformations.

The resulting SQL remains standard and executable across different engines, which limits technology lock-in.

Business objects are defined once and then exposed to consuming tools.

Data sources

The data stays in its original systems.

Oracle, SQL Server, PostgreSQL, DB2, BigQuery, Snowflake, Redshift, Azure SQL (and others), or files can be queried within a single request.

The SQL Push-Down mechanism avoids any systematic copying of data: queries run directly against the existing systems, respecting the architectures already in place.

Typical use case: build a dashboard from data spread across several platforms with no prior migration.

SQL business layer

Dimensions, measures, hierarchies, and calculation rules are centralized in {oa-lake}.

This layer becomes the single entry point for reporting tools.

Business rules are no longer duplicated across every dataviz platform.

Key point: a single definition of indicators for the entire decision-support system.

Parquet cache

Data can be materialized in Parquet format.

This caching layer reduces response times and limits load on source systems. It’s managed automatically based on observed usage, with no manual intervention.

Typical use case: speed up the most expensive analytical processing.

Data visualization

Reporting tools connect to the model exposed by {oa-lake}.

Power BI, Looker, Spotfire, Tableau, Qlik, SAP BusinessObjects, or any SQL-compatible tool can use the same business objects.

Switching dataviz tools no longer requires rebuilding dimensions, measures, and calculation rules.

Progressive migration

Several tools can coexist during a transition phase.

The same business model can be exposed simultaneously to several reporting platforms.

Result: progressive migration of usage with no change to the data systems.

Data Lineage

The data lineage reconstructed by {openAudit} remains available across the whole chain.

Dependencies are documented from the source systems all the way to the indicators exposed in the dashboards.

Key point: transformations, calculations, and usage remain traceable across the entire architecture.

DataViz Migration - an interoperable semantic layer - {oa-lake-ctx}

Thu, 02 Jul 2026 00:00:00 +0000

Legacy decision-support platforms typically relied on a semantic layer that let users work with business objects rather than tables or SQL queries.

“Customers”, “contracts”, “products”, or “revenue” were exposed in a functional form, independent of the underlying physical storage structures.

To address this need, we designed {oa-lake-ctx}.

{oa-lake-ctx} is the semantic layer of the {oa-lake} architecture. It centralizes business objects, dimensions, measures, KPIs, calculation rules, and security mechanisms.

Key point: consuming tools work with business objects. The physical structures stay hidden.

Architecture

{oa-lake-ctx} centralizes dimensions, measures, hierarchies, KPIs, and calculation rules.

These components are defined once and then exposed to consuming tools.

Unlike some semantic layers on the market, {oa-lake-ctx} requires no specific physical model, no pivot table, and no imposed navigation path.

Relationships are calculated dynamically from the existing model.

Key point: a single business model for all data and all consuming tools.

Data sources

{oa-lake-ctx} can use several data systems simultaneously.

Data can come from Oracle, SQL Server, PostgreSQL, DB2, Snowflake, BigQuery, Redshift databases, files, or other SQL-compatible sources.

Business objects can combine several sources within a single functional definition.

Typical use case: build a KPI from data spread across several platforms.

Business objects

The catalogs exposed by {oa-lake-ctx} group together dimensions, measures, KPIs, hierarchies, and business nomenclature.

Relationships between objects are defined in the semantic layer, not in the reporting tools.

Catalogs can be organized independently of the databases’ physical structures.

SQL generation

SQL queries are generated dynamically from the selected business objects, dimensions, measures, and filters.

Joins, aggregations, and calculation rules are applied by the engine.

User filters are propagated all the way to the source systems via the SQL Push-Down mechanism: no systematic copying of data — queries run against the existing systems, respecting the architectures already in place.

The generated SQL can be executed by {oa-lake} or by any compatible SQL engine.

Key point: users work with business objects; the SQL is generated automatically.

Smart Cache

Data stays in the source systems whenever possible.

The most heavily used datasets can be materialized in Parquet format.

The cache is driven by {oa-lake}’s Cache Manager, managed automatically based on observed usage.

Cache management remains independent of consuming tools.

Key point: the most expensive queries can be accelerated without modifying the source systems.

SQL Engine

{oa-lake} relies on a columnar, in-memory SQL engine.

The queries generated by {oa-lake-ctx} are executed by this engine or propagated to the source systems depending on the context.

The SQL remains portable, standard, and executable on different engines, independent of the storage technologies in use — which limits technology lock-in.

Key point: the SQL engine forms the execution layer of the architecture.

Security

Security is managed at the semantic layer level.

Access restrictions, user profiles, business catalogs, and security filters are applied before queries are generated.

This approach makes it possible to reproduce the security mechanisms found in platforms such as SAP BusinessObjects or IBM Cognos.

Key point: security rules are defined once and then applied to all connected tools.

Reporting

The catalogs exposed by {oa-lake-ctx} can be consumed by any tool capable of querying a PostgreSQL source.

Power BI, Looker, Spotfire, Tableau, Qlik, or other platforms access the same business objects, dimensions, measures, and KPIs.

The functional definitions remain identical regardless of the tool being used.

Result: several platforms can share the same business catalog with no duplication of calculation rules.

Governance

Business objects, KPIs, dimensions, measures, and calculation rules are centralized in a single repository.

Functional changes are made in {oa-lake-ctx} and are then immediately available to all connected tools.

Key point: {oa-lake-ctx} is the central point for defining business objects and KPIs.