ETL Migration |

ETL Migration - from legacy ETL to an open SQL ETL - {oa.tbx}

Thu, 02 Jul 2026 00:00:00 +0000

Once ETL processes have been converted to SQL by {openAudit}, they can be run inside the {oa.tbx} container.

{oa.tbx} provides an execution environment for SQL, Python, or Java processes, driven by configuration and orchestration repositories.

Key point: the migrated processes become SQL instructions that can be executed independently of the original ETL platform.

Principles

SQL is encapsulated in open containers, in ETL mode.

Containers: transformations run inside an external container, {oa.tbx}, detached from the target database. Temporary containers act as intermediate storage areas where data is loaded before transformation. Jobs are tested to verify correct behavior, and possibly the data itself, to guarantee its integrity.
Orchestration with Airflow: Airflow drives the execution of the containers, step by step. Each transformation, within a container, is defined as a job.

The scheduling of an ETL migrated to Airflow automatically picks up all of the original instructions and scheduling logic.

Execution chain

ETL jobs are converted to SQL by {openAudit}.

The generated processes are integrated into a configuration repository and a control repository used by {oa.tbx}.

The workflows are then orchestrated by tools such as Airflow or Dagster.

Execution container

{oa.tbx} runs the processes described in the workflows.

Dataflows can combine:

SQL;
Python;
Java;
others.

The engine can also rely on embedded databases such as DuckDB for certain intermediate processing steps.

Sources and targets

Input data can come from any technology accessible via SQL or JDBC.

Results can be loaded into any target system compatible with standard write mechanisms.

The architecture remains independent of source and target technologies.

Orchestration

Orchestration remains separate from the processes themselves.

Dependencies, execution parameters, schedules, and triggers are handled by the workflow layer.

Business logic stays contained within the SQL scripts and dataflows executed by {oa.tbx}.

Benefits

{oa.tbx} runs open SQL pipelines, orchestrated by Airflow and independent of proprietary Cloud platform runtimes.

Across all migrated jobs — with volumes ranging from a few thousand to more than 50 million rows — execution times are often just a few seconds, with an almost systematic OK status.

Governance and data lineage

Once ETL processes have been converted to SQL, {oa.tbx} keeps the complete trace between the original logic and the executed SQL. This data lineage is exposed at two levels of detail, suited to different needs.

Key point: a synthetic view for governance, an exhaustive detail view for maintenance.

High-level pipeline view: only source and target tables are visible; intermediate ETL stages are grouped into a single SQL block. A readable view, even for complex pipelines.

Detail on click for a transformation: clicking a stage (a Transformer, for example) displays its generated SQL equivalent, along with its inputs/outputs and its original metadata (type, stage ID). The mapping between the original ETL logic and the executed SQL is immediate.

Result: complete governance of the migrated pipeline, from the overview down to the executed SQL, regardless of the original ETL platform.

SQL as the pivot format

The SQL produced by {openAudit} becomes the primary vehicle for the transformations.

Business rules, joins, aggregations, and calculations are expressed in an open, version-controllable format, independent of legacy ETL tools.

Result: a SQL-based ETL/ELT chain, orchestrated by workflows and run in an open environment.

ETL Migration - from legacy ETL to dbt

Thu, 02 Jul 2026 00:00:00 +0000

Migrating to dbt means transforming proprietary ETL processes into SQL models run directly in the target data engine.

Before any conversion, {openAudit} analyzes flows, dependencies, and usage to identify the scope that is actually worth migrating.

Key point: the migration starts with a rationalization phase of the existing estate.

Flow analysis

ETL processes are analyzed to reconstruct dependencies, transformations, and usage.

Data lineage is used to identify active flows, dead branches, and unconsumed components.

This phase reduces the volume of processes to be migrated.

Result: a migration scope refocused on the processes that are actually used.

Conversion to SQL

ETL jobs are broken down into elementary operations: filters, joins, aggregations, lookups, business calculations, and transformations.

These components are converted into documented SQL, independent of the source technology.

SQL becomes the pivot representation of the processes.

Key point: business logic is rebuilt in SQL before the dbt project is generated.

Generating dbt models

The SQL processes are organized into dbt models.

Dependencies between models are automatically generated from the flows identified during the analysis.

The project structure, references between models, and documentation can be produced automatically.

Progressive validation

Migration is carried out in batches.

The generated models are validated and then progressively integrated into the target environment.

Discrepancies can be checked process by process during the transition phase.

Typical use case: progressive migration of several hundred ETL jobs to dbt.

Target architecture

The processes become version-controllable, maintainable SQL models.

Dependencies are explicit, transformations are visible, and processes can be run directly by Snowflake, BigQuery, Databricks SQL, Redshift, PostgreSQL, or any other compatible engine.

Key point: the processes no longer depend on the legacy ETL platform but on a dbt project built on open SQL.

ETL Migration - legacy ETL connectors for SQL migration

Thu, 02 Jul 2026 00:00:00 +0000

ETL platforms rely on connectors, transformation components, parameterization mechanisms, and execution strategies specific to each vendor.

{openAudit} analyzes these components, reconstructs their behavior, and then generates a SQL representation independent of the source technology.

Key point: the conversion is based on analyzing the ETL components, not on a simple syntactic translation.

Component analysis

Technical repositories are analyzed to identify connectors, transformations, parameters, dependencies, and execution mechanisms.

The components are converted into a common intermediate model before SQL generation.

This approach is identical regardless of the source technology.

Connectors

Read and write connectors are analyzed to reconstruct the flows between systems.

Connection parameters, queries, data formats, and mappings are integrated into the migration model.

Flows can originate from relational databases, files, APIs, or Cloud platforms.

Typical use case: reconstructing exchanges between Oracle, SQL Server, PostgreSQL, flat files, or Cloud storage.

Transformations

Transformation components are converted into SQL instructions.

Filters, joins, aggregations, lookups, mappings, type conversions, column calculations, and conditional expressions are reconstructed in the target SQL model.

Key point: each transformation becomes an explicit SQL operation.

Parameterization

Environment variables, job parameters, shared libraries, and reusable components are analyzed during the conversion.

Dependencies between processes and parameterization mechanisms are preserved in the generated model.

Supported technologies

The migration engines cover in particular IBM DataStage, Talend, SAP BODS, SSIS, ODI, Informatica PowerCenter, Stambia, Semarchy xDI, and Ab Initio.

The methodology stays the same: component analysis, flow reconstruction, then SQL generation.

Result: a consistent approach applied to different ETL technologies.

SQL as the pivot format

Once the components have been reconstructed, the processes are converted into a SQL model independent of the source platform.

The generated SQL can then be adapted to Snowflake, BigQuery, Databricks SQL, Redshift, PostgreSQL, SQL Server, or any other compatible engine.

Key point: SQL becomes the common format for representing the processes.