ETL Migration - legacy ETL connectors for SQL migration
ETL platforms rely on connectors, transformation components, parameterization mechanisms, and execution strategies specific to each vendor.
{openAudit} analyzes these components, reconstructs their behavior, and then generates a SQL representation independent of the source technology.
Key point: the conversion is based on analyzing the ETL components, not on a simple syntactic translation.
Component analysis
Technical repositories are analyzed to identify connectors, transformations, parameters, dependencies, and execution mechanisms.
The components are converted into a common intermediate model before SQL generation.
This approach is identical regardless of the source technology.
Connectors
Read and write connectors are analyzed to reconstruct the flows between systems.
Connection parameters, queries, data formats, and mappings are integrated into the migration model.
Flows can originate from relational databases, files, APIs, or Cloud platforms.
Typical use case: reconstructing exchanges between Oracle, SQL Server, PostgreSQL, flat files, or Cloud storage.
Transformations
Transformation components are converted into SQL instructions.
Filters, joins, aggregations, lookups, mappings, type conversions, column calculations, and conditional expressions are reconstructed in the target SQL model.
Key point: each transformation becomes an explicit SQL operation.
Parameterization
Environment variables, job parameters, shared libraries, and reusable components are analyzed during the conversion.
Dependencies between processes and parameterization mechanisms are preserved in the generated model.
Supported technologies
The migration engines cover in particular:
The methodology stays the same: component analysis, flow reconstruction, then SQL generation.
Result: a consistent approach applied to different ETL technologies.
SQL as the pivot format
Once the components have been reconstructed, the processes are converted into a SQL model independent of the source platform.
The generated SQL can then be adapted to Snowflake, BigQuery, Databricks SQL, Redshift, PostgreSQL, SQL Server, or any other compatible engine.
Key point: SQL becomes the common format for representing the processes.
{oa-tbx}: the recommended target
Rewriting DataStage, PowerCenter, BODS, Talend, ODI, or SSIS jobs in another proprietary ETL means renewing the licenses, the dedicated infrastructure, and the vendor lock-in.
{oa-tbx} is an open-source, serverless SQL ETL with an embedded DuckDB engine. The jobs automatically transcribed into SQL by {openAudit} run there as pipelines orchestrated by Airflow, with no dedicated infrastructure.
The SQL produced by the connectors can be executed directly in {oa-tbx}.
- Automatic transcription: {openAudit} converts legacy jobs into flat SQL, without months of reverse engineering.
- Open orchestration: the job sequence of the source ETL is reproduced as-is as an Airflow DAG.
- Embedded DuckDB: an open-source analytical SQL engine with no server to administer; jobs ranging from a few thousand to over 50 million rows often run in a few seconds.
- Pipeline as Code: SQL jobs and Airflow DAGs in Python, versioned in Git and deployed with Terraform.
- Governance through data lineage: each original transformation stays linked to the SQL actually executed.
- Zero lock-in: containerized, Java / Python, no proprietary license or roadmap.
Key point: with {oa-tbx}, migrated flows become open SQL, readable and auditable line by line, and executable on any environment.
