Data lineage - the principles of connector containerization
{openAudit} collection relies on containerized components deployed in the Client’s environment, as close as possible to the systems analyzed: one container per technology (databases, ETL/ELT platforms, data visualization tools, audit databases), compatible with Docker and Podman, for on-premise, SaaS, or hybrid deployment.
Key point: extractors and anonymizers are decoupled from the processing engines; each component is deployed and updated independently.
Extraction
Extractors connect to the technologies analyzed through dedicated service accounts and collect metadata from databases, ETL/ELT platforms, data visualization tools, audit databases, and the scheduler. Each extractor produces a homogeneous output format, whatever the technology.
Anonymization
When required, an anonymization container sits between collection and the production of exchange files: character strings, application parameters, and literal values (SQL constants…) are replaced with technical variables. Dependencies are preserved, sensitive information removed.
Exchange files
The components produce standardized CSV files: technical objects, dependencies, transformations and, when available, usage information. The Client can archive, check, or validate them before transfer.
Processing
The central {openAudit} components use these files to reconstruct dependencies, consolidate metadata, integrate usage from audit logs, and produce the data lineage.
Result: a homogeneous collection chain, independent of the technologies analyzed and of deployment modes.
