close

1) {oA.Lineage}-dataflows: data lineage in dataflows

Starting from any "data point" (a field in a table, a table, a file, a schema), {openAudit} allows you to understand its origin and usage, both on-premises and in the Cloud, through a technical and multi-technological data lineage. The underlying analyses are replayed daily in an automated manner.

Information about the scheduling of chains is available, and the usage of each "data point" is defined on hover: who accesses what information, when, using which tools, which is of major interest in a "compliance" context.

{openAudit} offers different graphical representation modes for its data lineage (business/IT).

Use Cases:

Share a detailed understanding of flow construction, identify and correct breakpoints, Data Loss Prevention (DLP), BCBS 239 (Basel III), GDPR, etc.

Resolving breaks in Data Lineage

• Views: If they are stored, {openAudit} reads them, even if they are stacked (nested views).

• Dynamic SQL: If {openAudit} cannot directly resolve it, dynamic SQL is resolved with execution parameters or execution logs.

• Others: In case of information transfer via FTP or when the database schema is not specified (common in many ETL processes), {openAudit} resolves these breakpoints through structural recognition or by reading Batch/Shell.

Dynamically Associating Different Data Transformation Technologies

• {openAudit} analyzes all processing technologies (object/procedural language, ELT/ETL), on-premises or Cloud, and associates them in a single data flow at the finest level. Drill through provides access to the code.

• The process is dynamic, operated in delta mode, daily, and thus synchronized with the information system.

Different Levels of Analysis for Data Feeds

• Point Cloud: This view provides instant knowledge of the usage of a data point, abstracting away transformations. It is also possible, starting from a usage (a dashboard, data within a dashboard, a query), to instantly identify its operational sources.

• Mapping: This view, starting from any data point (field, table), displays a complete map of the upstream or downstream flow, from operational sources to data exposure (data visualization, query). Information used is highlighted, and data usage is specified on hover (who accesses the data, when, how).

• Granular Data Lineage: This view progressively tracks the deployment of data in the information system from a data point through iterative clicks, or conversely, traces back to operational sources. Each transformation (ELT/ETL job, procedural/object code) can be analyzed with "drill through." Detailed data usage (who accesses it, when, how) is specified.

---

2) {oA.lineage-viz}: data lineage in the data visualization layer

{openAudit} allows you to understand all data visualization technologies within the company on a single impact analysis interface: it's a grid that helps understand the hierarchy between each of the constituent elements of the dashboard and the physical source (or view), from the dashboard cell to the query that queries the database, through the semantic layer if there is one, etc.

Data lineage in the dashboard or in the data provisioning layers can be triggered from this interface.

Thus, all internal rules of data visualization technologies are highlighted and shared with everyone. This data lineage in the data visualization layer can be linked to data provisioning lineage.

Use Cases:

Shed light on complex business rules, perform impact analysis between a physical field and a dashboard data, etc.

Why it is useful

• Some data visualization technologies use semantic layers to create understanding for business users and give them autonomy. These semantic layers create abstraction: the underlying physical fields are hard to identify, making sourcing complex.

• In addition, data visualization technologies often query views, nested views, which further complicates sourcing.

• With the proliferation of data visualization technologies, performing multi-technological impact analysis (or sourcing) is complex.

• Data visualization technologies now allow for significant data preparation, creating high opacity.

Technically

• {openAudit} operates data lineage in the data visualization layer, in expressions, variables, etc., to identify fields directly or indirectly as the source of a data point.

• {openAudit} analyzes the content of views to identify physical fields that are the source of data in a data visualization layer, even if the views are stacked.

• {openAudit} combines analyses from different data visualization technologies into a single grid, allowing business and IT to perform impact analyses between all provisioning layers and all data visualization tools. Simply.

close

1) {oA.Harmony}-dataflows: Detecting Dead Branches in Dataflows

By analyzing information usage along with information flow (data lineage), {openAudit} identifies unnecessary flows and associated "data points" (tables/files).

On average, 50% of what is stored in the information system has no added value. These are countless "dead branches" made up of code, tables, views, and files that are wrongly maintained in systems, with a significant impact: system inertia, maintenance costs, unsustainable cloud bills, and significant environmental impacts (GHG).

Use Cases:

Decommissioning dead branches before migration, rationalizing a system to reduce maintenance, FinOps, GreenOps for more sustainable IT.

Technically

• {openAudit} analyzes audit database logs and the data visualization layer to discover which data is actually used.

• Starting from the fields used in databases to feed data visualization tools or ad-hoc queries (ODBC, JDBC), or even from specific ETL/ELT flows, {openAudit} identifies information flows, the tables that are sources, i.e., the "living branches" of the information system. In contrast, {openAudit} identifies "dead branches," i.e., tables, procedures, ETL/ELT jobs that build information that is never used.

• {openAudit} dynamically implements these analyses, allowing, by creating a significant historical depth, the formal identification of branches that are continually unused, along with everything they contain: tables, files, procedures, ETL/ELT jobs. Mass decommissioning can take place in record time.

• In the Cloud, through an analysis of certain logs, {openAudit} identifies the cost of keeping dead branches in the system. Machine resources that can be saved are also highlighted. The company can adopt a FinOps and GreenOps approach.

---

2) oA.Optimization-viz: Detecting Unnecessary, Replicated Dashboards

The data visualization layer unnecessarily contains complexity. User teams tend to copy and paste with minor variations.

{openAudit} will analyze sources, queries, expressions, variables, and ultimately displayed data to allow for optimizations. Outdated, broken, replicated dashboards are detected and can be archived or discarded.

We have developed features for SAP BO that allow certain actions to be automated en masse to quickly achieve a streamlined platform.

Use Cases:

Massive streamlining of the data visualization layer to reduce costs, reduce the risk of errors, and improve intelligibility.

Technically

• {openAudit} will directly parse files from the data visualization solution to retrieve intelligence, dashboard structure, and the semantic layer if there is one.

• {openAudit} will also access the repository to maintain ID consistency between different dashboard objects (semantic layer, query, dashboard, others).

• A probe from {openAudit} will retrieve certain logs from audit databases associated with data visualization solutions.

From there, {openAudit} will allow:

• To compare dashboards with each other and detect replication based on different criteria.

• To detect dashboard obsolescence.

• To identify broken formulas.

• To detect unnecessary queries, etc.

close

1) {oA.Reverse}-dataflows: Migrate Dataflow Technologies

Migrating procedural language technologies is often so delicate that companies prefer to stack technologies rather than decommission them. However, they struggle to maintain these languages due to a lack of experts capable of reverse engineering.

Today, the enthusiasm for the Cloud is changing this paradigm, and more and more companies are looking to get rid of these legacy languages quickly, with no other solution than embarking on risky and costly migrations.

Use Cases:

Changing DBMS, Cloud migration, maintaining a legacy system, etc.

The problem

• Large companies have accumulated processing technologies for a long time. Continuous stacking occurs because removing a technology often involves too many risks. But the associated skills are becoming scarce, and retro-documentation is rarely in place.

• At some point, companies need to take action! These can be consulting projects, long, costly, and risky. We believe it's better to automate the process.

Technically

• {openAudit} will "parse" the source code; it will break down all the code complexity using grammar that allows for exhaustive and ultra-granular analysis. All subtleties will be taken into consideration,

• {openAudit} deduces the overall kinematics and intelligence, which will be reconstructed in an agnostic algorithmic tree. Based on this, {openAudit} will produce "standard SQL,"

• Then the intelligence will be reconstructed at a minimum in the specific SQL of the target database (e.g., BigQuery for Google, Redshift for Amazon, Azure SQL for Microsoft, etc.),

• All complex processes that cannot be reproduced in simple SQL will be driven by an executable NodeJS or similar. Typical examples include "For Loops," variables, conditional code (If Else), switches, procedure calls, etc.,

• {openAudit} produces "Yaml" files (intuitive files). This way, the understanding of complexity is shared with the widest audience,

• If necessary, new orchestration mechanisms can be put in place to break down cursor cursors (looping loops) to optimize transformation chains.

---

2) {oA.Reverse}-dataviz: Migrate Dashboards to the Cloud

How to decommission an outdated dataviz technology that is too static, expensive, and incompatible with the target architecture, especially in the Cloud? How to confidently transition to the tools of the future, also favored by business users?

{openAudit} enables almost automated migrations between different dataviz technologies, saving an immense amount of time.

Use Cases:

Migrating SAP BO to Looker or PowerBI, migrating Qlik Sense to Power BI, etc. Many scenarios are possible!

Overall cinematics

• Most dataviz tools have two common elements: a semantic layer that interfaces between IT and business and a dashboard editor.

• We rely on the automated reverse engineering of {openAudit} to deconstruct the source complexity, allowing us to address it again in the target technology.

Methodology

• {openAudit} can feed the target technology from the unique semantic layer, a kind of pivot model. This model will have been automatically generated from the data visualization tools to be decommissioned,

• The structure of the initial dashboard will also have been analyzed by {openAudit}, and it can also be transcribed into the target technology,

Thus, sprawling, difficult, or impossible migration projects can be carried out in record time