Abstract: Systems and methods for automated techniques that generate queryable database table ownership attribution information in real-time. In addition to generating ownership attribution information, system and methods provide a novel framework for creating bi-partite graphs and generating insightful graph data.

Abstract: Incoming data requests from the perspective of the data lake, are gathered and analyzed to determine the usage of the data. Using the perspective of the data lake avoids the technical challenge of analyzing data usage by different computation points, which are at different locations, perform hard-to-track different operations, and are often reachable only through complicated access protocols. Another technical challenge of mapping between an object path and a table path is solved by generating object path datasets and table path datasets at different levels of abstractions. A comparison is performed, iteratively, from a lower level of granularity and the granularity is increased in the progressive steps. Matches from iterations are unionized to generate a final matching data. Observability metrics are generated using final matching data and are used to perform downstream operations such as controlling data table access, moving data tables to cold storage, decommissioning unused pipelines, etc.

Abstract: Systems and methods for automated techniques that generate queryable database table ownership attribution information in real-time. In addition to generating ownership attribution information, system and methods provide a novel framework for creating bi-partite graphs and generating insightful graph data.

Abstract: Systems and methods for updating a data catalog in real-time with changes to metadata are disclosed. An example method may be performed by one or more processors of a system and include detecting changes to metadata in a metadata store, generating an event queue indicating a plurality of change events associated with system objects based on the changes, each of the plurality of change events indicating a number of event attributes and a system object associated with the change event, identifying recent events among the plurality of change events, identifying relevant events among the recent events, extracting unique identifiers from the relevant events, identifying priority objects among the system objects, obtaining, from the metadata store, current metadata for each of the priority objects, and updating information associated with the priority objects in a data catalog, the updating occurring in real-time with the changes to the metadata in the metadata store.