Abstract: Systems and methods for automated techniques that generate queryable database table ownership attribution information in real-time. In addition to generating ownership attribution information, system and methods provide a novel framework for creating bi-partite graphs and generating insightful graph data.

Abstract: Incoming data requests from the perspective of the data lake, are gathered and analyzed to determine the usage of the data. Using the perspective of the data lake avoids the technical challenge of analyzing data usage by different computation points, which are at different locations, perform hard-to-track different operations, and are often reachable only through complicated access protocols. Another technical challenge of mapping between an object path and a table path is solved by generating object path datasets and table path datasets at different levels of abstractions. A comparison is performed, iteratively, from a lower level of granularity and the granularity is increased in the progressive steps. Matches from iterations are unionized to generate a final matching data. Observability metrics are generated using final matching data and are used to perform downstream operations such as controlling data table access, moving data tables to cold storage, decommissioning unused pipelines, etc.

Abstract: Systems and methods for automated techniques that generate queryable database table ownership attribution information in real-time. In addition to generating ownership attribution information, system and methods provide a novel framework for creating bi-partite graphs and generating insightful graph data.

Abstract: Systems and methods for updating a data catalog in real-time with changes to metadata are disclosed. An example method may be performed by one or more processors of a system and include detecting changes to metadata in a metadata store, generating an event queue indicating a plurality of change events associated with system objects based on the changes, each of the plurality of change events indicating a number of event attributes and a system object associated with the change event, identifying recent events among the plurality of change events, identifying relevant events among the recent events, extracting unique identifiers from the relevant events, identifying priority objects among the system objects, obtaining, from the metadata store, current metadata for each of the priority objects, and updating information associated with the priority objects in a data catalog, the updating occurring in real-time with the changes to the metadata in the metadata store.

Abstract: Systems and methods for generating augmented search results are disclosed. An example method may be performed by one or more processors of a ranking system and include receiving a search query, generating, for each respective data asset of a plurality of data assets, a query relevancy score indicating a likelihood that the respective data asset is relevant to the search query, retrieving, for each of a number of potentially relevant data assets, a utility score indicating a likelihood that the potentially relevant data asset is useful, generating, for each of the number of potentially relevant data assets, an augmented query score indicating a likelihood that the potentially relevant data asset is relevant to and useful for the search query based on the query relevancy scores and the utility scores, and generating augmented search results for the search query based on the augmented query scores.

Abstract: Systems and methods for generating augmented search results are disclosed. An example method may be performed by one or more processors of a ranking system and include receiving a search query, generating, for each respective data asset of a plurality of data assets, a query relevancy score indicating a likelihood that the respective data asset is relevant to the search query, retrieving, for each of a number of potentially relevant data assets, a utility score indicating a likelihood that the potentially relevant data asset is useful, generating, for each of the number of potentially relevant data assets, an augmented query score indicating a likelihood that the potentially relevant data asset is relevant to and useful for the search query based on the query relevancy scores and the utility scores, and generating augmented search results for the search query based on the augmented query scores.

Abstract: Systems and methods for identifying one or more master tables of a datalake are described. A system may obtain a plurality of computer readable tables of a datalake (with each computer readable table including one or more features). The system may also group the plurality of computer readable tables into a plurality of groups based on a number of features of each computer readable table of the plurality of computer readable tables. The system may further generate, for each of one or more groups of the plurality of groups, one or more neighborhoods based on a similarity of features between computer readable tables of the group. The system may also identify, for each neighborhood, one or more master tables from the one or more computer readable tables of the group. The system may further provide an indication of one or more master tables identified in the datalake.