Abstract: A data catalog system is disclosed that provides capabilities for uniquely identifying and retrieving data entities stored in diverse data sources managed by an organization. The data catalog system includes capabilities for generating a unique external identifier for a data entity (e.g., a data asset or a data object) by identifying a set of immutable configuration parameters associated with the data asset and identifying a set of data object attributes that uniquely identify data objects within the data asset. The generated unique external identifiers are stored as part of the metadata harvested by the data catalog system. The external identifiers are used to enforce a single representation of the data assets and the data objects in the data catalog system. The external object identifiers are used to perform data lookups and reconcile states of data entities during the metadata harvesting process.

Abstract: The present disclosure relates to a system and techniques for resolving dangling references resulting from a dependency relationship between computing resource objects uncovered during a harvesting process. In embodiments, a harvester application adds computing resource objects associated with a client to a resource collection as those computing resource objects are identified. Dependencies are identified as each computing resource object is added to the resource collection, which are resolved only if the computing resource objects associated with those dependencies have already been added to the resource collection. If the computing resource objects associated with the dependencies have not already been added to the resource collection, then the dependency is added to an observer pool. Observer modules are configured to check each computing resource object as it is processed during the harvest process in order to match those computing resource objects to unresolved dependencies.

Abstract: The present disclosure relates to a system and techniques for resolving dangling references resulting from a dependency relationship between computing resource objects uncovered during a harvesting process. In embodiments, a harvester application adds computing resource objects associated with a client to a resource collection as those computing resource objects are identified. Dependencies are identified as each computing resource object is added to the resource collection, which are resolved only if the computing resource objects associated with those dependencies have already been added to the resource collection. If the computing resource objects associated with the dependencies have not already been added to the resource collection, then the dependency is added to an observer pool. Observer modules are configured to check each computing resource object as it is processed during the harvest process in order to match those computing resource objects to unresolved dependencies.

Abstract: A machine-learning model may be previously trained with a supervised learning algorithm to identify whether a pair of labels provided as input are similar. A locality sensitive hashing forest (LSH) may be generated for the set of candidate labels. When a user later identifies an input label (e.g., by search query, by interface selection, etc.) the input label may be used to query the LSH forest to identify a subset of the candidate labels. This subset may be used to generate respective pairs comprising the input label, one of the subset candidate labels, and a corresponding feature set generated for the pair. This data may be provided to the model to identify a degree to which the pair of labels are similar. The user may be provided one or more recommendations including similar terms identified from the model's output.

Abstract: The present disclosure relates to system and techniques for enabling provisioning of cloud services for a client in an isolated yet scalable manner. In some embodiments, various computing resources are implemented within a cell (a self-sufficient unit). A number of cells are generated for a service or a group of services and distributed across a number of computing devices. Various cells may be generated that each pertain to a different aspect, or particular functionality, of the service. In some embodiments, cells providing various functionality for the service are implemented and distributed across different computing devices.

Abstract: The present disclosure relates to system and techniques for enabling provisioning of cloud services for a client in an isolated yet scalable manner. In some embodiments, various computing resources are implemented within a cell (a self-sufficient unit). A number of cells are generated for a service or a group of services and distributed across a number of computing devices. Various cells may be generated that each pertain to a different aspect, or particular functionality, of the service. In some embodiments, cells providing various functionality for the service are implemented and distributed across different computing devices.

Abstract: A machine-learning model may be previously trained with a supervised learning algorithm to identify whether a pair of labels provided as input are similar. A locality sensitive hashing forest (LSH) may be generated for the set of candidate labels. When a user later identifies an input label (e.g., by search query, by interface selection, etc.) the input label may be used to query the LSH forest to identify a subset of the candidate labels. This subset may be used to generate respective pairs comprising the input label, one of the subset candidate labels, and a corresponding feature set generated for the pair. This data may be provided to the model to identify a degree to which the pair of labels are similar. The user may be provided one or more recommendations including similar terms identified from the model's output.

Abstract: Systems, devices, and methods discussed herein are directed to utilizing patterns and logical entities to identify and maintain relationships between data assets. In some embodiments, a query comprising a logical entity qualifier, one or more pattern identifiers that indicate a pattern, and a data set identifier may be received. The pattern is executed against a data set corresponding to the data set identifier and one or more logical entities are generated based on this execution. A logical entity may be a label that represents a set of one or more data assets in a data set. Assets that share a label can share attributes that are described by the label. The label corresponding to each logical entity may be presented, where each label represents a different set of data assets which share a common trait. In some embodiments, the user may define a pattern by which commonality may be assessed.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can perform an ontology analysis of a schema definition, to determine the types of data, and datasets or entities, associated with that schema; and generate, or update, a model from a reference schema that includes an ontology defined based on relationships between datasets or entities, and their attributes. A reference HUB including one or more schemas can be used to analyze data flows, and further classify or make recommendations such as, for example, transformations enrichments, filtering, or cross-entity data fusion of an input data.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can include a software development component and graphical user interface, referred to herein in some embodiments as a pipeline editor, or Lambda Studio IDE, that provides a visual environment for use with the system, including providing real-time recommendations for performing semantic actions on data accessed from an input HUB, based on an understanding of the meaning or semantics associated with the data.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system provides a programmatic interface, referred to herein in some embodiments as a foreign function interface, by which a user or third-party can define a service, functional and business types, semantic actions, and patterns or predefined complex data flows based on functional and business types, in a declarative manner, to extend the functionality of the system.

Abstract: A machine-learning model may be previously trained with a supervised learning algorithm to identify whether a pair of labels provided as input are similar. A locality sensitive hashing forest (LSH) may be generated for the set of candidate labels. When a user later identifies an input label (e.g., by search query, by interface selection, etc.) the input label may be used to query the LSH forest to identify a subset of the candidate labels. This subset may be used to generate respective pairs comprising the input label, one of the subset candidate labels, and a corresponding feature set generated for the pair. This data may be provided to the model to identify a degree to which the pair of labels are similar. The user may be provided one or more recommendations including similar terms identified from the model's output.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can provide a service to recommend actions and transformations, on an input data, based on patterns identified from the functional decomposition of a data flow for a software application, including determining possible transformations of the data flow in subsequent applications. Data flows can be decomposed into a model describing transformations of data, predicates, and business rules applied to the data, and attributes used in the data flows.

Abstract: Systems, devices, and methods discussed herein are directed to utilizing patterns and logical entities to identify and maintain relationships between data assets. In some embodiments, a query comprising a logical entity qualifier, one or more pattern identifiers that indicate a pattern, and a data set identifier may be received. The pattern is executed against a data set corresponding to the data set identifier and one or more logical entities are generated based on this execution. A logical entity may be a label that represents a set of one or more data assets in a data set. Assets that share a label can share attributes that are described by the label. The label corresponding to each logical entity may be presented, where each label represents a different set of data assets which share a common trait. In some embodiments, the user may define a pattern by which commonality may be assessed.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can provide data governance functionality such as, for example, provenance (where a particular data came from), lineage (how the data was acquired/processed), security (who was responsible for the data), classification (what is the data about), impact (how impactful is the data to a business), retention (how long should the data live), and validity (whether the data should be excluded/included for analysis/processing), for each slice of data pertinent to a particular snapshot in time; which can then be used in making lifecycle decisions and dataflow recommendations.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can provide data governance functionality such as, for example, provenance (where a particular data came from), lineage (how the data was acquired/processed), security (who was responsible for the data), classification (what is the data about), impact (how impactful is the data to a business), retention (how long should the data live), and validity (whether the data should be excluded/included for analysis/processing), for each slice of data pertinent to a particular snapshot in time; which can then be used in making lifecycle decisions and dataflow recommendations.

Abstract: A data catalog system is disclosed that provides capabilities for uniquely identifying and retrieving data entities stored in diverse data sources managed by an organization. The data catalog system includes capabilities for generating a unique external identifier for a data entity (e.g., a data asset or a data object) by identifying a set of immutable configuration parameters associated with the data asset and identifying a set of data object attributes that uniquely identify data objects within the data asset. The generated unique external identifiers are stored as part of the metadata harvested by the data catalog system. The external identifiers are used to enforce a single representation of the data assets and the data objects in the data catalog system. The external object identifiers are used to perform data lookups and reconcile states of data entities during the metadata harvesting process.

Abstract: Systems, devices, and methods discussed herein are directed to utilizing patterns and logical entities to identify and maintain relationships between data assets. In some embodiments, a query comprising a logical entity qualifier, one or more pattern identifiers that indicate a pattern, and a data set identifier may be received. The pattern is executed against a data set corresponding to the data set identifier and one or more logical entities are generated based on this execution. A logical entity may be a label that represents a set of one or more data assets in a data set. Assets that share a label can share attributes that are described by the label. The label corresponding to each logical entity may be presented, where each label represents a different set of data assets which share a common trait. In some embodiments, the user may define a pattern by which commonality may be assessed.

Abstract: The present disclosure relates to system and techniques for enabling provisioning of cloud services for a client in an isolated yet scalable manner. In some embodiments, various computing resources are implemented within a cell (a self-sufficient unit). A number of cells are generated for a service or a group of services and distributed across a number of computing devices. Various cells may be generated that each pertain to a different aspect, or particular functionality, of the service. In some embodiments, cells providing various functionality for the service are implemented and distributed across different computing devices.

Abstract: The present disclosure relates to a system and techniques for resolving dangling references resulting from a dependency relationship between computing resource objects uncovered during a harvesting process. In embodiments, a harvester application adds computing resource objects associated with a client to a resource collection as those computing resource objects are identified. Dependencies are identified as each computing resource object is added to the resource collection, which are resolved only if the computing resource objects associated with those dependencies have already been added to the resource collection. If the computing resource objects associated with the dependencies have not already been added to the resource collection, then the dependency is added to an observer pool. Observer modules are configured to check each computing resource object as it is processed during the harvest process in order to match those computing resource objects to unresolved dependencies.