Abstract: Systems, methods, and computer-readable media for identifying an optimal cluster configuration for performing a job in a remote cluster computing system. In some examples, one or more applications and a sample of a production load as part of a job for a remote cluster computing system is received. Different clusters of nodes are instantiated in the remote cluster computing system to form different cluster configurations. Multi-Linear regression models segmented into different load regions are trained by running at least a portion of the sample on the instantiated different clusters of nodes. Expected completion times of the production load across varying cluster configurations are identified using the multi-linear regression models. An optimal cluster configuration of the varying cluster configurations is determined for the job based on the identified expected completion times.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can include a software development component and graphical user interface, referred to herein in some embodiments as a pipeline editor, or Lambda Studio IDE, that provides a visual environment for use with the system, including providing real-time recommendations for performing semantic actions on data accessed from an input HUB, based on an understanding of the meaning or semantics associated with the data.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can perform an ontology analysis of a schema definition, to determine the types of data, and datasets or entities, associated with that schema; and generate, or update, a model from a reference schema that includes an ontology defined based on relationships between datasets or entities, and their attributes. A reference HUB including one or more schemas can be used to analyze data flows, and further classify or make recommendations such as, for example, transformations enrichments, filtering, or cross-entity data fusion of an input data.

Abstract: Systems, methods, and computer-readable media for identifying an optimal cluster configuration for performing a job in a remote cluster computing system. In some examples, one or more applications and a sample of a production load as part of a job for a remote cluster computing system is received. Different clusters of nodes are instantiated in the remote cluster computing system to form different cluster configurations. Multi-Linear regression models segmented into different load regions are trained by running at least a portion of the sample on the instantiated different clusters of nodes. Expected completion times of the production load across varying cluster configurations are identified using the multi-linear regression models. An optimal cluster configuration of the varying cluster configurations is determined for the job based on the identified expected completion times.

Abstract: The present disclosure relates generally to a data enrichment service that automatically profiles data sets and provides visualizations of the profiles using a visual-interactive model within a client application (such as a web browser or mobile app). The visual profiling can be refined through end user interaction with the visualization objects and guide exploratory data visualization and discovery. Additionally, data sampling of heterogeneous data streams can be performed during ingestion to extract statistical attributes from multi-columnar data (e.g., standard deviation, median, mode, correlation coefficient, histogram, etc.). Data sampling can continue in real-time as data sources are updated.

Abstract: The present disclosure relates to performing similarity metric analysis and data enrichment using knowledge sources. A data enrichment service can compare an input data set to reference data sets stored in a knowledge source to identify similarly related data. A similarity metric can be calculated corresponding to the semantic similarity of two or more datasets. The similarity metric can be used to identify datasets based on their metadata attributes and data values enabling easier indexing and high performance retrieval of data values. A input data set can labeled with a category based on the data set having the best match with the input data set. The similarity of an input data set with a data set provided by a knowledge source can be used to query a knowledge source to obtain additional information about the data set. The additional information can be used to provide recommendations to the user.

Abstract: The present disclosure relates to performing similarity metric analysis and data enrichment using knowledge sources. A data enrichment service can compare an input data set to reference data sets stored in a knowledge source to identify similarly related data. A similarity metric can be calculated corresponding to the semantic similarity of two or more datasets. The similarity metric can be used to identify datasets based on their metadata attributes and data values enabling easier indexing and high performance retrieval of data values. A input data set can labeled with a category based on the data set having the best match with the input data set. The similarity of an input data set with a data set provided by a knowledge source can be used to query a knowledge source to obtain additional information about the data set. The additional information can be used to provide recommendations to the user.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can provide a service to recommend actions and transformations, on an input data, based on patterns identified from the functional decomposition of a data flow for a software application, including determining possible transformations of the data flow in subsequent applications. Data flows can be decomposed into a model describing transformations of data, predicates, and business rules applied to the data, and attributes used in the data flows.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can include a software development component and graphical user interface, referred to herein in some embodiments as a pipeline editor, or Lambda Studio IDE, that provides a visual environment for use with the system, including providing real-time recommendations for performing semantic actions on data accessed from an input HUB, based on an understanding of the meaning or semantics associated with the data.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can provide data governance functionality such as, for example, provenance (where a particular data came from), lineage (how the data was acquired/processed), security (who was responsible for the data), classification (what is the data about), impact (how impactful is the data to a business), retention (how long should the data live), and validity (whether the data should be excluded/included for analysis/processing), for each slice of data pertinent to a particular snapshot in time; which can then be used in making lifecycle decisions and dataflow recommendations.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system provides a programmatic interface, referred to herein in some embodiments as a foreign function interface, by which a user or third-party can define a service, functional and business types, semantic actions, and patterns or predefined complex data flows based on functional and business types, in a declarative manner, to extend the functionality of the system.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can provide support for auto-mapping of complex data structures, datasets or entities, between one or more sources or targets of data, referred to herein in some embodiments as HUBs. The auto-mapping can be driven by a metadata, schema, and statistical profiling of a dataset; and used to map a source dataset or entity associated with an input HUB, to a target dataset or entity or vice versa, to produce an output data prepared in a format or organization (projection) for use with one or more output HUBs.

Abstract: In accordance with various embodiments, described herein is a system (Data Artificial Intelligence system, Data AI system), for use with a data integration or other computing environment, that leverages machine learning (ML, DataFlow Machine Learning, DFML), for use in managing a flow of data (dataflow, DF), and building complex dataflow software applications (dataflow applications, pipelines). In accordance with an embodiment, the system can perform an ontology analysis of a schema definition, to determine the types of data, and datasets or entities, associated with that schema; and generate, or update, a model from a reference schema that includes an ontology defined based on relationships between datasets or entities, and their attributes. A reference HUB including one or more schemas can be used to analyze data flows, and further classify or make recommendations such as, for example, transformations enrichments, filtering, or cross-entity data fusion of an input data.

Abstract: The present disclosure relates generally to a data enrichment service that extracts, repairs, and enriches datasets, resulting in more precise entity resolution and correlation for purposes of subsequent indexing and clustering. As the data enrichment service can include a visual recommendation engine and language for performing large-scale data preparation, repair, and enrichment of heterogeneous datasets. This enables the user to select and see how the recommended enrichments (e.g., transformations and repairs) will affect the user's data and make adjustments as needed. The data enrichment service can receive feedback from users through a user interface and can filter recommendations based on the user feedback.

Abstract: The present disclosure relates generally to a data enrichment service that automatically profiles data sets and provides visualizations of the profiles using a visual-interactive model within a client application (such as a web browser or mobile app). The visual profiling can be refined through end user interaction with the visualization objects and guide exploratory data visualization and discovery. Additionally, data sampling of heterogeneous data streams can be performed during ingestion to extract statistical attributes from multi-columnar data (e.g., standard deviation, median, mode, correlation coefficient, histogram, etc.). Data sampling can continue in real-time as data sources are updated.

Abstract: Techniques are disclosure for a data enrichment system that enables declarative external data source importation and exportation. A user can specify via a user interface input for identifying different data sources from which to obtain input data. The data enrichment system is configured to import and export various types of sources storing resources such as URL-based resources and HDFS-based resources for high-speed bi-directional metadata and data interchange. Connection metadata (e.g., credentials, access paths, etc.) can be managed by the data enrichment system in a declarative format for managing and visualizing the connection metadata.

Abstract: The present disclosure describes techniques for entity classification and data enrichment of data sets. A data enrichment system is disclosed that can extract, repair, and enrich datasets, resulting in more precise entity resolution and classification for purposes of subsequent indexing and clustering. Disclosed techniques may include performing entity recognition to identify segments of interest that relate to an entity. Related data may be analyzed for classification, which can be used to transform the data for enrichment to its users.

Abstract: The present disclosure relates to performing similarity metric analysis and data enrichment using knowledge sources. A data enrichment service can compare an input data set to reference data sets stored in a knowledge source to identify similarly related data. A similarity metric can be calculated corresponding to the semantic similarity of two or more datasets. The similarity metric can be used to identify datasets based on their metadata attributes and data values enabling easier indexing and high performance retrieval of data values. A input data set can labeled with a category based on the data set having the best match with the input data set. The similarity of an input data set with a data set provided by a knowledge source can be used to query a knowledge source to obtain additional information about the data set. The additional information can be used to provide recommendations to the user.

Abstract: The subject disclosure generally relates to dynamic user experience adaptation and services provisioning. A user experience component can provide a user experience (UX) to a user. The UX can include, but is not limited to, an operating system, an application (e.g., word processor, electronic mail, computer aided drafting, video game, etc.), a user interface, and so forth. A monitoring component can monitor feedback generated in association with interaction with the user experience by the user. An update component can analyze the feedback, and update a user model associated with the user based at least in part on the analysis, and an adaptation component can modify the user experience based at least in part the user model.

Abstract: Techniques for defocusing queries over big datasets and dynamic datasets are provided to broaden search results and incorporate all potentially relevant data and avoid overly narrowing queries. An analytic component can receive queries directed at one region of a dataset and analyze the queries to generate inferences about the queries. The queries can then be defocused by a defocusing component and incorporate a larger dataset than originally searched to broaden the queries. The larger dataset can incorporate all, or a part of the original dataset and can also be disparate from the original dataset. Clusters of queries can also be merged and unified to deal with ‘local minima’ issues and broaden the understanding of the dataset. In other embodiments, dynamic data can be monitored and changes tracked, to ensure that all portions of the dataset are being searched by the queries.