Abstract: A computer-implemented method for optimizing a data pipeline system includes processing a data pipeline configuration manifest to generate a framework of the data pipeline system and a data flow logic package of the data pipeline system. The data pipeline configuration manifest includes an object-oriented metadata model of the data pipeline system. The computer-implemented method further includes monitoring performance of the data pipeline system during execution of the data flow logic package to obtain a performance metric for the data pipeline system, and modifying, with a processor, the framework of the data pipeline system based on the data pipeline configuration manifest and the performance metric.

Abstract: An administrator of an enterprise storage set may be tasked with storing a large number and variety of data sets on a large number and variety of storage components. However, the manual selection of a physical schema by an administrator may be time-consuming, may generate inefficient physical schemata, and may not be easily reevaluated as the data sets and storage set change. Presented herein are techniques for automatically determining a physical schema by comparing the storage factors of each data set (e.g., data size, relationships with other data sets, and usages of the data set by users) with the storage capabilities of the storage components, selecting a suitable storage component, and implementing the storage of the data set on the storage component. An embodiment of these techniques may thereby achieve an automated identification of a physical schema with improved efficiency and flexibility of the physical schema while conserving administrative resources.

Abstract: Systems and methods for dynamically managed data retention are described. The system comprises a tiered framework having a plurality of namespaces. The namespaces are configured by a user to have selected data retention attributes. Data including a manifest may be received by the system, processed, and directed to a namespace based upon the manifest. Data storage partitions may be created automatically in association with a namespace, and the data partitions may be assigned partition attributes. Data in a storage partition may be migrated automatically to another namespace based on the partition attributes. Code necessary for creating storage partitions and migrating data is generated by the data management system.

Abstract: Systems and methods for dynamically managed data retention are described. The system comprises a tiered framework having a plurality of namespaces. The namespaces are configured by a user to have selected data retention attributes. Data including a manifest may be received by the system, processed, and directed to a namespace based upon the manifest. Data storage partitions may be created automatically in association with a namespace, and the data partitions may be assigned partition attributes. Data in a storage partition may be migrated automatically to another namespace based on the partition attributes. Code necessary for creating storage partitions and migrating data is generated by the data management system.

Abstract: A computer-implemented method for optimizing a data pipeline system includes processing a data pipeline configuration manifest to generate a framework of the data pipeline system and a data flow logic package of the data pipeline system. The data pipeline configuration manifest includes an object-oriented metadata model of the data pipeline system. The computer-implemented method further includes monitoring performance of the data pipeline system during execution of the data flow logic package to obtain a performance metric for the data pipeline system, and modifying, with a processor, the framework of the data pipeline system based on the data pipeline configuration manifest and the performance metric.

Abstract: If a response is with a threshold of the predicted response, then the contradictory response is classified as being independent and it is not truly contradictory. If a response is beyond a threshold, the response is classified as not being independent and is likely worthy of further investigation.

Abstract: An administrator of an enterprise storage set may be tasked with storing a large number and variety of data sets on a large number and variety of storage components. However, the manual selection of a physical schema by an administrator may be time-consuming, may generate inefficient physical schemata, and may not be easily reevaluated as the data sets and storage set change. Presented herein are techniques for automatically determining a physical schema by comparing the storage factors of each data set (e.g., data size, relationships with other data sets, and usages of the data set by users) with the storage capabilities of the storage components, selecting a suitable storage component, and implementing the storage of the data set on the storage component. An embodiment of these techniques may thereby achieve an automated identification of a physical schema with improved efficiency and flexibility of the physical schema while conserving administrative resources.

Abstract: Systems and methods for dynamically managed data retention are described. The system comprises a tiered framework having a plurality of namespaces. The namespaces are configured by a user to have selected data retention attributes. Data including a manifest may be received by the system, processed, and directed to a namespace based upon the manifest. Data storage partitions may be created automatically in association with a namespace, and the data partitions may be assigned partition attributes. Data in a storage partition may be migrated automatically to another namespace based on the partition attributes. Code necessary for creating storage partitions and migrating data is generated by the data management system.

Abstract: If a response is with a threshold of the predicted response, then the contradictory response is classified as being independent and it is not truly contradictory. If a response is beyond a threshold, the response is classified as not being independent and is likely worthy of further investigation.