Abstract: A system, method and device for ingesting data files into remote computing environments is disclosed. The method includes receiving a plurality of data files and processing same according to a modified round-robin (MRR) process. The MRR assigns data files for processing by determining which of a plurality of remote processors are active, and by determining an amount of queued processing for the active remote processors. The method includes assigning each data file of the plurality of data files to a remote processor of the plurality of remote processors based on the remote processor (1) being active, and (2) having a relatively lower amount of queued processing. The method includes instructing a storage writer to store processed data files.

Abstract: Apparatus and methods for expanding a data transfer framework are disclosed. Exemplary implementations may: provide an ETL framework comprising a plurality of ETL modules and comprising code including variables; obtain a configuration file including data values to replace the variables for executing selected ETL modules, and including external command data configured to execute a new data transformation external to and absent from the ETL framework, the external command data including a reference to an external module generated in relation to an external interface; and execute the one or more selected ETL modules based on the code, the data values and the external command data. Exemplary implementations provide a flexible and expandable ETL framework that enable a new type of data transformation that is not currently supported by the framework, without having to modify the framework. The framework may impart native properties and characteristics of the framework to the external module.

Abstract: A system, method and memory for managing remotely stored data is disclosed. The system includes a processor, a communications module, and a memory. Instructions stored on the memory cause the processor to provide an agent within a remote computing environment (RCE) with access to data stored on the RCE and custom applications. Another agent on a local computing environment capable of communicating with the agent is provided. The other agent has access to configuration files. The other agent instructs the agent to execute custom applications based on the configuration files. The other agent receives metadata of the data in response to triggering the agent. The instructions cause the processor to populate, via the other agent, a catalogue of the plurality of data within the remote computing environment based on the received plurality of metadata, and provide the catalogue for display for reviewing the plurality of data.

Abstract: A method, device and memory for providing access to remotely stored data is provided. The method includes providing an interface capable of communicating with a remote computing environment storing a plurality of data sets. The method includes receiving a request, via the interface, to access at data sets according to access parameters. The method includes applying a first set of masking parameters when the data sets are accessed via respective views. The method includes satisfying the request by granting access to the views associated with the data sets, without copying the data sets. The views enable data from the data sets to be viewed as masked according to the first set of masking parameters, and the views are masked according to a second set of masking parameters associated with the access parameters.

Abstract: A system, device, and method for ingesting data into a remote computing environment are provided. The example device comprises a processor, a communications module, and a memory. The processor executes instructions on the memory to receive a data file, and extract metadata from the data file. The extracted metadata comprises at least one property of the data file. A configuration file, from a plurality of configuration files, that is associated with the data file is determined. The determination is performed, at least in part, based on correlating the extracted metadata with data file types used by the determined configuration file. The data file is ingested for storage in a remote computing environment based on the determined configuration file.

Abstract: A device, method and system for loading data into a remote computing environment is disclosed. The method includes receiving a request to load a new data set into a remote computing environment, the new data set impacting a data set stored thereon. The method includes identifying one or more changes to a current representation of the data set within the new data set, the one or more changes replacing information in the current representation. The method includes transmitting the identified one or more changes to a data store persisting the current representation. The method includes transmitting the replaced information to a data store persisting a previous representation of the data set. The method includes transmitting other information in the new data set that is determined to be invalid data to a data store persisting an invalid data set associated with the data set.

Abstract: A system, method and device for ingesting data files into remote computing environments is disclosed. The method includes receiving a plurality of data files and processing same according to a modified round-robin (MRR) process. The MRR assigns data files for processing by determining which of a plurality of remote processors are active, and by determining an amount of queued processing for the active remote processors. The method includes assigning each data file of the plurality of data files to a remote processor of the plurality of remote processors based on the remote processor (1) being active, and (2) having a relatively lower amount of queued processing. The method includes instructing a storage writer to store processed data files.

Abstract: Computing platforms, methods, and storage media for data movement are disclosed. Exemplary implementations may: obtain a data transfer command including a table-based characterization of a set of data to be transferred; automatically determine, based on the table-based characterization of the data to be transferred, a set of files to be transferred; and initiate transfer of the set of files. In an implementation, data is moved at the table level, and the files associated with the specified tables/date range are automatically determined. A single command, which may reference a CSV file, may be used to efficiently and reliably transfer a large amount of data without a user having to specify the specific files to be transferred.

Abstract: Computing platforms, methods, and storage media for data movement are disclosed. Exemplary implementations may: obtain a data transfer command including a table-based characterization of a set of data to be transferred from an on-premise platform to a cloud platform, the data transfer command further including a date specification; automatically determine, based on the table-based characterization of the data to be transferred and based on the date specification, a set of files to be transferred; and initiate transfer of the set of files from the on-premise platform to the cloud platform. In an implementation, data is moved at the table level, and the files associated with the specified tables/date range are automatically determined. A single command, which may reference a CSV file, may be used to efficiently and reliably transfer a large amount of data without a user having to specify the specific files to be transferred.