Abstract: Embodiments of the present systems and methods may provide the capability ensure that data is persisted and accessed correctly without depending on eventually consistent list operations on the object store. For example, in an embodiment, a computer-implemented method for data distribution may comprise attempting to persist a plurality of data parts from a plurality of processing tasks, generating a manifest including information indicating those attempts to persist data parts that have succeeded, and persisting the manifest with the data parts that have been successfully persisted.

Abstract: Embodiments of the present systems and methods may provide techniques to provide simple and accurate estimate of memory requirements for application invocation in a serverless environment. For example, a method may comprise selecting sample invocations of functions as a service from a larger plurality of invocations, submitting for execution the plurality of sample invocations and, for each sample invocation, submitting a specification of a memory size to be used for execution of each sample invocation, determining, whether the specification of the memory size to be used for execution of each sample invocation results in unsuccessful execution of at least some of the sample invocations due to insufficient memory and, if so, adjusting the specification of the memory size for at least some of the sample invocations, and submitting for execution at least those invocations in the larger plurality of invocations that were not included in the plurality of sample invocations.

Abstract: Embodiments of the present systems and methods may provide techniques to provide simple and accurate estimate of memory requirements for application invocation in a serverless environment. For example, a method may comprise selecting sample invocations of functions as a service from a larger plurality of invocations, submitting for execution the plurality of sample invocations and, for each sample invocation, submitting a specification of a memory size to be used for execution of each sample invocation, determining, whether the specification of the memory size to be used for execution of each sample invocation results in unsuccessful execution of at least some of the sample invocations due to insufficient memory and, if so, adjusting the specification of the memory size for at least some of the sample invocations, and submitting for execution at least those invocations in the larger plurality of invocations that were not included in the plurality of sample invocations.

Abstract: Embodiments of the present systems and methods may provide the capability ensure that data is persisted and accessed correctly without depending on eventually consistent list operations on the object store. For example, in an embodiment, a computer-implemented method for data distribution may comprise attempting to persist a plurality of data parts from a plurality of processing tasks, generating a manifest including information indicating those attempts to persist data parts that have succeeded, and persisting the manifest with the data parts that have been successfully persisted.

Abstract: Aspects of the present invention include a method, system and computer program product for performing data deduplication for eventually consistent distributed data storage (DDS) system. The method includes receiving data content from one or more clients by a DDS system, wherein the one or more clients do not coordinate transmitting of the data content. The method also includes calculating a hash for the data content by the distributed data storage system, writing the data content to an object used for data deduplication, wherein a name of the object is based on the hash and determining whether the data content is present in the distributed data storage system based on the name of an object previously stored on the DDS system. The method further includes keeping track of a number of references to the data content and delaying deletion of the data content for a predetermined period of time.

Abstract: A computer-implemented method for determining whether to perform a pushdown may include receiving a request for analytics to be performed by an analytics platform on data stored in bulk storage. An operation may be identified as a candidate for a pushdown, where the operation is selected from among one or more operations to be performed for fulfilling the request. The pushdown would require the operation to be performed at the bulk storage. The data may be sampled by reading one or more samples of the data, where the one or more samples are a fraction of the data. The operation may be performed, by a computer processor, on the one or more samples. It may be determined, based on performing the operation on the one or more samples, whether to perform the pushdown of the operation.

Abstract: Machines, systems and methods of uploading data files, the method comprising a first client machine dividing a first file into N data chunks to be uploaded to a server, wherein the N data chunks are of size kX, where k is an integer and X is size of a minimal size data chunk, wherein X is known by the server and by at least a second client machine used for uploading a second file to the server in data chunks of size k?X; and uploading the first file to the server, wherein a first unique signature is calculated for the first file based on applying a signature function to a collection of signatures calculated for the minimal size data chunks of size X that make up the data chunks of size kX in the first file, wherein the uploading of the first file is accomplished by uploading the data chunks of size kX to the server in any order.

Abstract: Machines, systems and methods of uploading data files, the method comprising a first client machine dividing a first file into N data chunks to be uploaded to a server, wherein the N data chunks are of size kX, where k is an integer and X is size of a minimal size data chunk, wherein X is known by the server and by at least a second client machine used for uploading a second file to the server in data chunks of size k?X; and uploading the first file to the server, wherein a first unique signature is calculated for the first file based on applying a signature function to a collection of signatures calculated for the minimal size data chunks of size X that make up the data chunks of size kX in the first file, wherein the uploading of the first file is accomplished by uploading the data chunks of size kX to the server in any order.

Abstract: Systems and methods for estimating data reduction ratio for a data set is provided. The method comprises selecting a plurality of m elements from a data set comprising a plurality of N elements; associating an identifier hi for each of the plurality of m elements; associating an identifier he for each of the plurality of elements in the data set; tracking number of times an element i appears in a base set that includes the plurality of m elements selected from the data set; calculating a value counti that indicates the number of times an identifier he matches an identifier hi; and estimating data reduction ratio for the plurality of N elements in the data set, based on number of m number elements selected from the data set and the value counti.

Abstract: Systems and methods for estimating data reduction ratio for a data set is provided. The method comprises selecting a plurality of m elements from a data set comprising a plurality of N elements; associating an identifier hi for each of the plurality of m elements; associating an identifier he for each of the plurality of elements in the data set; tracking number of times an element i appears in a base set that includes the plurality of m elements selected from the data set; calculating a value counti that indicates the number of times an identifier he matches an identifier hi; and estimating data reduction ratio for the plurality of N elements in the data set, based on number of m number elements selected from the data set and the value counti.