Abstract: Origin offload is a key performance indicator of a content delivery network (CDN). This patent document presents unique methods and systems for measuring origin offload and applying those measurements to improve the offload. The techniques presented herein enable resource-efficient measurement of origin offload by individual servers and aggregation and analysis of such measurements to produce significant insights. The teachings hereof can be used to better identify root causes of suboptimal offload performance, to tune CDN settings and configurations, and to modify network operations, deployment and/or capacity planning. In addition, discussed herein are improved metrics showing offload in relation to the maximum achievable offload for the particular traffic being served.

Abstract: Improved technology for managing the caching of objects that are rarely requested by clients. A cache system can be configured to assess a class of objects (such as objects associated with a particular domain) for cacheability, based on traffic observations. If the maximum possible cache offloading for the class of objects falls below a threshold level, which indicates a high proportion of non-cacheable or “single-hitter” content, then cache admission logic is configured to admit objects only after multiple clients requests during a time period (usually the object's time in cache, or eviction age). Otherwise, the cache admission logic may operate to admit objects to the cache after the first client request, assuming the object meets cacheability criteria. The technological improvements disclosed herein can be used to improve cache utilization, for example by preventing single-hitter objects from pushing out multi-hit objects (the objects that get hits after being added to cache).

Abstract: Origin offload is a key performance indicator of a content delivery network (CDN). This patent document presents unique methods and systems for measuring origin offload and applying those measurements to improve the offload. The techniques presented herein enable resource-efficient measurement of origin offload by individual servers and aggregation and analysis of such measurements to produce significant insights. The teachings hereof can be used to better identify root causes of suboptimal offload performance, to tune CDN settings and configurations, and to modify network operations, deployment and/or capacity planning. in addition, discussed herein are improved metrics showing offload in relation to the maximum achievable offload for the particular traffic being served.

Abstract: Origin offload is a key performance indicator of a content delivery network (CDN). This patent document presents unique methods and systems for measuring origin offload and applying those measurements to improve the offload. The techniques presented herein enable resource-efficient measurement of origin offload by individual servers and aggregation and analysis of such measurements to produce significant insights. The teachings hereof can be used to better identify root causes of suboptimal offload performance, to tune CDN settings and configurations, and to modify network operations, deployment and/or capacity planning. In addition, discussed herein are improved metrics showing offload in relation to the maximum achievable offload for the particular traffic being served.

Abstract: A computer-implemented method includes dividing a data set into a plurality of regions and dividing the plurality of regions into a plurality of chunks of fixed size. The computer-implemented method further includes determining a sample size of the plurality of chunks to be sampled for each region, wherein the sample size is determined based, at least in part, on an acceptance of a likelihood of identifying at least one collision between two regions corresponding to logical entities of a first cluster of logical entities. The computer-implemented method further includes sampling the plurality of chunks for each region based on the determined sample size. The computer-implemented method further includes generating a hash value for each chunk sampled and storing each hash value in an index. The computer-implemented method further includes identifying one or more collisions between the plurality of regions. A corresponding computer system and computer program product are also disclosed.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include partitioning a dataset into a full set of logical data units, and selecting a sample subset of the full set, the sample subset including a random sample of the full set based on a sampling ratio. A set of target hash values are selected from a full range of hash values, and, using a hash function, a respective unit hash value is calculated for each of the logical data units in the sample subset. A histogram is computed that indicates a duplication count of each of the unit hash values that matches a given target hash value, and based on the histogram, a number of distinct logical data units in the full set is estimated.

Abstract: A method and system for determining a position of tagged objects by their proximity to each other, where the position of a target object is provided as common names of other objects nearby. This allows new ways of tracking objects that are required to be together with each other, or to be within a certain area, while reducing the power consumption of electronic tags and tag readers.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include configuring a storage system to store multiple storage entities for access by one or more host computers in communication with the storage system, and specifying a compression condition including a minimum compression ratio. The storage system can then estimate an expected compression ratio for a given storage entity, compress the given storage entity upon the expected compression ratio meeting the compression condition, and provide, to a given host computer, access to the compressed given storage entity.

Abstract: Various embodiments for data management in a replicated storage environment, by a processor device, are provided. In one embodiment, a method comprises storing a plurality of data replicas under a plurality of heterogeneous compression algorithms, wherein one of the data replicas is optimized for a data operation.

Abstract: Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate, the heuristic computed by, for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, and calculating an actual number of the characters for each of the data classes and the non-classifiable data.

Abstract: Methods and systems for data transfer include adding a data chunks to a priority queue in an order based on utilization priority. A reducibility score for the data chunks is determined. A data reduction operation is performed on a data chunk having a highest reducibility in the priority queue using a processor if sufficient resources are available. The data chunk having the lowest reducibility score is moved from the priority queue to a transfer queue for transmission if the transfer queue is not full.

Abstract: A globally distributed scan list is determined. A determination is made whether the first data replica in the first plurality of data stored on a first device is in sync with a second data replica in the second plurality of data on a second device. In response to determining that the first data replica is not in sync with the second data replica, the first data replica is added to an unsynced queue. The neighbor data of the first plurality of data is added to a suspect queue. The priority to check the neighbor data is increased if the neighbor data is already in the suspect queue. Unsynced neighbor data is added to the unsynced queue. The priority for recovery of the data in the unsynced queue is determined. The priority is based on the vulnerability of the data. A data replica in the unsynced queue is recovered.

Abstract: A computer-implemented method includes dividing a data set into a plurality of regions and dividing the plurality of regions into a plurality of chunks of fixed size. The computer-implemented method further includes determining a sample size of the plurality of chunks to be sampled for each region, wherein the sample size is determined based, at least in part, on an acceptance of a likelihood of identifying at least one collision between two regions corresponding to logical entities of a first cluster of logical entities. The computer-implemented method further includes sampling the plurality of chunks for each region based on the determined sample size. The computer-implemented method further includes generating a hash value for each chunk sampled and storing each hash value in an index. The computer-implemented method further includes identifying one or more collisions between the plurality of regions. A corresponding computer system and computer program product are also disclosed.

Abstract: A computer-implemented method includes dividing a data set into a plurality of regions and dividing the plurality of regions into a plurality of chunks of fixed size. The computer-implemented method further includes determining a sample size of the plurality of chunks to be sampled for each region, wherein the sample size is determined based, at least in part, on an acceptance of a likelihood of identifying at least one collision between two regions corresponding to logical entities of a first cluster of logical entities. The computer-implemented method further includes sampling the plurality of chunks for each region based on the determined sample size. The computer-implemented method further includes generating a hash value for each chunk sampled and storing each hash value in an index. The computer-implemented method further includes identifying one or more collisions between the plurality of regions. A corresponding computer system and computer program product are also disclosed.

Abstract: Machines, systems and methods for enhancing data recovery in a data storage system, the method comprising determining whether one or more data storage mediums in a data storage system are unavailable; determining data that are at a risk of loss, due to said one or more data storage mediums being unavailable; from among the data that is determined to be at the risk of loss, identifying data that is highly vulnerable to loss; and creating one or more temporary replicas of the data that is highly vulnerable to loss.

Abstract: A method, including partitioning a dataset into a first number of data units, and selecting, based on a sampling ratio, a second number of the data units. A hash value is calculated for each of the selected data units, and a first histogram is computed indicating a first duplication count for each of the calculated hash values. Based on respective frequencies of the calculated hash values, a second histogram is computed indicating an observed frequency for each of the first duplication counts in the first histogram, and based on the sampling ratio and the second histogram, a target function is derived. A third histogram that minimizes the target function is derived, the third histogram including, for the first number of the storage units, second duplication counts and a respective predicted frequency for each of the second duplication counts. Finally, a deduplication ratio is determined based on the third histogram.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include partitioning a dataset into a full set of logical data units, and selecting a sample subset of the full set, the sample subset including a random sample of the full set based on a sampling ratio. A set of target hash values are selected from a full range of hash values, and, using a hash function, a respective unit hash value is calculated for each of the logical data units in the sample subset. A histogram is computed that indicates a duplication count of each of the unit hash values that matches a given target hash value, and based on the histogram, a number of distinct logical data units in the full set is estimated.

Abstract: A method, including identifying, using a sampling ratio, a random number of logical data units. A hash is calculated for each of the identified logical data units, and a first histogram is computed indicating a duplication count of each of the calculated hashes. Based on respective frequencies of the calculated hashes, a second histogram is computed indicating observed frequencies of each of the duplication counts in the first histogram, and based on the sampling ratio and the second histogram, a target function is derived. A range of acceptable results is derived for the target function, and based on the range of the acceptable results, a set of plausible duplication frequency histograms is defined. A first given plausible duplication frequency histogram having a highest number of distinct logical data units is identified, and a second given plausible duplication frequency histogram having a lowest number of distinct logical data units is identified.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include partitioning a dataset into a full set of logical data units, and selecting a sample subset of the full set, the sample subset including a random sample of the full set based on a sampling ratio. A set of target hash values are selected from a full range of hash values, and, using a hash function, a respective unit hash value is calculated for each of the logical data units in the sample subset. A histogram is computed that indicates a duplication count of each of the unit hash values that matches a given target hash value, and based on the histogram, a number of distinct logical data units in the full set is estimated.

Abstract: A method for storage systems improvement includes collecting information that indicates one or more failure correlations for disks in a storage system. The disks are then separated into a plurality of virtual failure domains based on the indicated one or more failure correlations. The method then determines that all data objects of a set of redundant data objects are included in a first virtual failure domain. Responsive to determining that all data objects of the set of redundant data objects are included in the first virtual failure domain, the method then migrates at least one data object of the set of redundant data objects from a first disk in the first virtual failure domain to a second disk in a second virtual failure domain.