Abstract: One embodiment provides a method, including: receiving, at a central system, a query requesting access to a dataset, wherein the central system communicates with a plurality of data storage locations, each having a governance policy for data stored at the data storage location, wherein different portions of the dataset are stored within different of the plurality of data storage locations; sending a sub-query formulated based upon the query; receiving a governance enforcement actions listing corresponding to the portion of the dataset stored within the corresponding data storage location; generating a meta-policy of enforcement actions for all of the plurality of data storage locations storing portions of the dataset, wherein the meta-policy identifies enforcement actions and an order of the enforcement actions to be applied to the dataset; and providing the meta-policy to each of the plurality of data storage locations.

Abstract: Embodiments of the present systems and methods may provide a data access approval process that supports complex and fine-grained policies and can be applied to different data items at scale, which provides improvement over current technologies. For example, in an embodiment, a computer-implemented method for controlling access to data by computer systems may comprise generating an intermediate representation by integrating a combination of data access policies, data attributes including attributes per data subject, and the data itself to form the intermediate representation, receiving a request for access to the data, rewriting the request for access to the data to incorporate the intermediate representation so as to provide access only to data allowed by the policies integrated into the intermediate representation, and executing the rewritten request and providing only data allowed by the policies integrated into the intermediate representation.

Abstract: One embodiment provides a method, including: receiving, at a central system, a query requesting access to a dataset, wherein the central system communicates with a plurality of data storage locations, each having a governance policy for data stored at the data storage location, wherein different portions of the dataset are stored within different of the plurality of data storage locations; sending a sub-query formulated based upon the query; receiving a governance enforcement actions listing corresponding to the portion of the dataset stored within the corresponding data storage location; generating a meta-policy of enforcement actions for all of the plurality of data storage locations storing portions of the dataset, wherein the meta-policy identifies enforcement actions and an order of the enforcement actions to be applied to the dataset; and providing the meta-policy to each of the plurality of data storage locations.

Abstract: A computer-implemented method according to one embodiment includes identifying a plurality of storage systems within a storage environment, determining characteristics of each of the plurality of storage systems, the characteristics including one or more data reduction techniques implemented by each of the plurality of storage systems, performing a plurality of storage simulations of one or more data volumes, utilizing the characteristics of each of the plurality of storage systems, and determining one of the plurality of storage systems to store the one or more data volumes, based on results of the plurality of storage simulations.

Abstract: A method for managing a data cache, comprising: storing a cache management list comprising a plurality of entries and having: a tail part stored in a first storage and documenting recently accessed data items stored in the data cache, a body part stored in a second storage and documenting less recently accessed data items stored in the data cache, and a head part stored in the first storage and documenting least recently accessed data items stored in the data cache; and in each of a plurality of iterations: receiving at least one data access request; documenting the data access request in the tail; identifying a plurality of duplicated entries present in the body and the tail; and removing each of the plurality of duplicated entries from the body in the second storage according to a physical organization in the second storage of the plurality of duplicated entries.

Abstract: A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Abstract: Enforcement of policies for tabular data access as a collection of columns over a plurality of different information assets is provided. In an enforcement knowledge graph, information asset-assigned terms are found that correspond to information assets in a virtual information asset that references a set of tabular data. Transitive closures of the information asset-assigned terms are found in a business glossary to form a table of business glossary terms. Term intersection is determined between a hash table of any column-assigned terms and the table of business glossary terms. The information assets are assigned to the virtual information asset when the term intersection is not empty. A set of policy rules associated with the set of tabular data and a context of a user making a data access request to the set of tabular data is applied to the virtual information asset to determine an access enforcement decision.

Abstract: A method for managing a data cache, comprising: storing a cache management list comprising a plurality of entries and having: a tail part stored in a first storage and documenting recently accessed data items stored in the data cache, a body part stored in a second storage and documenting less recently accessed data items stored in the data cache, and a head part stored in the first storage and documenting least recently accessed data items stored in the data cache; and in each of a plurality of iterations: receiving at least one data access request; documenting the data access request in the tail; identifying a plurality of duplicated entries present in the body and the tail; and removing each of the plurality of duplicated entries from the body in the second storage according to a physical organization in the second storage of the plurality of duplicated entries.

Abstract: A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Abstract: A method, apparatus, system, and computer program product for evaluating enforcement decisions on an asset using a policy. Rules in the policy are applied by a computer system to the asset taking into account a context for a request to access the asset in response receiving to the request to access the asset, and wherein the rules in the policy determine whether access to the asset is allowed. A determination is made by the computer system as to whether a conflict is present in an initial decision made using the rules in the policy. A set of conflict resolution processes are applied by the computer system when the conflict is present such that a final decision is made on the request to access the asset.

Abstract: Computer program products, as well as corresponding systems and methods are configured for performing deduplication in conjunction with random read and write operations, and include: computing a fingerprint of data included in a write request; determining whether a short term dictionary comprises an entry corresponding to the fingerprint; in response to determining the short term dictionary comprises the entry corresponding to the fingerprint, writing the data to a data store in a deduplicating manner; in response to determining the short term dictionary does not comprise the entry, determining whether a long term dictionary corresponding to the namespace comprises the entry; in response to determining the long term dictionary comprises the entry, writing the data to the data store in the deduplicating manner; and in response to determining the long term dictionary does not comprise the entry, writing the data to the data store in a non-deduplicating manner.

Abstract: In one embodiment, a deduplicating storage system includes a processor and logic integrated with and/or executable by the processor. The logic is configured to cause the processor to perform a method which includes: computing a fingerprint of a data chunk, and determining whether a short term dictionary corresponding to the namespace comprises an entry corresponding to the fingerprint. In response to determining the short term dictionary does not comprise the entry, a determination is made whether a long term dictionary corresponding to the namespace comprises the entry. In response to determining the long term dictionary comprises the entry: the data chunk is written to the data store in the deduplicating manner, and the short term dictionary is repopulated with the entry. Moreover, in response to determining the long term dictionary does not comprise the entry, the data chunk is written to the data store in a non-deduplicating manner.

Abstract: A method, apparatus, system, and computer program product for evaluating enforcement decisions on an asset using a policy. Rules in the policy are applied by a computer system to the asset taking into account a context for a request to access the asset in response receiving to the request to access the asset, and wherein the rules in the policy determine whether access to the asset is allowed. A determination is made by the computer system as to whether a conflict is present in an initial decision made using the rules in the policy. A set of conflict resolution processes are applied by the computer system when the conflict is present such that a final decision is made on the request to access the asset.

Abstract: Enforcement of policies for tabular data access as a collection of columns over a plurality of different information assets is provided. In an enforcement knowledge graph, information asset-assigned terms are found that correspond to information assets in a virtual information asset that references a set of tabular data. Transitive closures of the information asset-assigned terms are found in a business glossary to form a table of business glossary terms. Term intersection is determined between a hash table of any column-assigned terms and the table of business glossary terms. The information assets are assigned to the virtual information asset when the term intersection is not empty. A set of policy rules associated with the set of tabular data and a context of a user making a data access request to the set of tabular data is applied to the virtual information asset to determine an access enforcement decision.

Abstract: A computer-implemented method includes dividing a data set into a plurality of regions and dividing the plurality of regions into a plurality of chunks of fixed size. The computer-implemented method further includes determining a sample size of the plurality of chunks to be sampled for each region, wherein the sample size is determined based, at least in part, on an acceptance of a likelihood of identifying at least one collision between two regions corresponding to logical entities of a first cluster of logical entities. The computer-implemented method further includes sampling the plurality of chunks for each region based on the determined sample size. The computer-implemented method further includes generating a hash value for each chunk sampled and storing each hash value in an index. The computer-implemented method further includes identifying one or more collisions between the plurality of regions. A corresponding computer system and computer program product are also disclosed.

Abstract: A computer-implemented method includes receiving a set of basis fingerprints corresponding to image chunks within a basis set of image regions wherein each image region within the basis set of image regions comprises one or more image chunks, and generating a fingerprint for each image chunk of a plurality of selected image chunks within an unprocessed region of a machine image to produce a plurality of sampled fingerprints. The method also includes determining a similarity metric for the unprocessed region from the sampled fingerprints and the basis fingerprints, comparing the similarity metric for the unprocessed region with a selected threshold, and including the unprocessed region within the basis set of image regions in response to determining that the similarity metric is less than the selected threshold. A corresponding computer program product and computer system are also disclosed herein.

Abstract: A computer-implemented method includes receiving a set of basis fingerprints corresponding to image chunks within a basis set of image regions wherein each image region within the basis set of image regions comprises one or more image chunks, and generating a fingerprint for each image chunk of a plurality of selected image chunks within an unprocessed region of a machine image to produce a plurality of sampled fingerprints. The method also includes determining a similarity metric for the unprocessed region from the sampled fingerprints and the basis fingerprints, comparing the similarity metric for the unprocessed region with a selected threshold, and including the unprocessed region within the basis set of image regions in response to determining that the similarity metric is less than the selected threshold. A corresponding computer program product and computer system are also disclosed herein.

Abstract: Methods, computing systems and computer program products implement embodiments of the present invention that include partitioning a dataset into a full set of logical data units, and selecting a sample subset of the full set, the sample subset including a random sample of the full set based on a sampling ratio. A set of target hash values are selected from a full range of hash values, and, using a hash function, a respective unit hash value is calculated for each of the logical data units in the sample subset. A histogram is computed that indicates a duplication count of each of the unit hash values that matches a given target hash value, and based on the histogram, a number of distinct logical data units in the full set is estimated.

Abstract: A computer-implemented method according to one embodiment includes identifying a plurality of storage systems within a storage environment, determining characteristics of each of the plurality of storage systems, the characteristics including one or more data reduction techniques implemented by each of the plurality of storage systems, performing a plurality of storage simulations of one or more data volumes, utilizing the characteristics of each of the plurality of storage systems, and determining one of the plurality of storage systems to store the one or more data volumes, based on results of the plurality of storage simulations.

Abstract: In one embodiment, a deduplicating storage system includes a processor and logic integrated with and/or executable by the processor. The logic is configured to cause the processor to perform a method which includes: computing a fingerprint of a data chunk, and determining whether a short term dictionary corresponding to the namespace comprises an entry corresponding to the fingerprint. In response to determining the short term dictionary does not comprise the entry, a determination is made whether a long term dictionary corresponding to the namespace comprises the entry. In response to determining the long term dictionary comprises the entry: the data chunk is written to the data store in the deduplicating manner, and the short term dictionary is repopulated with the entry. Moreover, in response to determining the long term dictionary does not comprise the entry, the data chunk is written to the data store in a non-deduplicating manner.