Abstract: A technique for generating a list of all N-bit unsigned binary numbers by starting with an initial number less than some power of 2, successively multiplying the number by that power of 2 and adding the largest non-negative number less than that power of 2 such that the new number is not a duplicate of any of those already generated, and using the resulting lists to generate efficient hashing and serial decoding hardware and software.

Abstract: A method for data distribution, including distributing logical addresses among an initial set of devices so as provide balanced access, and transferring the data to the devices in accordance with the logical addresses. If a device is added to the initial set, forming an extended set, the logical addresses are redistributed among the extended set so as to cause some logical addresses to be transferred from the devices in the initial set to the additional device. There is substantially no transfer of the logical addresses among the initial set. If a surplus device is removed from the initial set, forming a depleted set, the logical addresses of the surplus device are redistributed among the depleted set. There is substantially no transfer of the logical addresses among the depleted set. In both cases the balanced access is maintained.

Abstract: Provided are a cache memory using a linear hash function and a method of operating the same. The cache memory includes: a first hash function module for converting a main memory address received from a central processing unit (CPU) into a first index value using a first hash function; a second hash function module for converting the main memory address into a second index value using a second hash function; a first comparator for comparing a tag value of a data block located at the first index value in the first bank with a tag value of the main memory address; and a second comparator for comparing a tag value of a data block located at the second index value in the second bank with the tag value of the main memory address.

Abstract: An embodiment relates generally to a method of preventing resource access conflicts in a software component. The method includes intercepting a lock operation in the software component and testing an associated lock type of the lock operation against a set of rules. The method also includes determining an action based on the associated lock type conflicting one of the rules of the set of rules.

Abstract: A compression device recognizes patterns of data and compressing the data, and sends the compressed data to a decompression device that identifies a cached version of the data to decompress the data. Both the compression device and the decompression device cache the data in packets they receive. Each device has a disk, on which each device writes the data in the same order. The compression device looks for repetitions of any block of data between multiple packets or datagrams that are transmitted across the network. The compression device encodes the repeated blocks of data by replacing them with a pointer to a location on disk. The decompression device receives the pointer and replaces the pointer with the contents of the data block that it reads from its disk.

Abstract: A system, method, and computer program product for harvesting an image from a local disk of a managed endpoint to an image library is provided. In an embodiment of the method for harvesting an image, a managed endpoint is provided with a boot image that causes the endpoint to instantiate a RAM disk and execute the boot image from the RAM disk. The boot image is used to harvest an image by determining data on a local disk of the managed endpoint to be included in the image that are not already stored in the image library. In one embodiment, this is done by comparing hashes calculated on the data on the local disk to hashes of data in the image library. The data not already stored in the image library are then copied to the image library.

Abstract: This invention achieves data capacity efficiency via data de-duplication and maximizes a power-saving effect by disk operation control. In a storage system, when data is received from a computer, a value representing the bit string for the data is calculated and whether or not a value identical to the calculated value is stored in a data management table is judged. If it is judged that an identical value is not stored, the received data is registered in the data management table and stored, based on a group management table, in disk device(s) associated with a logical unit number of a logical unit constituting an active group. Meanwhile, if it is judged that an identical value is stored, the received data is stored in disk device(s) based on the group information managed in the group management table and the management information managed in the data management table.

Abstract: A method for protecting electronic keys sets a plurality of hash functions, divides an electronic key into a plurality of key segments, creates a data storage structure for each of the key segments, and calculates a hash address for each of the key segments of the electronic key using each of the hash functions. The method further obtains a plurality of hash addresses of the plurality of key segments corresponding to the plurality of hash functions, stores information of the data storage structure of each key segment in a hash table according to the hash address of the key segment corresponding to one of the hash functions.

Abstract: A computer system includes a disk space comprising at least one type of memory and an operating system for controlling allocations and access to the disk space. A runtime machine runs applications through at least one of the operating system or directly on at least one processor of the computer system. In addition, the runtime machine manages a selected runtime disk space allocated to the runtime machine by the operating system and manages a separate method cache within the selected virtual disk space. The virtual machine controls caching within the method cache of a separate result of at least one method of the application marked as cache capable. For a next instance of the method detected by the runtime machine, the runtime machine accesses the cached separate result of the method in lieu of executing the method again.

Abstract: A method for implementing a hash map to improve performance consistency is disclosed herein. In one embodiment, such a method includes providing a hash map comprising a set of tables, the set of tables initially containing a first table. When the first table is full, the method augments the hash map by adding a second table to the set of tables. Similarly, when the second table is full, the method augments the hash map by adding a third table to the set of tables. A similar technique may be used to add additional tables to the hash map. When searching for a value in any of the tables in the hash map, the method uses the same hash code. A corresponding computer program product and apparatus are also disclosed herein.

Abstract: A virtual block device is an interface with applications that appears to the applications as a memory device, such as a standard block device. The virtual block device interacts with additional elements to do data deduplication to files at the block level such that one or more files accessed using the virtual block device have at least one block which is shared by the one or more files.

Abstract: A multimode accessible storage facility (10) is described that allows block access in a block access mode and row access in a row access mode. The facility comprises—a memory unit (20) comprising a plurality of memory banks (20.0, . . . , 20.F) each having a respective bank index (0, . . . , F), —an address generator (30) for generating for each of said memory banks a rotated bank address as a function of an input address and a shift parameter, —an input vector data rotator (40) for rotating an input vector and for providing vector elements of the rotated input vector to a respective bank of the memory unit, and —an output vector rotator (50) for inverse rotating a vector comprising vector elements retrieved from respective banks of the memory unit and for providing the rotated output vector.

Abstract: A system has a data structure in which a value can be obtained from a key. In a write access, a first pair <Key,Hash(Value)> and a second pair <Hash(Value),Value> are stored respectively in a volatile storage device. The first pair <Key,Hash(Value)> is saved in a nonvolatile storage device before returning a response, and the second pair <Hash(Value),Value> is saved in the first storage device at any time with the second pair saved in the volatile storage device. In a read access in which a value is obtained from a key, it is determined that data is not stored normally if the second pair is not found in processing in which after obtaining the hash value of the value from the first pair, the second pair is read.

Abstract: A comparand word is input to a plurality of hash circuits, with each hash circuit responding to a different portion of the comparand word. The hash circuits output a hash signal which enables or precharges portions of a content addressable memory CAM. The comparand word is also input to the CAM. The CAM compares the comparand word in the precharged portions of the CAM and outputs information responsive to the comparison. When Internet addresses are processed, the output information is either port information or an index for locating port information.

Abstract: Disclosed herein are a method, a system, and a computer-readable recording medium for providing distributed programming environment by using a distributed space. According to an aspect of the present invention, there is provided a method for processing data in distributed environment, the method including: generating a virtual space using resources provided by a plurality of nodes; and reading or writing data from or in the virtual space by a first application, wherein the data are mapped to a specific location region on the virtual space determined according to attributes of the data and the first application performs a reading operation or a writing operation for the data in the location region.

Abstract: Page data of a virtual machine is represented for efficient save and restore operations. One form of representation applies to each page with an easily identifiable pattern. The page is described, saved, and restored in terms of metadata reflective of the pattern rather than a complete page of data reflecting the pattern. During a save or restore operation, however, the metadata of the page is represented, but not the page data. Another form of representation applies to each page sharing a canonical instance of a complex pattern that is instantiated in memory during execution, and explicitly saved and restored. Each page sharing the canonical page is saved and restored as a metadata reference, without the need to actually save redundant copies of the page data.

Abstract: A mechanism for random cache line selection in virtualization systems is disclosed. A method includes maintaining a secondary data structure representing a plurality of memory pages, the secondary data structure indexed by a subset of each memory page, determining an index of a received new memory page by utilizing a subset of the new memory page that is a same size and at a same offset as the subset of each memory page, comparing the index of the new memory page with the indices of the secondary data structure for a match, utilizing a main data structure to perform a full page memory comparison with the new memory page if a match is found in the secondary data structure, and updating at least one of the size of the subset, the number of subsets, and the offsets of the subsets used to index the memory page.

Abstract: Embodiments of the invention relate to associating a source string with a target content unit stored on a content addressable storage (CAS) system. This may be accomplished, in some embodiments, by storing on the CAS system an associative content unit that includes the source string in its binding part and includes the target content unit in its non-binding part.

Abstract: Systems and methods for pre-fetching of data in a memory are provided. By pre-fetching stored data from a slower memory into a faster memory, the amount of time required for data retrieval and/or processing may be reduced. First, data is received and pre-scanned to generate a sample fingerprint. Fingerprints stored in a faster memory that are similar to the sample fingerprint are identified. Data stored in the slower memory associated with the identified stored fingerprints is copied into the faster memory. The copied data may be compared to the received data. Various embodiments may be included in a network memory architecture to allow for faster data matching and instruction generation in a central appliance.

Abstract: Memory management of processing systems running in a virtual computer environment and of processes running in an operating system environment includes identifying a usage pattern of a page in memory. The usage pattern is identified by tracking operations conducted with respect to the page. The memory management also includes designating the page as a candidate for sharing when the usage pattern reflects that a number of updates made to the page does not exceed a predefined threshold value. The candidate page is allocated to a first process or virtual machine. The memory management also includes sharing access to the candidate page with a second process or virtual machine when content in the candidate page matches content of page allocated for the second process or virtual machine to an address space of the candidate page.

Abstract: Computers are caused to provide a hash table wherein each entry is associated with a binary key and indexed by a selected portion of a hash value of the associated key, and points to a data structure location for storing non-selected portions of, or the entire hash value of, the binary key, and action data corresponding to the value of the binary key. Content addressable memory entries store a binary key, or a value unique to it, and an association to a corresponding action. Pointers to the data structure use selected portions of binary key hash values as an index when not selected portions of hash values of other binary keys, and associations are established between CAM entry and associated data structure locations when selected portions of the hash values of the binary keys are the same as selected portions of hash values of one or more other binary keys.

Abstract: Systems for performing inline wire speed data deduplication are described herein. Some embodiments include a device for inline data deduplication that includes one or more input ports for receiving an input data stream containing duplicates, one or more output ports for providing a data deduplicated output data stream, and an inline data deduplication engine coupled to said one or more input ports and said one or more output ports to process input data containing duplicates into output data which is data deduplicated, said inline data deduplication engine having an inline data deduplication bandwidth of at least 4 Gigabytes per second.

Abstract: Systems, methods embodied on computer-readable media, and other embodiments associated with index entry eviction are described. One example method includes selecting an index entry for eviction from a bucket of index entries based on a time value, a utility value, and a precedence value. A precedence value may be a value associated with an index entry that is static over time. Additionally, results of a function that compares two precedence values may be static over time. The example method may also include providing an index entry identifier that identifies the index entry.

Abstract: A method for partitioning during an online node add. The method includes providing a data storage cluster with first and second nodes, and storing a table of data in the data storage cluster with a first partition storing a set of rows or data elements in the first node and a second partition storing a set of rows or data elements in the second node. The method includes adding a third node to the cluster and adding a third partition to the table using a partitioning mechanism to create a distribution mapping for data elements in the first, second, and third partitions. The distribution mapping provides substantially uniform distribution of the data elements over the first, second, and third partitions by the partitioning mechanism using modulo hash partitioning as a function of data elements or by combining hash and list partitioning such that data is retained on the original partitions.

Abstract: Access to content addressable data on a network is facilitated using digital information storing devices or data repositories (“silos”) that monitor broadcast data requests over the network. A number of silos automatically monitor both data requests and data itself that are broadcast over a network. The silos selectively store data. Each silo responds to data requests broadcast over the network with data the silo has previously intercepted. A content addressable file scheme is used to enable the data repositories to reliably identify data being requested. When a data request is received, each silo evaluates whether it has all or a portion of the data being requested and responds to requests when it has the data. Requests for data are implemented by broadcasting a cryptographic hash data identifier of the data file needed. The data identifier is used by a silo to determine which data to receive and store.

Abstract: An encryption for a distributed global online backup system with global two-stage deduplication in the absence of an indexing database where data blocks are encrypted using their SHA-1 signatures as encryption keys.

Abstract: Apparatus and method that schedules movement of packets within network devices, such as network processors, includes a calendar using a segmented hierarchical routine to identify the next packet to be moved from one of a plurality of flow queues.

Abstract: A method for estimating contents of a cache determines table descriptors referenced by a query, and scans each page header stored in the cache for the table descriptor. If the table descriptor matches any of the referenced table descriptors, a page count value corresponding to the matching referenced table descriptor is increased. Alternatively, a housekeeper thread periodically performs the scan and stores the page count values in a central lookup table accessible by threads during a query run. Alternatively, each thread independently maintains a hash table with page count entries corresponding to table descriptors for each table in the database system. A thread increases or decreases the page count value when copying or removing pages from the cache. A page count value for each referenced table descriptor is determined from a sum of the values in the hash tables. A master thread performs bookkeeping and prevents hash table overflows.

Abstract: A network device determines the presence of the pre-specified string in a message based on a sequence matching rule. A sequence represents non-contiguous portions of the message. A combination of content addressable memory, programmable processing units, and the programmable control unit may determine the presence of the pre-specified string in the message by comparing the non-contiguous portions of the message. Such an approach may reduce the computational resources required for searching the pre-specified string in the message.

Abstract: The subject matter of this specification can be embodied in, among other things, a computer-implemented method for counting one or more unique search results within a plurality of search results includes creating hash values for information in each of the search results using a first hash function. The first hash function has a predetermined hash value range size. The method further includes identifying a predetermined number of smallest hash values within the created hash values. The method further includes estimating a first number of unique search results based on the predetermined hash value range size, the predetermined number, and a largest hash value in the smallest hash values.

Abstract: A computer readable storage medium, embodying instructions executable by a computer to perform a method, the method including: validating a memory write of data segments using a first number of leaf hashes of a first hash tree, where each of the first number of leaf hashes is associated with one of the data segments of a first block size, generating interior node hashes based on the first number of leaf hashes, where each of the interior node hashes is associated with a second block size, generating a first root hash using the interior node hashes, where the first root hash is associated with a remote procedure call size, transmitting the first root rash and the data segments to a network file system, where the transmission is performed using the remote procedure call size, and validating the transmission of the data segments using the first root hash.

Abstract: An approach identifies an amount of high order bits used to store a memory address in a memory address field that is included in a memory. This approach calculates at least one minimum number of low order bits not used to store the address with the calculation being based on the identified amount of high order bits. The approach retrieves a data element from one of the identified minimum number of low order bits of the address field and also retrieves a second data element from one of the one of the identified minimum number of low order bits of the address field.

Abstract: A system, method and computer program product for resizing a hash table while supporting hash table scalability and concurrency. The hash table has one or more hash buckets each containing one or more items that are chained together in a linked list. Each item in the hash table is processed to determine if the item requires relocation from a first bucket associated with a first table size to second bucket associated with a second table size. If the item requires relocation, it is linked to the second bucket without moving or copying the item in memory. The item is unlinked from the first bucket after waiting until there is no current hash table reader whose search of the hash table could be affected by the unlinking, again without moving or copying the item in memory.

Abstract: The subject disclosure is directed towards a data deduplication technology in which a hash index service's index maintains a hash index in a secondary storage device such as a hard drive, along with a compact index table and look-ahead cache in RAM that operate to reduce the I/O to access the secondary storage device during deduplication operations. Also described is a session cache for maintaining data during a deduplication session, and encoding of a read-only compact index table for efficiency.

Abstract: The subject disclosure is directed towards a data deduplication technology in which a hash index service's index maintains a hash index in a secondary storage device such as a hard drive, along with a compact index table and look-ahead cache in RAM that operate to reduce the I/O to access the secondary storage device during deduplication operations. Also described is a session cache for maintaining data during a deduplication session, and encoding of a read-only compact index table for efficiency.

Abstract: Described is using flash memory, RAM-based data structures and mechanisms to provide a flash store for caching data items (e.g., key-value pairs) in flash pages. A RAM-based index maps data items to flash pages, and a RAM-based write buffer maintains data items to be written to the flash store, e.g., when a full page can be written. A recycle mechanism makes used pages in the flash store available by destaging a data item to a hard disk or reinserting it into the write buffer, based on its access pattern. The flash store may be used in a data deduplication system, in which the data items comprise chunk-identifier, metadata pairs, in which each chunk-identifier corresponds to a hash of a chunk of data that indicates. The RAM and flash are accessed with the chunk-identifier (e.g., as a key) to determine whether a chunk is a new chunk or a duplicate.

Abstract: A storage system comprises one or more pool volumes having chunks for storing data; one or more primary volumes; writable snapshots as virtual volumes for each primary volume which is a common ancestor of the writable snapshots, each primary volume and corresponding writable snapshots being members forming a snapshot group; and a storage controller which includes a processor, a memory storing, for each snapshot group, group information of the members within the snapshot group, and a deduplication module. The deduplication module may identify a snapshot group for deduplication based on the group information and perform deduplication of data for the identified snapshot group in a deduplication area, or perform deduplication of data in a deduplication area which is specified based on the group information of a snapshot group being generated in the storage system.

Abstract: A system comprising a plurality of storage systems, which uses storage devices of multiple levels of reliability. The reliability as a whole system is increased by keeping the error code for the relatively low reliability storage disks in the relatively high reliability storage system. The error code is calculated using hash functions and the value is used to compare with the hash value of the data read from the relatively low reliability storage disks.

Abstract: A write barrier is implemented using thread-local hash table based write barrier buffers. The write barrier, executed by mutator threads, stores addresses of written memory locations or objects in the thread-local hash tables, and during garbage collection, an explicit or implicit union of the addresses in each hash table is used in a manner that is tolerant to an address appearing in more than one hash table.

Abstract: A data deduplication method using a small hash digest dictionary in fast-access memory. The method includes receiving customer data, dividing the data into smaller chunks, and assigning hash values to each chunk. For each chunk, the method includes performing lookup for a duplicate chunk by accessing a small dictionary in memory with the chunk's hash value. When no entry, the small dictionary is updated to include the hash value to fill the dictionary with earliest received data. When an entry is found, the entry's hash value is compared with lookup value and if matched, reference data is returned and an entry counter is incremented. If not matched, additional accesses are attempted such as with additional indexes calculated using the hash value. Collisions may trigger an entry replacement such that some initially entered entries are replaced when determined to not be most repeating values such as based on their counter value.

Abstract: A cache-aware Bloom filter system segments a bit vector of a cache-aware Bloom filter into fixed-size blocks. The system hashes an item to be inserted into the cache-aware Bloom filter to identify one of the fixed-size blocks as a selected block for receiving the item and hashes the item k times to generate k hashed values for encoding the item for insertion in the in the selected block. The system sets bits within the selected block with addresses corresponding to the k hashed values such that accessing the item in the cache-aware Bloom filter requires accessing only the selected block to check the k hashed values. The size of the fixed-size block corresponds to a cache-line size of an associated computer architecture on which the cache-aware Bloom filter is installed.

Abstract: The illustrative embodiments described herein provide a computer implemented method, apparatus, and computer program product for increasing efficiency associated with data access. In one illustrative embodiment a memory chip is presented comprising of a plurality of memory units for storing data; a plurality of processing units for processing the data; and a word line and a bit line external to the plurality of memory units, wherein the plurality of processing units directly access the word line and the bit line in accessing the data.

Abstract: A system and method for data deduplication is presented. A deduplication system includes a plurality of data deduplication nodes; and a data deduplication control node communicatively coupled to the plurality of data deduplication nodes. The deduplication control node performs the operations of performing data deduplication operations at a deduplication system having one or more deduplication nodes at which deduplication data blocks are stored; monitoring a performance characteristic of the deduplication system; and adding an additional deduplication node to the deduplication system when the performance characteristic indicates performance of the system below a predetermined minimum.

Abstract: Described embodiments provide a hash processor for a system having multiple processing modules and a shared memory. The hash processor includes a descriptor table with N entries, each entry corresponding to a hash table of the hash processor. A direct mapped table in the shared memory includes at least one memory block including N hash buckets. The direct mapped table includes a predetermined number of hash buckets for each hash table. Each hash bucket includes one or more hash key and value pairs, and a link value. Memory blocks in the shared memory include dynamic hash buckets available for allocation to a hash table. A dynamic hash bucket is allocated to a hash table when the hash buckets in the direct mapped table are filled beyond a threshold. The link value in the hash bucket is set to the address of the dynamic hash bucket allocated to the hash table.

Abstract: During mirroring operations, de-duplication operations are applied to the mirror volumes. Data stored to a first volume is mirrored to a second volume. The second volume is a virtual volume having a plurality of logical addresses, such that segments of physical storage capacity are allocated for a specified logical address as needed when data is stored to the specified logical address. A de-duplication operation is carried out on the second volume following a split from the first volume. A particular segment of the second volume is identified as having data that is the same as another segment in the second volume or in the same consistency group. A link is created from the particular segment to the other segment and the particular segment is released from the second volume.

Abstract: A mechanism for dynamic placement of virtual machines (VMs) during live migration based on memory is disclosed. A method of embodiments of the invention includes determining candidate target host machines capable of receiving a VM to be migrated, obtaining a hash value for memory pages of the VM to be migrated, obtaining for each candidate target host machine hash values for shared memory pages utilized by one or more VMs hosted by the candidate target host machine, comparing for each candidate target host machine the hash values for the memory pages of the VM to be migrated with the hash values for the shared memory pages, and adjusting a score in a general selection algorithm for the candidate target host machine with the most identical matches of the hash values for the shared memory pages with the hash values for the memory pages of the VM to be migrated.

Abstract: An apparatus and a method operating on data at a server node of a data grid system with distributed cache is described. A coordinator receives a request to change a topology of a cache cluster from a first group of cache nodes to a second group of cache nodes. The request includes a cache node joining or leaving the first group. A key for the second group is rehashed without blocking access to the first group while rehashing.

Abstract: Described herein is a method and system for distributing whole and fragmented requests and responses across a multi-core system. Each core executes a packet engine that further processes data packets and data packet fragments allocated to that core. A flow distributor executing within the multi-core system forwards client requests to a packet engine on a core that is selected based on a value generated when a hash is applied to a tuple comprising a client IP address, a client port, a server IP address and a server port identified in the request. The packet engine maintains each element of the tuple and forwards the request to the selected core. The packet engine can also process data packet fragments by assembling the fragments prior to transmitting them to the selected core, or by transmitting the data packet fragments to the selected core.

Abstract: Some embodiments relate to a migration policy that controls migration of content from a transactional storage system to a fixed content storage system. In some embodiments, content may be migrated from the transactional storage system to the fixed content storage system based on at least one criteria that relates to whether at least a portion of the content is binding.

Abstract: An Ethernet controller, for use within an Ethernet network of other Ethernet controller connected together by a bus, is provided. The Ethernet controller includes a plurality of ports including at least one bus port associated with ports connected to other switching Ethernet controllers, a hash table for storing addresses of ports within the Ethernet network, a hash table address control, a storage buffer including a multiplicity of contiguous buffers in which to temporarily store said packet, an empty list including a multiplicity of single bit buffers, a packet storage manager, a packet transfer manager and a write-only bus communication unit. The hash table address control hashes the address of a packet to initial hash table location values, changes the hash table location values by a fixed jump amount if the address values stored in the initial hash table location do not match the received address, and provides at least an output port number of the port associated with the received address.